2024-04-26T01:18:58.746 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T01:18:58.747 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243 --verbose --name leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} -- /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/orig.config.yaml 2024-04-26T01:18:58.844 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:18:58.944 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7674243' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3448 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: afa1933b0fdbb6c99c947d1eda34d661d23cd327 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlsE4bIuJOYjuYlcXDpAU37FmMSFR02PenOXHuN6ARC6AXVZlCAcocRMjgVxzY0r2yPVaMOlof5GJe05/u/eMg= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDiP3f3MXF4PdVtlzdlkuUVuhuBZGLA13XYRkho5YDU/a88iMCf2zGKaCdBwhNeacLDgYewsdU/77wxVmEpZx+o= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOCKKC3srvdG16hmd3HpmC2lMy2JjxgpFJH8vzz/TbhLOHYb5EU+0HvWhKPFjzv3t6CRwXt4gvn+ExZ2BGOLd4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:02:28 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T01:18:58.944 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa; will attempt to use it 2024-04-26T01:18:58.945 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks 2024-04-26T01:18:58.945 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T01:18:58.947 INFO:teuthology.task.internal:Checking packages... 2024-04-26T01:18:58.979 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-26T01:18:58.980 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T01:18:58.980 INFO:teuthology.packaging:ref: None 2024-04-26T01:18:58.980 INFO:teuthology.packaging:tag: None 2024-04-26T01:18:58.980 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-26T01:18:58.980 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:18:58.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-26T01:18:59.159 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-26T01:18:59.160 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T01:18:59.167 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T01:18:59.167 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T01:18:59.208 INFO:teuthology.task.internal:Saving configuration 2024-04-26T01:18:59.230 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T01:18:59.239 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T01:18:59.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.640272', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlsE4bIuJOYjuYlcXDpAU37FmMSFR02PenOXHuN6ARC6AXVZlCAcocRMjgVxzY0r2yPVaMOlof5GJe05/u/eMg='} 2024-04-26T01:18:59.278 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.637573', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDiP3f3MXF4PdVtlzdlkuUVuhuBZGLA13XYRkho5YDU/a88iMCf2zGKaCdBwhNeacLDgYewsdU/77wxVmEpZx+o='} 2024-04-26T01:18:59.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.639046', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOCKKC3srvdG16hmd3HpmC2lMy2JjxgpFJH8vzz/TbhLOHYb5EU+0HvWhKPFjzv3t6CRwXt4gvn+ExZ2BGOLd4='} 2024-04-26T01:18:59.300 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T01:18:59.307 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T01:18:59.307 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T01:18:59.307 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T01:18:59.307 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T01:18:59.415 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f63284930d0>, signals=[15]) 2024-04-26T01:18:59.416 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T01:18:59.423 INFO:teuthology.task.internal:Opening connections... 2024-04-26T01:18:59.423 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:18:59.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:18:59.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-04-26T01:18:59.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:18:59.629 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-04-26T01:18:59.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:18:59.747 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T01:18:59.755 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-04-26T01:18:59.775 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-04-26T01:18:59.775 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-04-26T01:18:59.833 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-04-26T01:18:59.833 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:18:59.834 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:18:59.835 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:18:59.836 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-04-26T01:18:59.856 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-04-26T01:18:59.875 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-04-26T01:18:59.875 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-04-26T01:18:59.931 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:18:59.932 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:18:59.933 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:18:59.933 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-04-26T01:18:59.956 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-04-26T01:18:59.974 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-04-26T01:18:59.974 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-04-26T01:19:00.032 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T01:19:00.033 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T01:19:00.034 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T01:19:00.034 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-04-26T01:19:00.056 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T01:19:00.068 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T01:19:00.074 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T01:19:00.075 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:00.080 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:00.083 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:19:00.100 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T01:19:00.106 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T01:19:00.106 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:00.138 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:00.141 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:19:00.163 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T01:19:00.231 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T01:19:00.249 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-26T01:19:00.249 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T01:19:00.249 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T01:19:00.250 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:19:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:19:00.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T01:19:00.443 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:19:00.444 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T01:19:00.444 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-04-26T01:19:00.464 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-437.el9.x86_64 2024-04-26T01:19:00.464 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:19:00.465 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T01:19:00.465 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum upgrade -y linux-firmware 2024-04-26T01:19:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-26T01:19:04.854 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-26T01:19:09.505 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-04-26T01:19:11.003 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-26T01:19:14.171 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 21 MB 00:03 2024-04-26T01:19:19.396 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-04-26T01:19:21.481 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-26T01:19:21.482 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-04-26T01:19:21.482 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-26T01:19:21.571 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:19:21.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T01:19:21.798 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T01:19:21.799 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T01:19:21.801 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi052.front.sepia.ceph.com to complete... 2024-04-26T01:19:59.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26 01:19:59 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T01:19:59.973 INFO:teuthology.orchestra.run.smithi052.stderr:FINISHED --2024-04-26 01:19:59-- 2024-04-26T01:19:59.974 INFO:teuthology.orchestra.run.smithi052.stderr:Total wall clock time: 38s 2024-04-26T01:19:59.974 INFO:teuthology.orchestra.run.smithi052.stderr:Downloaded: 1 files, 341M in 38s (9.02 MB/s) 2024-04-26T01:19:59.977 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T01:19:59.977 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T01:20:02.172 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying... ######################################## 2024-04-26T01:20:02.309 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing... ######################################## 2024-04-26T01:20:02.425 INFO:teuthology.orchestra.run.smithi052.stdout:Updating / installing... 2024-04-26T01:20:10.793 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T01:20:48.659 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T01:20:48.659 DEBUG:teuthology.orchestra.run.smithi052:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T01:20:48.739 INFO:teuthology.orchestra.run.smithi052.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.739 INFO:teuthology.orchestra.run.smithi052.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.739 INFO:teuthology.orchestra.run.smithi052.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T01:20:48.740 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T01:20:48.741 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T01:20:48.742 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T01:20:48.743 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T01:20:48.744 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T01:20:48.745 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T01:20:48.746 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T01:20:48.747 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T01:20:48.748 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T01:20:48.749 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T01:20:48.750 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T01:20:48.751 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T01:20:48.752 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T01:20:48.753 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T01:20:48.754 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T01:20:48.755 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T01:20:48.756 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T01:20:48.757 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T01:20:48.758 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T01:20:48.759 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T01:20:48.760 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T01:20:48.761 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T01:20:48.762 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T01:20:48.763 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T01:20:48.764 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T01:20:48.765 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T01:20:48.766 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T01:20:48.767 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T01:20:48.768 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T01:20:48.769 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T01:20:48.770 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T01:20:48.771 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T01:20:48.772 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T01:20:48.773 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T01:20:48.774 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T01:20:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T01:20:48.776 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T01:20:48.777 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T01:20:48.778 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T01:20:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T01:20:48.780 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T01:20:48.781 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T01:20:48.782 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T01:20:48.783 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T01:20:48.784 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T01:20:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T01:20:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T01:20:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T01:20:48.788 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T01:20:48.789 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T01:20:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T01:20:48.791 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T01:20:48.792 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T01:20:48.793 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T01:20:48.794 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T01:20:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T01:20:48.796 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T01:20:48.797 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T01:20:48.798 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T01:20:48.799 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T01:20:48.800 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T01:20:48.801 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T01:20:48.802 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T01:20:48.803 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T01:20:48.804 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T01:20:48.805 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T01:20:48.806 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T01:20:48.807 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T01:20:48.808 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T01:20:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T01:20:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T01:20:48.811 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T01:20:48.812 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T01:20:48.813 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T01:20:48.814 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T01:20:48.815 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T01:20:48.816 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T01:20:48.817 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T01:20:48.818 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T01:20:48.819 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T01:20:48.819 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T01:20:48.821 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T01:20:48.821 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T01:20:48.821 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T01:20:48.821 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T01:20:48.822 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T01:20:48.823 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T01:20:48.824 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T01:20:48.825 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T01:20:48.826 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T01:20:48.827 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T01:20:48.828 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T01:20:48.829 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T01:20:48.830 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T01:20:48.831 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T01:20:48.832 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T01:20:48.833 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T01:20:48.834 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T01:20:48.835 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T01:20:48.836 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T01:20:48.837 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T01:20:48.838 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T01:20:48.839 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T01:20:48.840 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T01:20:48.841 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T01:20:48.842 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T01:20:48.843 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T01:20:48.844 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T01:20:48.845 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T01:20:48.846 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T01:20:48.847 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T01:20:48.848 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T01:20:48.849 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T01:20:48.850 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T01:20:48.851 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T01:20:48.852 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T01:20:48.853 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T01:20:48.854 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T01:20:48.855 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T01:20:48.856 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T01:20:48.857 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T01:20:48.858 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T01:20:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T01:20:48.860 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T01:20:48.861 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T01:20:48.862 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T01:20:48.863 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T01:20:48.864 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T01:20:48.865 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T01:20:48.866 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T01:20:48.867 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T01:20:48.868 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T01:20:48.869 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T01:20:48.870 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T01:20:48.871 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T01:20:48.872 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T01:20:48.873 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T01:20:48.874 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T01:20:48.875 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T01:20:48.876 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T01:20:48.878 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T01:20:48.879 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T01:20:48.880 INFO:teuthology.orchestra.run.smithi052.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:20:48.885 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.885 DEBUG:teuthology.orchestra.run.smithi052:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T01:20:48.910 INFO:teuthology.orchestra.run.smithi052.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout:fi 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout:for file in vmlinuz System.map config; do 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout: fi 2024-04-26T01:20:48.911 INFO:teuthology.orchestra.run.smithi052.stdout:done 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:fi 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:fi 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.912 INFO:teuthology.orchestra.run.smithi052.stdout:fi 2024-04-26T01:20:48.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-04-26T01:20:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T01:20:48.989 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-04-26T01:20:48.990 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:20:48.990 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T01:20:48.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T01:20:48.991 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:48.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T01:20:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-04-26T01:20:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T01:20:50.979 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-04-26T01:20:50.981 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-04-26T01:20:50.996 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.4gX28txvu8 2024-04-26T01:20:50.997 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4gX28txvu8 2024-04-26T01:20:51.066 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.4gX28txvu8 2024-04-26T01:20:51.207 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.4gX28txvu8 is 6KB 2024-04-26T01:20:51.228 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.4gX28txvu8 2024-04-26T01:20:51.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T01:20:51.309 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T01:20:51.309 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T01:20:51.309 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T01:20:51.309 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T01:20:51.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T01:20:51.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-04-26T01:21:21.461 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T01:21:21.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:21:21.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:21:39.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-04-26T01:21:48.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:21:48.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:22:48.875 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T01:23:00.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:23:00.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:23:01.345 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-04-26T01:23:01.966 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:23:01.966 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T01:23:01.966 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-04-26T01:23:02.020 INFO:teuthology.orchestra.run.smithi052.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T01:23:02.021 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T01:23:02.021 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:23:02.021 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T01:23:03.022 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:03.023 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T01:23:03.029 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T01:23:03.030 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:03.037 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:03.040 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:23:03.088 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T01:23:03.095 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T01:23:03.106 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T01:23:03.106 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:03.113 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:03.115 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:23:03.246 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T01:23:03.252 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T01:23:03.252 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:23:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:03.268 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:23:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:03.285 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:03.291 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:03.310 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:23:03.356 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.364 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.378 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.395 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.416 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.481 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:23:03.486 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T01:23:03.493 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T01:23:03.493 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:03.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:03.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:23:03.581 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T01:23:03.590 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T01:23:03.590 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:03.597 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:03.600 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:23:03.647 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:03.731 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:03.825 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:03.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:03.997 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:23:03.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:04.086 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:04.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:04.188 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:04.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:04.313 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:23:04.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:04.378 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:04.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:23:04.476 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:04.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:23:04.603 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:23:04.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:23:04.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-04-26T01:23:04.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-04-26T01:23:04.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-04-26T01:23:04.715 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:04.756 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:04.775 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T01:23:05.303 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T01:23:05.310 INFO:teuthology.task.internal:Starting timer... 2024-04-26T01:23:05.310 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T01:23:05.337 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T01:23:05.367 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T01:23:05.367 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-04-26T01:23:05.489 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-04-26T01:23:05.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-04-26T01:23:05.587 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-04-26T01:23:05.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-04-26T01:23:05.651 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-04-26T01:23:05.654 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T01:23:05.654 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-04-26T01:23:05.721 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-04-26T01:23:05.721 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-04-26T01:23:05.743 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-04-26T01:23:05.744 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-04-26T01:23:05.768 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-04-26T01:23:05.768 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-04-26T01:23:05.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:05.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:05.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T01:23:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:05.899 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T01:23:05.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:05.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:06.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-04-26T01:23:06.037 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T01:23:06.048 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T01:23:06.053 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T01:23:06.202 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T01:23:06.219 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T01:23:06.220 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi183.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T01:28:18.296 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-04-26T01:28:18.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:28:18.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:18.408 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-04-26T01:28:18.556 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-04-26T01:28:18.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-26T01:28:18.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:18.635 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-04-26T01:28:18.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-04-26T01:28:18.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-04-26T01:28:18.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:28:18.789 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-04-26T01:28:18.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-04-26T01:28:18.877 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T01:28:18.893 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T01:28:18.893 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:18.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:18.902 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:18.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:18.904 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:28:18.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:28:18.929 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:18.943 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:18.944 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:18.956 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:18.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-04-26T01:28:18.976 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-04-26T01:28:18.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-04-26T01:28:18.989 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-04-26T01:28:18.990 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:18.994 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T01:28:19.002 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:19.003 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:19.014 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:19.036 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T01:28:19.057 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:19.060 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:19.060 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-04-26T01:28:19.061 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.061 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.061 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.061 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.066 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:19.069 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:19.069 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-04-26T01:28:19.069 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.070 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.109 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-04-26T01:28:19.141 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-04-26T01:28:19.185 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T01:28:19.229 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T01:28:19.376 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-04-26T01:28:19.384 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T01:28:19.385 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-04-26T01:28:19.385 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.385 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.385 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.385 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T01:28:19.387 INFO:teuthology.run_tasks:Running task install... 2024-04-26T01:28:19.395 DEBUG:teuthology.task.install:project ceph 2024-04-26T01:28:19.395 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-26T01:28:19.395 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-26T01:28:19.396 INFO:teuthology.task.install:Using flavor: default 2024-04-26T01:28:19.401 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T01:28:19.401 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T01:28:19.402 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:19.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:19.404 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:19.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:19.405 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-26T01:28:19.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:28:19.556 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:19.556 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:19.622 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:19.622 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:19.630 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-26T01:28:19.630 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-26T01:28:19.680 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:19.680 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:28:19.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:19.777 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:19.778 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:19.792 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:19.793 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:28:19.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:19.803 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T01:28:19.803 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:28:19.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T01:28:19.822 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:19.822 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:19.829 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T01:28:19.829 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T01:28:19.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:19.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:19.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T01:28:19.969 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:19.969 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:20.025 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T01:28:20.032 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-04-26T01:28:20.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-04-26T01:28:20.034 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-04-26T01:28:20.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-04-26T01:28:20.103 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-04-26T01:28:20.107 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-04-26T01:28:20.258 INFO:teuthology.orchestra.run.smithi191.stdout:48 files removed 2024-04-26T01:28:20.266 INFO:teuthology.orchestra.run.smithi183.stdout:48 files removed 2024-04-26T01:28:20.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:20.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:20.438 INFO:teuthology.orchestra.run.smithi052.stdout:48 files removed 2024-04-26T01:28:20.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T01:28:20.895 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 239 kB/s | 83 kB 00:00 2024-04-26T01:28:20.923 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 227 kB/s | 83 kB 00:00 2024-04-26T01:28:21.270 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T01:28:21.298 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T01:28:21.403 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-04-26T01:28:21.604 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-04-26T01:28:21.657 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-26T01:28:21.978 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-26T01:28:22.206 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-26T01:28:22.277 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-26T01:28:22.496 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-04-26T01:28:23.715 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-04-26T01:28:24.016 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-26T01:28:24.144 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-26T01:28:25.865 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-26T01:28:28.172 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-26T01:28:28.259 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.2 MB 00:00 2024-04-26T01:28:29.570 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-26T01:28:29.832 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-26T01:28:30.652 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.2 MB 00:00 2024-04-26T01:28:32.505 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-26T01:28:32.775 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-04-26T01:28:35.435 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 21 MB 00:02 2024-04-26T01:28:35.670 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 21 MB 00:05 2024-04-26T01:28:37.821 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T01:28:40.014 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:40.014 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:40.025 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:28:40.026 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:28:40.027 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:28:40.028 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:28:40.029 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:28:40.031 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:28:40.032 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:28:40.093 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:28:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:28:40.110 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:28:40.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:28:40.112 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:28:40.113 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:28:40.114 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:28:40.115 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:28:40.116 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:28:40.117 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:28:40.118 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:28:40.119 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:28:40.120 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout:Install 148 Packages 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 4 Packages 2024-04-26T01:28:40.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:28:40.122 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 222 M 2024-04-26T01:28:40.122 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-04-26T01:28:40.708 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T01:28:41.014 INFO:teuthology.orchestra.run.smithi183.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-26T01:28:41.306 INFO:teuthology.orchestra.run.smithi183.stdout:(2/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 10 MB/s | 5.4 MB 00:00 2024-04-26T01:28:41.356 INFO:teuthology.orchestra.run.smithi183.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-26T01:28:41.523 INFO:teuthology.orchestra.run.smithi183.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 14 MB/s | 2.4 MB 00:00 2024-04-26T01:28:41.582 INFO:teuthology.orchestra.run.smithi183.stdout:(5/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.5 MB/s | 890 kB 00:00 2024-04-26T01:28:41.632 INFO:teuthology.orchestra.run.smithi183.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 14 MB/s | 1.5 MB 00:00 2024-04-26T01:28:41.674 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-26T01:28:42.674 INFO:teuthology.orchestra.run.smithi183.stdout:(7/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 16 MB/s | 17 MB 00:01 2024-04-26T01:28:42.859 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:42.860 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:42.871 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:28:42.872 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:28:42.872 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:28:42.873 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:28:42.875 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:28:42.877 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:28:42.878 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:28:42.938 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:42.948 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:28:42.949 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:28:42.950 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:28:42.951 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:28:42.952 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:28:42.953 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:28:42.954 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:28:42.955 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:28:42.956 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:28:42.957 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:28:42.958 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:28:42.959 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:28:42.960 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout:Install 148 Packages 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 4 Packages 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:28:42.961 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 222 M 2024-04-26T01:28:42.962 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-04-26T01:28:43.541 INFO:teuthology.orchestra.run.smithi183.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 7.6 MB/s | 21 MB 00:02 2024-04-26T01:28:43.658 INFO:teuthology.orchestra.run.smithi183.stdout:(9/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 11 MB/s | 11 MB 00:00 2024-04-26T01:28:43.683 INFO:teuthology.orchestra.run.smithi183.stdout:(10/152): ceph-selinux-19.0.0-3156.gd69f13d8.el 177 kB/s | 25 kB 00:00 2024-04-26T01:28:43.759 INFO:teuthology.orchestra.run.smithi183.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.9 MB/s | 225 kB 00:00 2024-04-26T01:28:43.809 INFO:teuthology.orchestra.run.smithi191.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 31 kB/s | 6.6 kB 00:00 2024-04-26T01:28:43.842 INFO:teuthology.orchestra.run.smithi183.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 325 kB/s | 27 kB 00:00 2024-04-26T01:28:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 381 kB/s | 32 kB 00:00 2024-04-26T01:28:44.051 INFO:teuthology.orchestra.run.smithi183.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 5.8 MB/s | 747 kB 00:00 2024-04-26T01:28:44.144 INFO:teuthology.orchestra.run.smithi183.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 163 kB 00:00 2024-04-26T01:28:44.227 INFO:teuthology.orchestra.run.smithi183.stdout:(16/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x8 1.8 MB/s | 4.7 MB 00:02 2024-04-26T01:28:44.252 INFO:teuthology.orchestra.run.smithi183.stdout:(17/152): librados-devel-19.0.0-3156.gd69f13d8. 1.1 MB/s | 126 kB 00:00 2024-04-26T01:28:44.325 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:44.327 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T01:28:44.348 INFO:teuthology.orchestra.run.smithi052.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T01:28:44.350 INFO:teuthology.orchestra.run.smithi052.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T01:28:44.351 INFO:teuthology.orchestra.run.smithi052.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T01:28:44.353 INFO:teuthology.orchestra.run.smithi052.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T01:28:44.355 INFO:teuthology.orchestra.run.smithi052.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T01:28:44.358 INFO:teuthology.orchestra.run.smithi052.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T01:28:44.360 INFO:teuthology.orchestra.run.smithi052.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T01:28:44.384 INFO:teuthology.orchestra.run.smithi191.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.5 MB/s | 890 kB 00:00 2024-04-26T01:28:44.437 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-04-26T01:28:44.458 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-04-26T01:28:44.458 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-26T01:28:44.459 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-26T01:28:44.460 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-26T01:28:44.461 INFO:teuthology.orchestra.run.smithi052.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-26T01:28:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T01:28:44.463 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T01:28:44.464 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T01:28:44.465 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T01:28:44.466 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T01:28:44.467 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T01:28:44.468 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T01:28:44.469 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:Install 148 Packages 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 4 Packages 2024-04-26T01:28:44.470 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:28:44.471 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 222 M 2024-04-26T01:28:44.471 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-04-26T01:28:44.551 INFO:teuthology.orchestra.run.smithi191.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 870 kB/s | 145 kB 00:00 2024-04-26T01:28:44.561 INFO:teuthology.orchestra.run.smithi183.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 1.5 MB/s | 500 kB 00:00 2024-04-26T01:28:44.669 INFO:teuthology.orchestra.run.smithi183.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 418 kB/s | 45 kB 00:00 2024-04-26T01:28:44.802 INFO:teuthology.orchestra.run.smithi183.stdout:(20/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 9.7 MB/s | 5.4 MB 00:00 2024-04-26T01:28:44.819 INFO:teuthology.orchestra.run.smithi183.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 897 kB/s | 135 kB 00:00 2024-04-26T01:28:44.887 INFO:teuthology.orchestra.run.smithi183.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.9 MB/s | 161 kB 00:00 2024-04-26T01:28:44.987 INFO:teuthology.orchestra.run.smithi183.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.9 MB/s | 298 kB 00:00 2024-04-26T01:28:45.020 INFO:teuthology.orchestra.run.smithi183.stdout:(24/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 320 kB 00:00 2024-04-26T01:28:45.071 INFO:teuthology.orchestra.run.smithi183.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.2 MB/s | 99 kB 00:00 2024-04-26T01:28:45.113 INFO:teuthology.orchestra.run.smithi183.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 928 kB/s | 85 kB 00:00 2024-04-26T01:28:45.388 INFO:teuthology.orchestra.run.smithi183.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 620 kB/s | 170 kB 00:00 2024-04-26T01:28:45.443 INFO:teuthology.orchestra.run.smithi191.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 2.6 MB/s | 2.4 MB 00:00 2024-04-26T01:28:45.446 INFO:teuthology.orchestra.run.smithi183.stdout:(28/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 8.2 MB/s | 3.1 MB 00:00 2024-04-26T01:28:45.472 INFO:teuthology.orchestra.run.smithi183.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3156.g 409 kB/s | 34 kB 00:00 2024-04-26T01:28:45.539 INFO:teuthology.orchestra.run.smithi183.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.5 MB/s | 142 kB 00:00 2024-04-26T01:28:45.851 INFO:teuthology.orchestra.run.smithi191.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 3.7 MB/s | 1.5 MB 00:00 2024-04-26T01:28:45.901 INFO:teuthology.orchestra.run.smithi052.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-26T01:28:46.414 INFO:teuthology.orchestra.run.smithi183.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-04-26T01:28:46.514 INFO:teuthology.orchestra.run.smithi183.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.4 MB/s | 249 kB 00:00 2024-04-26T01:28:46.606 INFO:teuthology.orchestra.run.smithi183.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 537 kB/s | 49 kB 00:00 2024-04-26T01:28:46.698 INFO:teuthology.orchestra.run.smithi183.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 180 kB/s | 16 kB 00:00 2024-04-26T01:28:46.798 INFO:teuthology.orchestra.run.smithi183.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.7 MB/s | 273 kB 00:00 2024-04-26T01:28:46.851 INFO:teuthology.orchestra.run.smithi052.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 938 kB/s | 890 kB 00:00 2024-04-26T01:28:46.990 INFO:teuthology.orchestra.run.smithi183.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 3.9 MB/s | 762 kB 00:00 2024-04-26T01:28:47.043 INFO:teuthology.orchestra.run.smithi052.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 759 kB/s | 145 kB 00:00 2024-04-26T01:28:47.090 INFO:teuthology.orchestra.run.smithi183.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 183 kB/s | 18 kB 00:00 2024-04-26T01:28:47.199 INFO:teuthology.orchestra.run.smithi183.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 740 kB/s | 80 kB 00:00 2024-04-26T01:28:47.274 INFO:teuthology.orchestra.run.smithi183.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-26T01:28:47.308 INFO:teuthology.orchestra.run.smithi183.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 195 kB 00:00 2024-04-26T01:28:47.350 INFO:teuthology.orchestra.run.smithi183.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-04-26T01:28:47.352 INFO:teuthology.orchestra.run.smithi191.stdout:(6/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 3.1 MB/s | 4.7 MB 00:01 2024-04-26T01:28:47.409 INFO:teuthology.orchestra.run.smithi183.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-26T01:28:47.442 INFO:teuthology.orchestra.run.smithi183.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-26T01:28:47.476 INFO:teuthology.orchestra.run.smithi183.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 4.0 MB/s | 135 kB 00:00 2024-04-26T01:28:47.510 INFO:teuthology.orchestra.run.smithi183.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 3.7 MB/s | 125 kB 00:00 2024-04-26T01:28:47.543 INFO:teuthology.orchestra.run.smithi183.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 6.3 MB/s | 215 kB 00:00 2024-04-26T01:28:47.585 INFO:teuthology.orchestra.run.smithi183.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 16 MB/s | 689 kB 00:00 2024-04-26T01:28:47.644 INFO:teuthology.orchestra.run.smithi183.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-26T01:28:48.061 INFO:teuthology.orchestra.run.smithi183.stdout:(49/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 11 MB/s | 49 MB 00:04 2024-04-26T01:28:48.086 INFO:teuthology.orchestra.run.smithi183.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 242 kB/s | 107 kB 00:00 2024-04-26T01:28:48.111 INFO:teuthology.orchestra.run.smithi183.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 433 kB/s | 21 kB 00:00 2024-04-26T01:28:48.136 INFO:teuthology.orchestra.run.smithi183.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-26T01:28:48.161 INFO:teuthology.orchestra.run.smithi183.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 662 kB/s | 33 kB 00:00 2024-04-26T01:28:48.168 INFO:teuthology.orchestra.run.smithi052.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 2.1 MB/s | 2.4 MB 00:01 2024-04-26T01:28:48.212 INFO:teuthology.orchestra.run.smithi183.stdout:(54/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 1.3 MB/s | 3.5 MB 00:02 2024-04-26T01:28:48.262 INFO:teuthology.orchestra.run.smithi183.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-04-26T01:28:48.279 INFO:teuthology.orchestra.run.smithi183.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 158 kB/s | 18 kB 00:00 2024-04-26T01:28:48.296 INFO:teuthology.orchestra.run.smithi183.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 138 kB/s | 11 kB 00:00 2024-04-26T01:28:48.322 INFO:teuthology.orchestra.run.smithi183.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-26T01:28:48.338 INFO:teuthology.orchestra.run.smithi183.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-04-26T01:28:48.371 INFO:teuthology.orchestra.run.smithi183.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-04-26T01:28:48.397 INFO:teuthology.orchestra.run.smithi183.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 10 MB/s | 582 kB 00:00 2024-04-26T01:28:48.414 INFO:teuthology.orchestra.run.smithi183.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 344 kB/s | 25 kB 00:00 2024-04-26T01:28:48.439 INFO:teuthology.orchestra.run.smithi183.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 719 kB/s | 47 kB 00:00 2024-04-26T01:28:48.473 INFO:teuthology.orchestra.run.smithi183.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-26T01:28:48.498 INFO:teuthology.orchestra.run.smithi183.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-26T01:28:48.523 INFO:teuthology.orchestra.run.smithi183.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 425 kB/s | 35 kB 00:00 2024-04-26T01:28:48.723 INFO:teuthology.orchestra.run.smithi183.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-04-26T01:28:48.765 INFO:teuthology.orchestra.run.smithi183.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-04-26T01:28:48.810 INFO:teuthology.orchestra.run.smithi052.stdout:(5/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 6.4 MB/s | 21 MB 00:03 2024-04-26T01:28:48.840 INFO:teuthology.orchestra.run.smithi183.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-26T01:28:48.874 INFO:teuthology.orchestra.run.smithi183.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 6.0 MB/s | 911 kB 00:00 2024-04-26T01:28:48.899 INFO:teuthology.orchestra.run.smithi183.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-26T01:28:48.927 INFO:teuthology.orchestra.run.smithi052.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-26T01:28:48.985 INFO:teuthology.orchestra.run.smithi052.stdout:(7/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-26T01:28:49.008 INFO:teuthology.orchestra.run.smithi183.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-04-26T01:28:49.125 INFO:teuthology.orchestra.run.smithi183.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-26T01:28:49.150 INFO:teuthology.orchestra.run.smithi183.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 350 kB/s | 87 kB 00:00 2024-04-26T01:28:49.186 INFO:teuthology.orchestra.run.smithi052.stdout:(8/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 13 MB/s | 4.7 MB 00:00 2024-04-26T01:28:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 629 kB/s | 25 kB 00:00 2024-04-26T01:28:49.244 INFO:teuthology.orchestra.run.smithi183.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-04-26T01:28:49.259 INFO:teuthology.orchestra.run.smithi183.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-26T01:28:49.266 INFO:teuthology.orchestra.run.smithi191.stdout:(7/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 970 kB/s | 5.4 MB 00:05 2024-04-26T01:28:49.284 INFO:teuthology.orchestra.run.smithi183.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-26T01:28:49.309 INFO:teuthology.orchestra.run.smithi183.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 388 kB/s | 26 kB 00:00 2024-04-26T01:28:49.335 INFO:teuthology.orchestra.run.smithi183.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-26T01:28:49.360 INFO:teuthology.orchestra.run.smithi183.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-26T01:28:49.395 INFO:teuthology.orchestra.run.smithi183.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 417 kB/s | 35 kB 00:00 2024-04-26T01:28:49.435 INFO:teuthology.orchestra.run.smithi183.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-26T01:28:49.503 INFO:teuthology.orchestra.run.smithi183.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-04-26T01:28:49.527 INFO:teuthology.orchestra.run.smithi183.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-26T01:28:49.552 INFO:teuthology.orchestra.run.smithi183.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 671 kB/s | 77 kB 00:00 2024-04-26T01:28:49.578 INFO:teuthology.orchestra.run.smithi183.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 846 kB/s | 42 kB 00:00 2024-04-26T01:28:49.603 INFO:teuthology.orchestra.run.smithi183.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-26T01:28:49.635 INFO:teuthology.orchestra.run.smithi191.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 3.5 MB/s | 21 MB 00:06 2024-04-26T01:28:49.636 INFO:teuthology.orchestra.run.smithi183.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-04-26T01:28:49.662 INFO:teuthology.orchestra.run.smithi183.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-26T01:28:49.687 INFO:teuthology.orchestra.run.smithi183.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 615 kB/s | 51 kB 00:00 2024-04-26T01:28:49.712 INFO:teuthology.orchestra.run.smithi183.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 718 kB/s | 54 kB 00:00 2024-04-26T01:28:49.752 INFO:teuthology.orchestra.run.smithi191.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 216 kB/s | 25 kB 00:00 2024-04-26T01:28:49.771 INFO:teuthology.orchestra.run.smithi183.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 499 kB/s | 42 kB 00:00 2024-04-26T01:28:49.871 INFO:teuthology.orchestra.run.smithi183.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 279 kB/s | 44 kB 00:00 2024-04-26T01:28:50.046 INFO:teuthology.orchestra.run.smithi183.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-26T01:28:50.071 INFO:teuthology.orchestra.run.smithi183.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 740 kB/s | 221 kB 00:00 2024-04-26T01:28:50.097 INFO:teuthology.orchestra.run.smithi183.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 161 kB/s | 36 kB 00:00 2024-04-26T01:28:50.122 INFO:teuthology.orchestra.run.smithi183.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 634 kB/s | 47 kB 00:00 2024-04-26T01:28:50.147 INFO:teuthology.orchestra.run.smithi183.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-26T01:28:50.173 INFO:teuthology.orchestra.run.smithi183.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-26T01:28:50.198 INFO:teuthology.orchestra.run.smithi183.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-26T01:28:50.223 INFO:teuthology.orchestra.run.smithi183.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T01:28:50.248 INFO:teuthology.orchestra.run.smithi183.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 597 kB/s | 45 kB 00:00 2024-04-26T01:28:50.273 INFO:teuthology.orchestra.run.smithi183.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-26T01:28:50.307 INFO:teuthology.orchestra.run.smithi183.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-26T01:28:50.699 INFO:teuthology.orchestra.run.smithi183.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-04-26T01:28:50.781 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:50.943 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:51.049 INFO:teuthology.orchestra.run.smithi183.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 685 kB/s | 548 kB 00:00 2024-04-26T01:28:51.132 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.47.0.53) 2024-04-26T01:28:51.215 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.47.0.53) 2024-04-26T01:28:51.219 INFO:teuthology.orchestra.run.smithi052.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 4.7 MB/s | 11 MB 00:02 2024-04-26T01:28:51.300 INFO:teuthology.orchestra.run.smithi183.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-04-26T01:28:51.325 INFO:teuthology.orchestra.run.smithi183.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 95 kB/s | 26 kB 00:00 2024-04-26T01:28:51.328 INFO:teuthology.orchestra.run.smithi052.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.0 MB/s | 225 kB 00:00 2024-04-26T01:28:51.420 INFO:teuthology.orchestra.run.smithi052.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 299 kB/s | 27 kB 00:00 2024-04-26T01:28:51.433 INFO:teuthology.orchestra.run.smithi183.stdout:(109/152): gperftools-libs-2.9.1-2.el9.x86_64.r 267 kB/s | 309 kB 00:01 2024-04-26T01:28:51.512 INFO:teuthology.orchestra.run.smithi052.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 350 kB/s | 32 kB 00:00 2024-04-26T01:28:51.550 INFO:teuthology.orchestra.run.smithi183.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-26T01:28:51.575 INFO:teuthology.orchestra.run.smithi183.stdout:(111/152): liboath-2.6.7-2.el9.x86_64.rpm 176 kB/s | 49 kB 00:00 2024-04-26T01:28:51.632 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:51.673 INFO:teuthology.orchestra.run.smithi052.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.6 MB/s | 747 kB 00:00 2024-04-26T01:28:51.694 INFO:teuthology.orchestra.run.smithi191.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.9 MB/s | 17 MB 00:04 2024-04-26T01:28:51.718 INFO:teuthology.orchestra.run.smithi183.stdout:(112/152): libunwind-1.6.2-1.el9.x86_64.rpm 172 kB/s | 67 kB 00:00 2024-04-26T01:28:51.718 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:51.771 INFO:teuthology.orchestra.run.smithi052.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 163 kB 00:00 2024-04-26T01:28:51.784 INFO:teuthology.orchestra.run.smithi183.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-26T01:28:51.826 INFO:teuthology.orchestra.run.smithi183.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-26T01:28:51.827 INFO:teuthology.orchestra.run.smithi191.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.6 MB/s | 225 kB 00:00 2024-04-26T01:28:51.851 INFO:teuthology.orchestra.run.smithi183.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-04-26T01:28:51.863 INFO:teuthology.orchestra.run.smithi052.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.4 MB/s | 126 kB 00:00 2024-04-26T01:28:51.903 INFO:teuthology.orchestra.run.smithi191.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 360 kB/s | 27 kB 00:00 2024-04-26T01:28:51.935 INFO:teuthology.orchestra.run.smithi183.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-26T01:28:51.969 INFO:teuthology.orchestra.run.smithi183.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 175 kB/s | 32 kB 00:00 2024-04-26T01:28:51.988 INFO:teuthology.orchestra.run.smithi052.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.9 MB/s | 500 kB 00:00 2024-04-26T01:28:51.995 INFO:teuthology.orchestra.run.smithi191.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 346 kB/s | 32 kB 00:00 2024-04-26T01:28:52.069 INFO:teuthology.orchestra.run.smithi183.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 791 kB/s | 172 kB 00:00 2024-04-26T01:28:52.094 INFO:teuthology.orchestra.run.smithi183.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-04-26T01:28:52.186 INFO:teuthology.orchestra.run.smithi183.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-04-26T01:28:52.220 INFO:teuthology.orchestra.run.smithi183.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-26T01:28:52.337 INFO:teuthology.orchestra.run.smithi183.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 825 kB/s | 303 kB 00:00 2024-04-26T01:28:52.362 INFO:teuthology.orchestra.run.smithi183.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-26T01:28:52.378 INFO:teuthology.orchestra.run.smithi191.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 1.9 MB/s | 747 kB 00:00 2024-04-26T01:28:52.413 INFO:teuthology.orchestra.run.smithi052.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 4.8 MB/s | 17 MB 00:03 2024-04-26T01:28:52.437 INFO:teuthology.orchestra.run.smithi183.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-26T01:28:52.462 INFO:teuthology.orchestra.run.smithi191.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.9 MB/s | 163 kB 00:00 2024-04-26T01:28:52.462 INFO:teuthology.orchestra.run.smithi183.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-26T01:28:52.513 INFO:teuthology.orchestra.run.smithi052.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 456 kB/s | 45 kB 00:00 2024-04-26T01:28:52.537 INFO:teuthology.orchestra.run.smithi191.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.7 MB/s | 126 kB 00:00 2024-04-26T01:28:52.538 INFO:teuthology.orchestra.run.smithi183.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-26T01:28:52.563 INFO:teuthology.orchestra.run.smithi183.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-26T01:28:52.622 INFO:teuthology.orchestra.run.smithi052.stdout:(20/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 8.5 MB/s | 5.4 MB 00:00 2024-04-26T01:28:52.646 INFO:teuthology.orchestra.run.smithi183.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-26T01:28:52.656 INFO:teuthology.orchestra.run.smithi052.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 953 kB/s | 135 kB 00:00 2024-04-26T01:28:52.698 INFO:teuthology.orchestra.run.smithi052.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.1 MB/s | 161 kB 00:00 2024-04-26T01:28:52.704 INFO:teuthology.orchestra.run.smithi191.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.9 MB/s | 500 kB 00:00 2024-04-26T01:28:52.747 INFO:teuthology.orchestra.run.smithi183.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.6 MB/s | 2.0 MB 00:00 2024-04-26T01:28:52.764 INFO:teuthology.orchestra.run.smithi183.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-04-26T01:28:52.790 INFO:teuthology.orchestra.run.smithi052.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 298 kB 00:00 2024-04-26T01:28:52.797 INFO:teuthology.orchestra.run.smithi183.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-04-26T01:28:52.823 INFO:teuthology.orchestra.run.smithi052.stdout:(24/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.9 MB/s | 320 kB 00:00 2024-04-26T01:28:52.864 INFO:teuthology.orchestra.run.smithi183.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-04-26T01:28:52.889 INFO:teuthology.orchestra.run.smithi183.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 460 kB/s | 58 kB 00:00 2024-04-26T01:28:52.890 INFO:teuthology.orchestra.run.smithi052.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.0 MB/s | 99 kB 00:00 2024-04-26T01:28:52.915 INFO:teuthology.orchestra.run.smithi183.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-26T01:28:52.915 INFO:teuthology.orchestra.run.smithi052.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 936 kB/s | 85 kB 00:00 2024-04-26T01:28:52.965 INFO:teuthology.orchestra.run.smithi183.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-26T01:28:53.007 INFO:teuthology.orchestra.run.smithi183.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 767 kB/s | 90 kB 00:00 2024-04-26T01:28:53.032 INFO:teuthology.orchestra.run.smithi183.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 240 kB/s | 28 kB 00:00 2024-04-26T01:28:53.041 INFO:teuthology.orchestra.run.smithi052.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.3 MB/s | 170 kB 00:00 2024-04-26T01:28:53.066 INFO:teuthology.orchestra.run.smithi183.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 305 kB/s | 31 kB 00:00 2024-04-26T01:28:53.141 INFO:teuthology.orchestra.run.smithi183.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-26T01:28:53.149 INFO:teuthology.orchestra.run.smithi052.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 319 kB/s | 34 kB 00:00 2024-04-26T01:28:53.166 INFO:teuthology.orchestra.run.smithi183.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-26T01:28:53.233 INFO:teuthology.orchestra.run.smithi052.stdout:(29/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 9.0 MB/s | 3.1 MB 00:00 2024-04-26T01:28:53.250 INFO:teuthology.orchestra.run.smithi183.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 195 kB/s | 36 kB 00:00 2024-04-26T01:28:53.258 INFO:teuthology.orchestra.run.smithi052.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-26T01:28:53.275 INFO:teuthology.orchestra.run.smithi183.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 381 kB/s | 51 kB 00:00 2024-04-26T01:28:53.317 INFO:teuthology.orchestra.run.smithi183.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-04-26T01:28:53.392 INFO:teuthology.orchestra.run.smithi183.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 631 kB/s | 90 kB 00:00 2024-04-26T01:28:53.418 INFO:teuthology.orchestra.run.smithi183.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-26T01:28:53.443 INFO:teuthology.orchestra.run.smithi183.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 159 kB/s | 20 kB 00:00 2024-04-26T01:28:53.527 INFO:teuthology.orchestra.run.smithi183.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-26T01:28:53.602 INFO:teuthology.orchestra.run.smithi183.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-26T01:28:53.661 INFO:teuthology.orchestra.run.smithi183.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-26T01:28:53.768 INFO:teuthology.orchestra.run.smithi052.stdout:(31/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 11 MB/s | 49 MB 00:04 2024-04-26T01:28:53.786 INFO:teuthology.orchestra.run.smithi183.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-04-26T01:28:53.901 INFO:teuthology.orchestra.run.smithi052.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 5.3 MB/s | 3.5 MB 00:00 2024-04-26T01:28:53.935 INFO:teuthology.orchestra.run.smithi052.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.5 MB/s | 249 kB 00:00 2024-04-26T01:28:53.961 INFO:teuthology.orchestra.run.smithi183.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 7.2 MB/s | 3.1 MB 00:00 2024-04-26T01:28:53.977 INFO:teuthology.orchestra.run.smithi052.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 664 kB/s | 49 kB 00:00 2024-04-26T01:28:54.002 INFO:teuthology.orchestra.run.smithi052.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 249 kB/s | 16 kB 00:00 2024-04-26T01:28:54.052 INFO:teuthology.orchestra.run.smithi052.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.6 MB/s | 273 kB 00:00 2024-04-26T01:28:54.055 INFO:teuthology.orchestra.run.smithi191.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.0 MB/s | 5.4 MB 00:01 2024-04-26T01:28:54.114 INFO:teuthology.orchestra.run.smithi191.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 776 kB/s | 45 kB 00:00 2024-04-26T01:28:54.127 INFO:teuthology.orchestra.run.smithi052.stdout:(37/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 6.0 MB/s | 762 kB 00:00 2024-04-26T01:28:54.144 INFO:teuthology.orchestra.run.smithi052.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 201 kB/s | 18 kB 00:00 2024-04-26T01:28:54.164 INFO:teuthology.orchestra.run.smithi191.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 2.6 MB/s | 135 kB 00:00 2024-04-26T01:28:54.214 INFO:teuthology.orchestra.run.smithi191.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 3.2 MB/s | 161 kB 00:00 2024-04-26T01:28:54.289 INFO:teuthology.orchestra.run.smithi191.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 4.2 MB/s | 320 kB 00:00 2024-04-26T01:28:54.356 INFO:teuthology.orchestra.run.smithi191.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 4.4 MB/s | 298 kB 00:00 2024-04-26T01:28:54.395 INFO:teuthology.orchestra.run.smithi052.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 302 kB/s | 80 kB 00:00 2024-04-26T01:28:54.398 INFO:teuthology.orchestra.run.smithi191.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 99 kB 00:00 2024-04-26T01:28:54.440 INFO:teuthology.orchestra.run.smithi191.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 2.0 MB/s | 85 kB 00:00 2024-04-26T01:28:54.520 INFO:teuthology.orchestra.run.smithi052.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-04-26T01:28:54.662 INFO:teuthology.orchestra.run.smithi052.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-26T01:28:54.687 INFO:teuthology.orchestra.run.smithi052.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-04-26T01:28:54.890 INFO:teuthology.orchestra.run.smithi191.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 6.9 MB/s | 3.1 MB 00:00 2024-04-26T01:28:54.941 INFO:teuthology.orchestra.run.smithi191.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 3.3 MB/s | 170 kB 00:00 2024-04-26T01:28:54.946 INFO:teuthology.orchestra.run.smithi052.stdout:(43/152): ceph-mgr-diskprediction-local-19.0.0- 4.4 MB/s | 7.4 MB 00:01 2024-04-26T01:28:54.983 INFO:teuthology.orchestra.run.smithi191.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 823 kB/s | 34 kB 00:00 2024-04-26T01:28:55.033 INFO:teuthology.orchestra.run.smithi191.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 2.8 MB/s | 142 kB 00:00 2024-04-26T01:28:55.061 INFO:teuthology.orchestra.run.smithi183.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 2.1 MB/s | 3.4 MB 00:01 2024-04-26T01:28:55.070 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:28:55.070 INFO:teuthology.orchestra.run.smithi183.stdout:Total 15 MB/s | 222 MB 00:14 2024-04-26T01:28:55.121 INFO:teuthology.orchestra.run.smithi052.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 2.7 MB/s | 1.2 MB 00:00 2024-04-26T01:28:55.172 INFO:teuthology.orchestra.run.smithi052.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 220 kB/s | 106 kB 00:00 2024-04-26T01:28:55.205 INFO:teuthology.orchestra.run.smithi052.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 530 kB/s | 135 kB 00:00 2024-04-26T01:28:55.230 INFO:teuthology.orchestra.run.smithi052.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-26T01:28:55.264 INFO:teuthology.orchestra.run.smithi052.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 215 kB 00:00 2024-04-26T01:28:55.331 INFO:teuthology.orchestra.run.smithi052.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 5.5 MB/s | 689 kB 00:00 2024-04-26T01:28:55.423 INFO:teuthology.orchestra.run.smithi052.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 943 kB 00:00 2024-04-26T01:28:55.448 INFO:teuthology.orchestra.run.smithi052.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 594 kB/s | 107 kB 00:00 2024-04-26T01:28:55.473 INFO:teuthology.orchestra.run.smithi052.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 157 kB/s | 21 kB 00:00 2024-04-26T01:28:55.500 INFO:teuthology.orchestra.run.smithi191.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 7.6 MB/s | 3.5 MB 00:00 2024-04-26T01:28:55.507 INFO:teuthology.orchestra.run.smithi052.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.8 MB/s | 313 kB 00:00 2024-04-26T01:28:55.540 INFO:teuthology.orchestra.run.smithi052.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 520 kB/s | 33 kB 00:00 2024-04-26T01:28:55.716 INFO:teuthology.orchestra.run.smithi052.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-26T01:28:55.858 INFO:teuthology.orchestra.run.smithi052.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 53 kB/s | 18 kB 00:00 2024-04-26T01:28:55.876 INFO:teuthology.orchestra.run.smithi191.stdout:(31/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 1.6 MB/s | 11 MB 00:06 2024-04-26T01:28:55.958 INFO:teuthology.orchestra.run.smithi052.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 27 kB/s | 11 kB 00:00 2024-04-26T01:28:55.976 INFO:teuthology.orchestra.run.smithi191.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.4 MB/s | 249 kB 00:00 2024-04-26T01:28:56.000 INFO:teuthology.orchestra.run.smithi052.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.9 MB/s | 160 kB 00:00 2024-04-26T01:28:56.034 INFO:teuthology.orchestra.run.smithi191.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 847 kB/s | 49 kB 00:00 2024-04-26T01:28:56.050 INFO:teuthology.orchestra.run.smithi052.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 500 kB/s | 45 kB 00:00 2024-04-26T01:28:56.084 INFO:teuthology.orchestra.run.smithi052.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-26T01:28:56.118 INFO:teuthology.orchestra.run.smithi191.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 197 kB/s | 16 kB 00:00 2024-04-26T01:28:56.126 INFO:teuthology.orchestra.run.smithi052.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 582 kB 00:00 2024-04-26T01:28:56.151 INFO:teuthology.orchestra.run.smithi052.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 420 kB/s | 25 kB 00:00 2024-04-26T01:28:56.152 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-04-26T01:28:56.176 INFO:teuthology.orchestra.run.smithi052.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 555 kB/s | 47 kB 00:00 2024-04-26T01:28:56.210 INFO:teuthology.orchestra.run.smithi052.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.5 MB/s | 292 kB 00:00 2024-04-26T01:28:56.226 INFO:teuthology.orchestra.run.smithi191.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.5 MB/s | 273 kB 00:00 2024-04-26T01:28:56.243 INFO:teuthology.orchestra.run.smithi052.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 188 kB 00:00 2024-04-26T01:28:56.269 INFO:teuthology.orchestra.run.smithi052.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 400 kB/s | 35 kB 00:00 2024-04-26T01:28:56.295 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-04-26T01:28:56.296 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-04-26T01:28:56.411 INFO:teuthology.orchestra.run.smithi052.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-26T01:28:56.518 INFO:teuthology.orchestra.run.smithi191.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 7.2 MB/s | 7.4 MB 00:01 2024-04-26T01:28:56.519 INFO:teuthology.orchestra.run.smithi052.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 6.9 MB/s | 1.6 MB 00:00 2024-04-26T01:28:56.552 INFO:teuthology.orchestra.run.smithi191.stdout:(37/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.3 MB/s | 762 kB 00:00 2024-04-26T01:28:56.569 INFO:teuthology.orchestra.run.smithi191.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 364 kB/s | 18 kB 00:00 2024-04-26T01:28:56.603 INFO:teuthology.orchestra.run.smithi052.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T01:28:56.645 INFO:teuthology.orchestra.run.smithi052.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 4.1 MB/s | 911 kB 00:00 2024-04-26T01:28:56.670 INFO:teuthology.orchestra.run.smithi052.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 935 kB/s | 133 kB 00:00 2024-04-26T01:28:56.703 INFO:teuthology.orchestra.run.smithi191.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-04-26T01:28:56.853 INFO:teuthology.orchestra.run.smithi191.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-26T01:28:56.879 INFO:teuthology.orchestra.run.smithi052.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.4 MB/s | 1.0 MB 00:00 2024-04-26T01:28:56.912 INFO:teuthology.orchestra.run.smithi191.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.3 MB/s | 253 kB 00:00 2024-04-26T01:28:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-26T01:28:57.004 INFO:teuthology.orchestra.run.smithi052.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-04-26T01:28:57.029 INFO:teuthology.orchestra.run.smithi052.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 253 kB/s | 87 kB 00:00 2024-04-26T01:28:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T01:28:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:(44/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-04-26T01:28:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-26T01:28:57.155 INFO:teuthology.orchestra.run.smithi191.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-26T01:28:57.180 INFO:teuthology.orchestra.run.smithi191.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-04-26T01:28:57.205 INFO:teuthology.orchestra.run.smithi191.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-26T01:28:57.247 INFO:teuthology.orchestra.run.smithi191.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-26T01:28:57.271 INFO:teuthology.orchestra.run.smithi052.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-26T01:28:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-26T01:28:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-26T01:28:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-26T01:28:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 565 kB/s | 33 kB 00:00 2024-04-26T01:28:57.382 INFO:teuthology.orchestra.run.smithi052.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 728 kB/s | 244 kB 00:00 2024-04-26T01:28:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 380 kB/s | 18 kB 00:00 2024-04-26T01:28:57.438 INFO:teuthology.orchestra.run.smithi052.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 634 kB/s | 248 kB 00:00 2024-04-26T01:28:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-04-26T01:28:57.471 INFO:teuthology.orchestra.run.smithi052.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 808 kB/s | 26 kB 00:00 2024-04-26T01:28:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 135 kB/s | 11 kB 00:00 2024-04-26T01:28:57.496 INFO:teuthology.orchestra.run.smithi052.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 339 kB/s | 19 kB 00:00 2024-04-26T01:28:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-26T01:28:57.530 INFO:teuthology.orchestra.run.smithi052.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-04-26T01:28:57.541 INFO:teuthology.orchestra.run.smithi191.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-04-26T01:28:57.572 INFO:teuthology.orchestra.run.smithi052.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 479 kB/s | 35 kB 00:00 2024-04-26T01:28:57.609 INFO:teuthology.orchestra.run.smithi191.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 662 kB 00:00 2024-04-26T01:28:57.634 INFO:teuthology.orchestra.run.smithi191.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-26T01:28:57.659 INFO:teuthology.orchestra.run.smithi191.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 515 kB/s | 25 kB 00:00 2024-04-26T01:28:57.685 INFO:teuthology.orchestra.run.smithi191.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-26T01:28:57.706 INFO:teuthology.orchestra.run.smithi052.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-26T01:28:57.710 INFO:teuthology.orchestra.run.smithi191.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-26T01:28:57.731 INFO:teuthology.orchestra.run.smithi052.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-26T01:28:57.735 INFO:teuthology.orchestra.run.smithi191.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-26T01:28:57.756 INFO:teuthology.orchestra.run.smithi052.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 222 kB 00:00 2024-04-26T01:28:57.761 INFO:teuthology.orchestra.run.smithi191.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-26T01:28:57.806 INFO:teuthology.orchestra.run.smithi052.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T01:28:57.849 INFO:teuthology.orchestra.run.smithi052.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 462 kB/s | 42 kB 00:00 2024-04-26T01:28:57.882 INFO:teuthology.orchestra.run.smithi052.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-26T01:28:57.915 INFO:teuthology.orchestra.run.smithi052.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-26T01:28:57.936 INFO:teuthology.orchestra.run.smithi191.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-26T01:28:57.949 INFO:teuthology.orchestra.run.smithi052.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.2 MB/s | 279 kB 00:00 2024-04-26T01:28:57.974 INFO:teuthology.orchestra.run.smithi052.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 572 kB/s | 51 kB 00:00 2024-04-26T01:28:58.008 INFO:teuthology.orchestra.run.smithi052.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 610 kB/s | 54 kB 00:00 2024-04-26T01:28:58.011 INFO:teuthology.orchestra.run.smithi191.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-26T01:28:58.053 INFO:teuthology.orchestra.run.smithi191.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-26T01:28:58.061 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-04-26T01:28:58.062 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-04-26T01:28:58.075 INFO:teuthology.orchestra.run.smithi052.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 423 kB/s | 42 kB 00:00 2024-04-26T01:28:58.095 INFO:teuthology.orchestra.run.smithi191.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-26T01:28:58.121 INFO:teuthology.orchestra.run.smithi191.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-26T01:28:58.246 INFO:teuthology.orchestra.run.smithi191.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-26T01:28:58.359 INFO:teuthology.orchestra.run.smithi052.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 134 kB/s | 44 kB 00:00 2024-04-26T01:28:58.388 INFO:teuthology.orchestra.run.smithi191.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-26T01:28:58.413 INFO:teuthology.orchestra.run.smithi191.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 530 kB/s | 87 kB 00:00 2024-04-26T01:28:58.472 INFO:teuthology.orchestra.run.smithi191.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-26T01:28:58.547 INFO:teuthology.orchestra.run.smithi191.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.3 MB/s | 248 kB 00:00 2024-04-26T01:28:58.584 INFO:teuthology.orchestra.run.smithi052.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-26T01:28:58.609 INFO:teuthology.orchestra.run.smithi052.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 429 kB/s | 221 kB 00:00 2024-04-26T01:28:58.634 INFO:teuthology.orchestra.run.smithi052.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 138 kB/s | 36 kB 00:00 2024-04-26T01:28:58.668 INFO:teuthology.orchestra.run.smithi052.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 830 kB/s | 47 kB 00:00 2024-04-26T01:28:58.693 INFO:teuthology.orchestra.run.smithi052.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-26T01:28:58.752 INFO:teuthology.orchestra.run.smithi052.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 513 kB/s | 58 kB 00:00 2024-04-26T01:28:58.755 INFO:teuthology.orchestra.run.smithi191.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T01:28:58.777 INFO:teuthology.orchestra.run.smithi052.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-26T01:28:58.802 INFO:teuthology.orchestra.run.smithi052.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 597 kB/s | 64 kB 00:00 2024-04-26T01:28:58.806 INFO:teuthology.orchestra.run.smithi191.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 100 kB/s | 26 kB 00:00 2024-04-26T01:28:58.823 INFO:teuthology.orchestra.run.smithi191.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-04-26T01:28:58.827 INFO:teuthology.orchestra.run.smithi052.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 613 kB/s | 45 kB 00:00 2024-04-26T01:28:58.862 INFO:teuthology.orchestra.run.smithi052.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 388 kB/s | 22 kB 00:00 2024-04-26T01:28:58.865 INFO:teuthology.orchestra.run.smithi191.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T01:28:58.882 INFO:teuthology.orchestra.run.smithi191.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-04-26T01:28:58.912 INFO:teuthology.orchestra.run.smithi052.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-04-26T01:28:58.949 INFO:teuthology.orchestra.run.smithi191.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-26T01:28:59.062 INFO:teuthology.orchestra.run.smithi052.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-04-26T01:28:59.076 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.082 INFO:teuthology.orchestra.run.smithi191.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-04-26T01:28:59.108 INFO:teuthology.orchestra.run.smithi191.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-04-26T01:28:59.133 INFO:teuthology.orchestra.run.smithi191.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T01:28:59.150 INFO:teuthology.orchestra.run.smithi191.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-26T01:28:59.175 INFO:teuthology.orchestra.run.smithi191.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-04-26T01:28:59.195 INFO:teuthology.orchestra.run.smithi052.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 943 kB/s | 309 kB 00:00 2024-04-26T01:28:59.201 INFO:teuthology.orchestra.run.smithi191.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T01:28:59.226 INFO:teuthology.orchestra.run.smithi191.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-26T01:28:59.229 INFO:teuthology.orchestra.run.smithi052.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-26T01:28:59.231 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.231 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.243 INFO:teuthology.orchestra.run.smithi191.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-26T01:28:59.260 INFO:teuthology.orchestra.run.smithi191.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-26T01:28:59.271 INFO:teuthology.orchestra.run.smithi052.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-04-26T01:28:59.273 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.302 INFO:teuthology.orchestra.run.smithi191.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-26T01:28:59.321 INFO:teuthology.orchestra.run.smithi052.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-26T01:28:59.323 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.352 INFO:teuthology.orchestra.run.smithi191.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 912 kB/s | 44 kB 00:00 2024-04-26T01:28:59.371 INFO:teuthology.orchestra.run.smithi052.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-04-26T01:28:59.403 INFO:teuthology.orchestra.run.smithi191.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.6 MB/s | 221 kB 00:00 2024-04-26T01:28:59.445 INFO:teuthology.orchestra.run.smithi191.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 915 kB/s | 36 kB 00:00 2024-04-26T01:28:59.445 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.457 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.468 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://forksystems.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.147.64.7) 2024-04-26T01:28:59.487 INFO:teuthology.orchestra.run.smithi191.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2024-04-26T01:28:59.529 INFO:teuthology.orchestra.run.smithi191.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 7.5 MB/s | 305 kB 00:00 2024-04-26T01:28:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.542 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.578 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.607 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://volico.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 208.68.237.5) 2024-04-26T01:28:59.679 INFO:teuthology.orchestra.run.smithi191.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-26T01:28:59.704 INFO:teuthology.orchestra.run.smithi191.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 330 kB/s | 58 kB 00:00 2024-04-26T01:28:59.730 INFO:teuthology.orchestra.run.smithi191.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-26T01:28:59.755 INFO:teuthology.orchestra.run.smithi191.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T01:28:59.782 INFO:teuthology.orchestra.run.smithi191.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-04-26T01:28:59.805 INFO:teuthology.orchestra.run.smithi191.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-04-26T01:28:59.847 INFO:teuthology.orchestra.run.smithi191.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-26T01:28:59.877 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:59.913 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:28:59.957 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.47.0.53) 2024-04-26T01:29:00.043 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:29:00.079 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 50.47.0.53) 2024-04-26T01:29:00.223 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://ziply.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 50.47.0.53) 2024-04-26T01:29:00.334 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.382 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for https://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.418 INFO:teuthology.orchestra.run.smithi191.stdout:(104/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 4.6 MB/s | 49 MB 00:10 2024-04-26T01:29:00.433 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for https://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.434 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.483 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] parquet-libs-9.0.0-10.el9.x86_64.rpm: Status code: 404 for http://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/parquet-libs-9.0.0-10.el9.x86_64.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.484 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Status code: 404 for http://ix-denver.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm (IP: 23.144.216.11) 2024-04-26T01:29:00.506 INFO:teuthology.orchestra.run.smithi191.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 213 kB/s | 19 kB 00:00 2024-04-26T01:29:00.573 INFO:teuthology.orchestra.run.smithi191.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 426 kB/s | 309 kB 00:00 2024-04-26T01:29:00.623 INFO:teuthology.orchestra.run.smithi191.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-04-26T01:29:00.655 INFO:teuthology.orchestra.run.smithi052.stdout:(111/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 18 kB/s | 26 kB 00:01 2024-04-26T01:29:00.682 INFO:teuthology.orchestra.run.smithi191.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-04-26T01:29:00.716 INFO:teuthology.orchestra.run.smithi191.stdout:(109/152): abseil-cpp-20211102.0-3.el9.x86_64.r 603 kB/s | 548 kB 00:00 2024-04-26T01:29:00.749 INFO:teuthology.orchestra.run.smithi191.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-26T01:29:00.816 INFO:teuthology.orchestra.run.smithi191.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-26T01:29:00.822 INFO:teuthology.orchestra.run.smithi052.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-26T01:29:00.873 INFO:teuthology.orchestra.run.smithi052.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 899 kB/s | 43 kB 00:00 2024-04-26T01:29:00.939 INFO:teuthology.orchestra.run.smithi052.stdout:(114/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 561 kB/s | 838 kB 00:01 2024-04-26T01:29:00.965 INFO:teuthology.orchestra.run.smithi052.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 357 kB/s | 32 kB 00:00 2024-04-26T01:29:01.007 INFO:teuthology.orchestra.run.smithi052.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 4.1 MB/s | 172 kB 00:00 2024-04-26T01:29:01.008 INFO:teuthology.orchestra.run.smithi191.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-26T01:29:01.032 INFO:teuthology.orchestra.run.smithi052.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-04-26T01:29:01.050 INFO:teuthology.orchestra.run.smithi191.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-26T01:29:01.082 INFO:teuthology.orchestra.run.smithi052.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-26T01:29:01.101 INFO:teuthology.orchestra.run.smithi191.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 644 kB/s | 32 kB 00:00 2024-04-26T01:29:01.124 INFO:teuthology.orchestra.run.smithi052.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 866 kB/s | 35 kB 00:00 2024-04-26T01:29:01.126 INFO:teuthology.orchestra.run.smithi191.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-04-26T01:29:01.143 INFO:teuthology.orchestra.run.smithi191.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-04-26T01:29:01.183 INFO:teuthology.orchestra.run.smithi052.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 3.5 MB/s | 206 kB 00:00 2024-04-26T01:29:01.201 INFO:teuthology.orchestra.run.smithi191.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T01:29:01.216 INFO:teuthology.orchestra.run.smithi052.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-04-26T01:29:01.294 INFO:teuthology.orchestra.run.smithi191.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-26T01:29:01.318 INFO:teuthology.orchestra.run.smithi052.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-04-26T01:29:01.336 INFO:teuthology.orchestra.run.smithi191.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 852 kB/s | 35 kB 00:00 2024-04-26T01:29:01.365 INFO:teuthology.orchestra.run.smithi052.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 980 kB/s | 144 kB 00:00 2024-04-26T01:29:01.369 INFO:teuthology.orchestra.run.smithi191.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-04-26T01:29:01.394 INFO:teuthology.orchestra.run.smithi052.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-04-26T01:29:01.419 INFO:teuthology.orchestra.run.smithi052.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 329 kB/s | 18 kB 00:00 2024-04-26T01:29:01.444 INFO:teuthology.orchestra.run.smithi191.stdout:(121/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-04-26T01:29:01.445 INFO:teuthology.orchestra.run.smithi052.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 469 kB/s | 23 kB 00:00 2024-04-26T01:29:01.470 INFO:teuthology.orchestra.run.smithi191.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-04-26T01:29:01.471 INFO:teuthology.orchestra.run.smithi052.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2024-04-26T01:29:01.495 INFO:teuthology.orchestra.run.smithi052.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 397 kB/s | 20 kB 00:00 2024-04-26T01:29:01.512 INFO:teuthology.orchestra.run.smithi191.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-26T01:29:01.537 INFO:teuthology.orchestra.run.smithi191.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-26T01:29:01.562 INFO:teuthology.orchestra.run.smithi191.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 354 kB/s | 18 kB 00:00 2024-04-26T01:29:01.587 INFO:teuthology.orchestra.run.smithi052.stdout:(129/152): libarrow-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-04-26T01:29:01.587 INFO:teuthology.orchestra.run.smithi191.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-26T01:29:01.612 INFO:teuthology.orchestra.run.smithi052.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 400 kB/s | 46 kB 00:00 2024-04-26T01:29:01.613 INFO:teuthology.orchestra.run.smithi191.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-26T01:29:01.638 INFO:teuthology.orchestra.run.smithi191.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-26T01:29:01.663 INFO:teuthology.orchestra.run.smithi052.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-26T01:29:01.680 INFO:teuthology.orchestra.run.smithi191.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-26T01:29:01.688 INFO:teuthology.orchestra.run.smithi052.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 772 kB/s | 58 kB 00:00 2024-04-26T01:29:01.713 INFO:teuthology.orchestra.run.smithi052.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 630 kB/s | 79 kB 00:00 2024-04-26T01:29:01.722 INFO:teuthology.orchestra.run.smithi191.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-26T01:29:01.739 INFO:teuthology.orchestra.run.smithi052.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 332 kB/s | 16 kB 00:00 2024-04-26T01:29:01.756 INFO:teuthology.orchestra.run.smithi052.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-26T01:29:01.764 INFO:teuthology.orchestra.run.smithi191.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-26T01:29:01.809 INFO:teuthology.orchestra.run.smithi052.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-26T01:29:01.840 INFO:teuthology.orchestra.run.smithi052.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-04-26T01:29:01.848 INFO:teuthology.orchestra.run.smithi191.stdout:(132/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-26T01:29:01.864 INFO:teuthology.orchestra.run.smithi052.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-04-26T01:29:01.890 INFO:teuthology.orchestra.run.smithi191.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 394 kB/s | 16 kB 00:00 2024-04-26T01:29:01.898 INFO:teuthology.orchestra.run.smithi052.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.1 MB/s | 188 kB 00:00 2024-04-26T01:29:01.924 INFO:teuthology.orchestra.run.smithi052.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 624 kB/s | 36 kB 00:00 2024-04-26T01:29:01.932 INFO:teuthology.orchestra.run.smithi191.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-26T01:29:01.949 INFO:teuthology.orchestra.run.smithi052.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 715 kB/s | 59 kB 00:00 2024-04-26T01:29:01.974 INFO:teuthology.orchestra.run.smithi052.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-04-26T01:29:01.983 INFO:teuthology.orchestra.run.smithi191.stdout:(135/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-04-26T01:29:01.999 INFO:teuthology.orchestra.run.smithi052.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-26T01:29:02.007 INFO:teuthology.orchestra.run.smithi191.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-04-26T01:29:02.032 INFO:teuthology.orchestra.run.smithi191.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-04-26T01:29:02.041 INFO:teuthology.orchestra.run.smithi052.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-26T01:29:02.066 INFO:teuthology.orchestra.run.smithi191.stdout:(138/152): python3-kubernetes-26.1.0-2.el9.noar 2.3 MB/s | 1.0 MB 00:00 2024-04-26T01:29:02.067 INFO:teuthology.orchestra.run.smithi052.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 303 kB/s | 20 kB 00:00 2024-04-26T01:29:02.091 INFO:teuthology.orchestra.run.smithi191.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-26T01:29:02.108 INFO:teuthology.orchestra.run.smithi052.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-26T01:29:02.117 INFO:teuthology.orchestra.run.smithi191.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-04-26T01:29:02.142 INFO:teuthology.orchestra.run.smithi191.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-04-26T01:29:02.142 INFO:teuthology.orchestra.run.smithi052.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-26T01:29:02.167 INFO:teuthology.orchestra.run.smithi191.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-26T01:29:02.193 INFO:teuthology.orchestra.run.smithi191.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T01:29:02.218 INFO:teuthology.orchestra.run.smithi191.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T01:29:02.243 INFO:teuthology.orchestra.run.smithi191.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-26T01:29:02.302 INFO:teuthology.orchestra.run.smithi191.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-26T01:29:02.336 INFO:teuthology.orchestra.run.smithi191.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-04-26T01:29:02.351 INFO:teuthology.orchestra.run.smithi052.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-26T01:29:02.561 INFO:teuthology.orchestra.run.smithi191.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-26T01:29:02.653 INFO:teuthology.orchestra.run.smithi191.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-04-26T01:29:02.690 INFO:teuthology.orchestra.run.smithi052.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 1.3 MB 00:00 2024-04-26T01:29:02.753 INFO:teuthology.orchestra.run.smithi191.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-26T01:29:02.793 INFO:teuthology.orchestra.run.smithi052.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-26T01:29:02.886 INFO:teuthology.orchestra.run.smithi191.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 5.8 MB/s | 3.4 MB 00:00 2024-04-26T01:29:02.910 INFO:teuthology.orchestra.run.smithi052.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 4.2 MB/s | 3.4 MB 00:00 2024-04-26T01:29:02.993 INFO:teuthology.orchestra.run.smithi052.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 4.0 MB/s | 3.1 MB 00:00 2024-04-26T01:29:03.002 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:29:03.002 INFO:teuthology.orchestra.run.smithi052.stdout:Total 12 MB/s | 222 MB 00:18 2024-04-26T01:29:03.370 INFO:teuthology.orchestra.run.smithi191.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 3.0 MB/s | 3.1 MB 00:01 2024-04-26T01:29:03.375 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:29:03.375 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-26T01:29:04.205 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-04-26T01:29:04.298 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:29:04.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:29:04.389 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:29:04.423 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:29:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:29:04.470 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-04-26T01:29:04.491 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:29:04.615 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-04-26T01:29:04.615 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-04-26T01:29:04.812 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:29:04.908 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:05.031 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:05.068 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:05.159 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:05.197 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:05.342 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:29:05.393 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:29:05.422 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:05.576 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:05.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:05.660 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:05.683 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:05.705 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:05.907 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:05.932 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-04-26T01:29:05.948 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:06.058 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:29:06.153 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:29:06.190 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:29:06.194 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-04-26T01:29:06.194 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-04-26T01:29:06.225 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:29:06.298 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:29:06.331 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:29:06.376 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:29:06.378 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-04-26T01:29:06.378 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-04-26T01:29:06.408 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:29:06.436 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:29:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:29:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:29:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:29:06.640 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:29:06.717 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:29:06.751 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:29:06.786 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:29:06.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:29:06.884 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:29:06.996 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:29:07.075 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:29:07.212 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:29:07.264 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:29:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:29:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:29:07.898 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:29:07.940 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:29:07.978 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:29:08.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:29:08.188 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:29:08.240 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:29:08.294 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:29:08.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:29:08.398 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:29:08.435 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:29:08.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:29:08.517 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:29:08.561 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:29:08.611 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:29:08.746 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:29:08.799 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:29:08.830 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:29:08.871 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:29:08.965 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:29:09.030 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:29:09.070 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:29:09.074 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-04-26T01:29:09.074 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-04-26T01:29:09.095 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:29:09.120 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:29:09.195 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:29:09.321 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:29:09.361 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:29:09.413 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:29:09.448 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:29:09.487 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:29:09.518 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:29:09.549 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:29:09.730 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:29:09.834 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:29:09.876 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:29:09.910 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:29:10.001 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:29:10.058 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:29:10.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:29:10.234 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:29:10.288 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:29:11.002 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:29:11.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:29:11.077 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:29:11.187 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:29:11.235 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:29:11.322 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:29:11.379 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:29:11.558 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:29:11.586 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:29:11.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:29:12.308 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:29:12.416 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:29:12.477 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-04-26T01:29:12.582 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:29:12.620 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:29:12.673 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:29:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:29:12.750 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:29:12.792 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:29:13.111 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:29:13.235 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:13.348 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:13.377 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:13.469 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:13.498 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:13.646 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:29:13.654 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:29:13.704 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:29:13.732 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:29:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:29:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:29:13.835 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:13.858 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:13.921 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:13.942 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:13.986 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:14.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:29:14.146 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:14.185 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:14.218 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:29:14.304 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:29:14.328 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:29:14.422 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:29:14.459 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:29:14.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:29:14.567 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:29:14.609 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:29:14.654 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:29:14.686 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:29:14.713 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:29:14.783 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:29:14.784 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:29:14.807 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:14.836 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:29:14.862 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:14.866 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:29:14.884 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:29:14.902 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:29:14.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:29:15.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:29:15.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:29:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:29:15.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:29:15.208 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:29:15.282 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:29:15.425 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:29:15.476 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:29:15.554 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-04-26T01:29:15.660 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T01:29:15.826 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T01:29:15.998 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:29:16.064 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T01:29:16.136 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T01:29:16.179 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T01:29:16.190 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:29:16.227 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:29:16.270 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:29:16.308 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:29:16.313 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T01:29:16.350 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:29:16.495 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:29:16.569 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:29:16.624 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:29:16.697 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:29:16.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:29:16.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:29:16.783 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T01:29:16.821 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:29:16.856 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:29:16.875 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:16.899 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:29:16.941 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:29:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:17.076 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:29:17.084 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:17.122 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:29:17.152 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:29:17.193 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:29:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:17.216 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:17.246 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:17.250 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:17.287 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:29:17.328 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:29:17.367 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:29:17.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:29:17.426 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:29:17.500 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:29:17.602 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T01:29:17.627 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:29:17.660 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:29:17.680 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T01:29:17.686 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:29:17.705 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:17.721 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:29:17.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:29:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:17.792 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:29:17.822 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:29:17.849 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:29:17.850 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:17.873 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:17.881 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:29:17.906 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:17.923 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:29:17.940 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:17.940 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:29:17.941 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:29:17.941 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:17.957 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:29:17.959 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:18.013 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-26T01:29:18.044 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:29:18.073 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:18.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:29:18.145 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T01:29:18.153 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:29:18.249 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:29:18.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:29:18.405 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:29:18.478 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:18.550 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T01:29:18.755 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T01:29:18.895 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-26T01:29:18.967 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T01:29:19.019 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T01:29:19.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:29:19.131 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T01:29:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:29:19.179 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T01:29:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:29:19.245 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T01:29:19.293 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T01:29:19.300 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:29:19.332 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T01:29:19.360 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:29:19.447 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:29:19.454 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T01:29:19.498 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:29:19.533 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T01:29:19.576 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T01:29:19.648 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T01:29:19.673 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:29:19.711 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:29:19.767 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T01:29:19.788 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:29:19.804 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-26T01:29:19.872 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-26T01:29:19.911 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-26T01:29:20.039 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T01:29:20.098 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T01:29:20.210 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T01:29:20.429 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:29:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T01:29:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T01:29:20.541 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:29:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T01:29:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T01:29:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T01:29:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T01:29:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-26T01:29:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T01:29:21.759 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:29:21.841 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:29:21.868 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:29:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T01:29:21.902 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:29:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T01:29:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T01:29:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T01:29:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T01:29:22.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:29:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T01:29:22.256 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T01:29:22.347 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T01:29:22.378 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:29:22.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:29:22.414 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T01:29:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T01:29:22.721 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T01:29:22.772 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T01:29:22.811 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T01:29:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T01:29:22.889 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:29:22.918 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:22.998 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:23.020 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:29:23.044 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T01:29:23.085 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T01:29:23.136 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T01:29:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T01:29:23.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-26T01:29:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T01:29:23.606 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T01:29:23.647 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T01:29:23.692 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T01:29:23.734 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T01:29:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T01:29:23.897 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T01:29:23.962 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T01:29:24.018 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T01:29:24.066 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T01:29:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T01:29:24.187 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T01:29:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T01:29:24.441 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T01:29:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T01:29:24.555 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T01:29:24.709 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T01:29:24.783 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T01:29:25.272 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:25.316 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:25.782 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T01:29:25.841 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T01:29:25.864 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:25.874 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T01:29:25.903 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:25.996 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:29:25.996 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:29:25.996 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:26.008 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T01:29:26.030 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:26.078 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T01:29:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T01:29:26.224 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T01:29:26.480 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T01:29:26.513 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T01:29:26.623 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T01:29:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T01:29:28.062 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T01:29:29.096 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:29.096 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-04-26T01:29:29.096 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-04-26T01:29:29.096 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-04-26T01:29:29.096 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-04-26T01:29:29.097 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-04-26T01:29:29.097 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-04-26T01:29:29.097 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-04-26T01:29:29.097 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:29.263 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:29.279 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:29.279 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:29:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:29.815 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T01:29:29.912 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T01:29:29.964 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T01:29:30.016 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T01:29:30.451 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T01:29:30.509 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:30.624 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-26T01:29:30.656 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-26T01:29:30.713 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T01:29:30.766 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:30.791 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:30.791 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:30.792 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:29:30.792 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:30.792 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:30.792 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:31.016 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:31.038 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:31.039 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:31.039 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:29:31.039 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:31.039 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:31.039 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:31.306 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-26T01:29:31.343 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:31.461 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-26T01:29:31.483 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:31.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:31.506 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:31.622 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:29:31.679 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:29:31.734 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:29:31.917 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:29:31.967 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:29:32.045 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:29:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:29:32.266 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:29:32.318 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:29:32.349 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:29:32.377 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:29:32.855 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:32.875 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:32.899 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:29:32.960 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:29:33.006 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:34.329 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:34.368 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:35.042 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-26T01:29:35.048 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:35.084 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:35.088 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:35.185 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:35.255 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-26T01:29:35.255 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T01:29:35.255 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T01:29:35.255 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:35.326 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:35.349 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:29:35.378 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:35.401 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:35.445 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:35.456 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:29:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:29:36.543 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:29:36.589 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:36.612 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:36.612 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:36.612 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:29:36.613 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:36.613 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:36.613 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:36.656 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:36.678 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:36.678 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:36.678 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:29:36.679 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:36.724 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:36.746 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:36.746 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:36.746 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:29:36.746 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:36.747 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:36.747 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:37.050 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:37.072 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:37.072 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:37.072 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:29:37.072 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:37.072 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:37.073 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:37.708 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:37.708 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-04-26T01:29:37.708 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-04-26T01:29:37.708 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-04-26T01:29:37.709 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-04-26T01:29:37.709 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-04-26T01:29:37.709 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-04-26T01:29:37.709 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-04-26T01:29:37.709 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:37.886 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:37.923 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:29:37.923 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:39.423 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:39.448 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:39.682 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:39.704 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:40.149 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:40.171 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:40.171 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:40.171 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:29:40.171 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:40.172 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:40.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:40.271 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:29:40.320 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:29:40.375 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:29:40.559 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:29:40.625 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:29:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:29:40.791 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:29:40.922 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:29:40.958 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:29:40.991 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:29:41.019 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:29:41.479 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:41.501 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:41.524 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:29:41.577 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:29:41.607 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:41.669 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:29:41.719 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:29:41.768 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:29:41.825 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:29:41.853 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:29:42.035 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:29:42.073 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:29:42.103 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:29:42.644 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:42.684 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:42.787 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:43.607 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:29:43.654 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:43.684 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:29:43.738 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:29:43.787 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:43.819 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:29:43.851 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:29:43.851 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:29:43.939 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:29:43.948 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:29:43.960 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:29:43.960 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:29:43.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:44.001 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:44.001 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:44.001 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:29:44.002 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:44.002 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:44.002 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:44.045 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:44.051 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:29:44.056 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:44.124 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:29:44.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:29:44.211 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:29:45.141 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:29:45.173 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:45.196 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:45.196 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:45.196 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:29:45.196 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:45.197 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:45.197 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:45.241 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:45.263 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:45.264 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:45.264 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:29:45.264 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:45.308 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:45.330 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:45.330 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:45.330 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:29:45.331 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:45.331 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:45.331 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:45.636 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:45.658 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:45.658 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:45.658 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:29:45.659 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:45.659 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:45.659 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-04-26T01:29:48.937 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:49.186 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:49.239 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-26T01:29:49.239 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:49.239 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T01:29:49.239 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:49.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:29:49.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:29:49.935 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:29:49.936 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:29:49.937 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:29:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:29:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:29:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:29:49.941 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:29:49.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:29:49.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:29:49.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:29:49.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:29:49.944 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:29:49.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:29:49.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:29:49.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:29:49.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:29:49.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:29:49.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:29:50.258 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:29:50.299 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:29:50.340 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:29:50.373 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:29:50.400 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:29:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:29:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:29:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:29:50.929 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T01:29:51.012 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:51.279 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:51.287 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:51.346 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:51.366 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:29:51.379 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-26T01:29:51.380 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:51.380 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T01:29:51.380 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:51.380 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T01:29:51.380 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:29:51.916 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.918 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:29:51.919 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:29:51.920 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:29:51.921 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:29:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:29:51.923 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:29:51.924 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:29:51.925 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:29:51.926 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:29:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:29:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:29:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-04-26T01:29:51.928 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T01:29:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T01:29:52.204 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:29:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T01:29:52.297 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:29:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T01:29:52.326 DEBUG:teuthology.parallel:result is None 2024-04-26T01:29:52.337 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:29:52.420 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:29:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:29:52.452 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:29:52.537 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:29:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T01:29:52.579 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:29:52.579 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:29:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T01:29:52.662 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:29:52.728 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:29:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T01:29:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:29:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:29:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T01:29:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T01:29:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T01:29:53.799 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:53.845 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-26T01:29:53.905 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-26T01:29:54.028 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-26T01:29:54.063 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:55.435 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-26T01:29:55.481 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:56.649 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-26T01:29:56.681 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:56.835 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-26T01:29:57.022 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-26T01:29:57.050 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T01:29:57.125 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:57.184 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:57.224 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-26T01:29:57.339 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-26T01:29:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-26T01:29:58.457 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T01:29:58.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:58.570 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-26T01:29:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T01:29:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:29:58.571 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:58.629 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:58.705 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-26T01:29:58.706 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:58.706 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T01:29:58.706 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:58.780 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:58.860 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-26T01:29:58.861 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:58.861 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T01:29:58.861 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:58.861 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T01:29:58.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:29:59.071 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:29:59.071 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:29:59.071 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:29:59.072 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:29:59.073 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:29:59.086 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:29:59.086 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:29:59.086 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:29:59.086 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:29:59.086 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:29:59.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:29:59.088 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:29:59.089 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:29:59.090 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:29:59.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:29:59.092 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:29:59.093 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:29:59.094 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:29:59.095 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:29:59.096 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:29:59.264 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:59.338 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-26T01:29:59.338 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T01:29:59.339 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T01:29:59.339 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:59.339 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T01:29:59.339 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.150 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.151 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.152 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:30:01.153 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:30:01.154 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:30:01.155 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:30:01.156 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:30:01.157 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:30:01.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:30:01.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:30:01.160 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.161 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:30:01.162 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:30:01.163 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:30:01.163 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:30:01.163 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-04-26T01:30:01.539 DEBUG:teuthology.parallel:result is None 2024-04-26T01:30:04.635 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-26T01:30:04.682 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-26T01:30:04.799 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-26T01:30:04.843 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T01:30:04.891 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T01:30:05.247 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-26T01:30:05.300 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T01:30:05.328 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-26T01:30:06.225 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:06.269 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:06.350 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T01:30:07.772 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-26T01:30:07.883 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T01:30:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T01:30:08.146 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T01:30:08.198 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T01:30:08.198 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:08.334 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T01:30:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:08.521 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T01:30:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:08.963 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:08.963 INFO:teuthology.orchestra.run.smithi052.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T01:30:08.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:30:14.632 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-26T01:30:14.633 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-26T01:30:14.634 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-26T01:30:14.635 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-26T01:30:14.635 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-26T01:30:14.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T01:30:14.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T01:30:14.649 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T01:30:14.650 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T01:30:14.651 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T01:30:14.652 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T01:30:14.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T01:30:14.654 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T01:30:14.655 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T01:30:14.656 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T01:30:17.050 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T01:30:17.051 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.052 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T01:30:17.053 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.054 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T01:30:17.055 INFO:teuthology.orchestra.run.smithi052.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T01:30:17.056 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T01:30:17.057 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T01:30:17.058 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T01:30:17.059 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T01:30:17.060 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T01:30:17.060 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T01:30:17.060 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T01:30:17.060 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T01:30:17.060 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T01:30:17.061 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T01:30:17.062 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-26T01:30:17.063 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:30:17.064 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-04-26T01:30:17.660 DEBUG:teuthology.parallel:result is None 2024-04-26T01:30:17.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:17.801 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:30:17.859 INFO:teuthology.orchestra.run.smithi052.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:17.860 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:17.860 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:30:17.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:18.078 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:30:18.103 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:18.103 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:18.103 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:30:18.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:18.246 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T01:30:18.272 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:18.272 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-26T01:30:18.272 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-26T01:30:18.274 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T01:30:18.274 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:30:18.274 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:30:18.363 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:30:18.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:30:18.409 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:30:18.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:30:18.440 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T01:30:18.440 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:30:18.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:30:18.524 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:30:18.634 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:30:18.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:30:18.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:30:18.731 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:30:18.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:30:18.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:30:18.825 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T01:30:18.826 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:30:18.826 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:30:18.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:30:19.000 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:30:19.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:30:19.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:30:19.090 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:30:19.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:30:19.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:30:19.178 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T01:30:19.178 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:30:19.178 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:30:19.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:30:19.365 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:30:19.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:30:19.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:30:19.453 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:30:19.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:30:19.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:30:19.541 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T01:30:19.653 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:30:19.653 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:19.654 INFO:tasks.cephadm:Cluster fsid is 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:30:19.654 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T01:30:19.654 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.52', 'mon.b': '172.21.15.183', 'mon.c': '172.21.15.191'} 2024-04-26T01:30:19.654 INFO:tasks.cephadm:First mon is mon.a on smithi052 2024-04-26T01:30:19.654 INFO:tasks.cephadm:First mgr is x 2024-04-26T01:30:19.654 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T01:30:19.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-04-26T01:30:19.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-04-26T01:30:19.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-04-26T01:30:19.781 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T01:30:19.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:20.000 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T01:30:20.220 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:30:20.222 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-26T01:30:20.222 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:20.961 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:30 /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:20.961 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:21.651 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:30 /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:21.651 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:22.348 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 26 01:30 /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:22.349 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:22.404 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:22.422 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T01:30:22.459 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-26T01:30:22.459 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:30:22.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:30:22.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-26T01:30:24.011 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:30:24.150 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:30:24.795 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:05.130 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-04-26T01:31:05.131 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:31:05.170 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-04-26T01:31:05.171 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-26T01:31:14.952 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:31:14.952 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-26T01:31:14.952 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-26T01:31:14.952 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-04-26T01:31:14.953 INFO:teuthology.orchestra.run.smithi052.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-26T01:31:14.953 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T01:31:14.953 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:31:15.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-04-26T01:31:15.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-04-26T01:31:15.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-04-26T01:31:15.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-04-26T01:31:15.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-04-26T01:31:15.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-04-26T01:31:15.325 INFO:tasks.cephadm:Writing seed config... 2024-04-26T01:31:15.326 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T01:31:15.326 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T01:31:15.329 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T01:31:15.330 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T01:31:15.331 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T01:31:15.332 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T01:31:15.333 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:31:15.333 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T01:31:15.394 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8b150f36-036c-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T01:31:15.395 DEBUG:teuthology.orchestra.run.smithi052:mon.a> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T01:31:15.443 DEBUG:teuthology.orchestra.run.smithi052:mgr.x> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T01:31:15.489 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T01:31:15.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:31:15.766 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-04-26T01:31:15.766 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', '8b150f36-036c-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.52', '--skip-admin-label'] 2024-04-26T01:31:15.767 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T01:31:15.767 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2024-04-26T01:31:15.831 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:31:15.832 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2024-04-26T01:31:15.832 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2024-04-26T01:31:15.851 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:31:15.851 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:31:15.870 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:31:15.870 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:31:15.889 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-04-26T01:31:15.906 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-04-26T01:31:15.906 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:31:15.906 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2024-04-26T01:31:15.957 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T01:31:15.957 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T01:31:15.957 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2024-04-26T01:31:15.957 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2024-04-26T01:31:15.975 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T01:31:15.975 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T01:31:15.992 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T01:31:15.992 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:31:16.014 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 139976551062688 on /run/cephadm/8b150f36-036c-11ef-bc93-c7b262605968.lock 2024-04-26T01:31:16.032 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 139976551062688 acquired on /run/cephadm/8b150f36-036c-11ef-bc93-c7b262605968.lock 2024-04-26T01:31:16.033 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 3300 ... 2024-04-26T01:31:16.033 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2024-04-26T01:31:16.033 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:3300, 172.21.15.52:6789], mon addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-04-26T01:31:16.041 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-04-26T01:31:16.041 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-04-26T01:31:16.049 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T01:31:16.049 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T01:31:16.049 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-04-26T01:31:16.056 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T01:31:16.057 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-04-26T01:31:16.057 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-04-26T01:31:16.057 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T01:31:16.058 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T01:31:16.059 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:31:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-26T01:31:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T01:31:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-26T01:31:16.813 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-26T01:31:16.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-26T01:31:16.814 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T01:31:18.055 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:31:18.055 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-26T01:31:18.055 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T01:31:19.145 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2024-04-26T01:31:19.145 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2024-04-26T01:31:20.339 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDnAytmv7b4JxAAT4lHh9c1I5MHeYSOzxaBtw== 2024-04-26T01:31:21.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDoAytmdfVAMBAA8Rgc4EsgeYATvUrjok5thQ== 2024-04-26T01:31:22.596 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDpAytmQTJ7OBAAU/o18lwRtl3fx+R9pZEs2g== 2024-04-26T01:31:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2024-04-26T01:31:23.817 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:31:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T01:31:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:31:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for a [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T01:31:23.818 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-04-26T01:31:23.819 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:23.819 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T01:31:23.819 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:31:23.819 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2024-04-26T01:31:24.943 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.a on 2024-04-26T01:31:25.341 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T01:31:25.592 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T01:31:25.896 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target. 2024-04-26T01:31:25.896 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target. 2024-04-26T01:31:26.299 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a 2024-04-26T01:31:26.299 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T01:31:26.589 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target.wants/ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968@.service. 2024-04-26T01:31:27.275 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 podman[27469]: 2024-04-26 01:31:27.268904164 +0000 UTC m=+0.381021998 container init 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T01:31:27.275 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 podman[27469]: 2024-04-26 01:31:27.279818955 +0000 UTC m=+0.391936790 container start 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:31:27.422 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:31:27.423 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-04-26T01:31:27.444 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:31:27.445 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:31:27.446 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-04-26T01:31:27.446 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:31:27.447 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2024-04-26T01:31:27.448 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2024-04-26T01:31:27.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 ceph-mon[27488]: mkfs 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:27.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 ceph-mon[27488]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:31:27.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 bash[27469]: 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 2024-04-26T01:31:27.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:27 smithi052 systemd[1]: Started Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:31:28.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:31:28.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: monmap epoch 1 2024-04-26T01:31:28.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: last_changed 2024-04-26T01:31:23.102434+0000 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: min_mon_release 19 (squid) 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: election_strategy: 1 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: fsmap 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: mgrmap e1: no daemons active 2024-04-26T01:31:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:28 smithi052 ceph-mon[27488]: from='client.? 172.21.15.52:0/2848139797' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T01:31:28.822 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T01:31:28.822 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.822439s) 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T01:31:28.823 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.086+0000 7fdf19000700 1 Processor -- start 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.086+0000 7fdf19000700 1 -- start start 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.086+0000 7fdf19000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.086+0000 7fdf19000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf14110c10 con 0x7fdf14110220 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.087+0000 7fdf11e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.087+0000 7fdf11e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38874/0 (socket says 172.21.15.52:38874) 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.087+0000 7fdf11e00700 1 -- 172.21.15.52:0/3350687562 learned_addr learned my addr 172.21.15.52:0/3350687562 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.087+0000 7fdf11e00700 1 -- 172.21.15.52:0/3350687562 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1410ece0 con 0x7fdf14110220 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.088+0000 7fdf11e00700 1 --2- 172.21.15.52:0/3350687562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdf040098a0 tx=0x7fdf04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f045b64520892ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.088+0000 7fdf11400700 1 -- 172.21.15.52:0/3350687562 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf04004160 con 0x7fdf14110220 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.088+0000 7fdf11400700 1 -- 172.21.15.52:0/3350687562 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf04004300 con 0x7fdf14110220 2024-04-26T01:31:28.824 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.088+0000 7fdf11400700 1 -- 172.21.15.52:0/3350687562 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf040036a0 con 0x7fdf14110220 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 -- 172.21.15.52:0/3350687562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 msgr2=0x7fdf14110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 --2- 172.21.15.52:0/3350687562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdf040098a0 tx=0x7fdf04009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 -- 172.21.15.52:0/3350687562 shutdown_connections 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 --2- 172.21.15.52:0/3350687562 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf14110640 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 -- 172.21.15.52:0/3350687562 >> 172.21.15.52:0/3350687562 conn(0x7fdf1407de20 msgr2=0x7fdf14080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 -- 172.21.15.52:0/3350687562 shutdown_connections 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.089+0000 7fdf19000700 1 -- 172.21.15.52:0/3350687562 wait complete. 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.090+0000 7fdf19000700 1 Processor -- start 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf19000700 1 -- start start 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf19000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf19000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1410f1b0 con 0x7fdf14110220 2024-04-26T01:31:28.825 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf11e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf11e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38878/0 (socket says 172.21.15.52:38878) 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.091+0000 7fdf11e00700 1 -- 172.21.15.52:0/2848139797 learned_addr learned my addr 172.21.15.52:0/2848139797 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf11e00700 1 -- 172.21.15.52:0/2848139797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf141a28b0 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf11e00700 1 --2- 172.21.15.52:0/2848139797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdf04000c00 tx=0x7fdf04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf04003db0 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf04011630 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf04011950 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf141a2b40 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.092+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf141a5830 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.093+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf0401a460 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.093+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf0401b780 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.094+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf140640a0 con 0x7fdf14110220 2024-04-26T01:31:28.826 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.097+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdf04031020 con 0x7fdf14110220 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.178+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fdf1402e170 con 0x7fdf14110220 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.178+0000 7fdf0b400700 1 -- 172.21.15.52:0/2848139797 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fdf0401ba20 con 0x7fdf14110220 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.179+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 msgr2=0x7fdf141a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.179+0000 7fdf19000700 1 --2- 172.21.15.52:0/2848139797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdf04000c00 tx=0x7fdf04004750 comp rx=0 tx=0).stop 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.180+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 shutdown_connections 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.180+0000 7fdf19000700 1 --2- 172.21.15.52:0/2848139797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf14110220 0x7fdf141a2370 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.180+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 >> 172.21.15.52:0/2848139797 conn(0x7fdf1407de20 msgr2=0x7fdf14198f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.180+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 shutdown_connections 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:28.180+0000 7fdf19000700 1 -- 172.21.15.52:0/2848139797 wait complete. 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2024-04-26T01:31:28.827 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T01:31:29.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:29 smithi052 ceph-mon[27488]: from='client.? 172.21.15.52:0/1667801100' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:31:29.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:29 smithi052 ceph-mon[27488]: from='client.? 172.21.15.52:0/1667801100' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T01:31:30.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:31:30.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.455+0000 7f52c6a00700 1 Processor -- start 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.455+0000 7f52c6a00700 1 -- start start 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.455+0000 7f52c6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.455+0000 7f52c6a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f52c007b7b0 con 0x7f52c007cd80 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.456+0000 7f52c5600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.456+0000 7f52c5600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38886/0 (socket says 172.21.15.52:38886) 2024-04-26T01:31:30.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.456+0000 7f52c5600700 1 -- 172.21.15.52:0/722049049 learned_addr learned my addr 172.21.15.52:0/722049049 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.457+0000 7f52c5600700 1 -- 172.21.15.52:0/722049049 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c007b930 con 0x7f52c007cd80 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.457+0000 7f52c5600700 1 --2- 172.21.15.52:0/722049049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f52b40098a0 tx=0x7f52b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5898f973e4414c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.457+0000 7f52bfe00700 1 -- 172.21.15.52:0/722049049 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52b4004160 con 0x7f52c007cd80 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.457+0000 7f52bfe00700 1 -- 172.21.15.52:0/722049049 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b4004300 con 0x7f52c007cd80 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.457+0000 7f52bfe00700 1 -- 172.21.15.52:0/722049049 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52b40036a0 con 0x7f52c007cd80 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 -- 172.21.15.52:0/722049049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 msgr2=0x7f52c007b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 --2- 172.21.15.52:0/722049049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f52b40098a0 tx=0x7f52b4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 -- 172.21.15.52:0/722049049 shutdown_connections 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 --2- 172.21.15.52:0/722049049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c007b1e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 -- 172.21.15.52:0/722049049 >> 172.21.15.52:0/722049049 conn(0x7f52c0103990 msgr2=0x7f52c0105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 -- 172.21.15.52:0/722049049 shutdown_connections 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.458+0000 7f52c6a00700 1 -- 172.21.15.52:0/722049049 wait complete. 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c6a00700 1 Processor -- start 2024-04-26T01:31:30.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c6a00700 1 -- start start 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c6a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f52c007be60 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c5600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c5600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38888/0 (socket says 172.21.15.52:38888) 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.460+0000 7f52c5600700 1 -- 172.21.15.52:0/1667801100 learned_addr learned my addr 172.21.15.52:0/1667801100 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52c5600700 1 -- 172.21.15.52:0/1667801100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c019e1a0 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52c5600700 1 --2- 172.21.15.52:0/1667801100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52b4003db0 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b4011630 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52b4011950 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.461+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c019e430 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.462+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c01a1120 con 0x7f52c007cd80 2024-04-26T01:31:30.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.462+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f52b4011af0 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.462+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f52b401c450 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.463+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c0069ba0 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.466+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f52b4032020 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.544+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f52c002e1c0 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.553+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7f52b4004260 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.554+0000 7f52be000700 1 -- 172.21.15.52:0/1667801100 <== mon.0 v2:172.21.15.52:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f52b401c650 con 0x7f52c007cd80 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 msgr2=0x7f52c019dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 --2- 172.21.15.52:0/1667801100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b4004750 comp rx=0 tx=0).stop 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 shutdown_connections 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 --2- 172.21.15.52:0/1667801100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52c007cd80 0x7f52c019dc60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 >> 172.21.15.52:0/1667801100 conn(0x7f52c0103990 msgr2=0x7f52c0105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 shutdown_connections 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:29.555+0000 7f52c6a00700 1 -- 172.21.15.52:0/1667801100 wait complete. 2024-04-26T01:31:30.231 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2024-04-26T01:31:31.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:30 smithi052 ceph-mon[27488]: from='client.? 172.21.15.52:0/1963827198' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:31:31.505 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.825+0000 7f8267400700 1 Processor -- start 2024-04-26T01:31:31.505 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.825+0000 7f8267400700 1 -- start start 2024-04-26T01:31:31.505 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.825+0000 7f8267400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:31.505 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.825+0000 7f8267400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f826007b810 con 0x7f826007cde0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.826+0000 7f8264a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.826+0000 7f8264a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38890/0 (socket says 172.21.15.52:38890) 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.826+0000 7f8264a00700 1 -- 172.21.15.52:0/3045895472 learned_addr learned my addr 172.21.15.52:0/3045895472 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.826+0000 7f8264a00700 1 -- 172.21.15.52:0/3045895472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826007b990 con 0x7f826007cde0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.826+0000 7f8264a00700 1 --2- 172.21.15.52:0/3045895472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f82500098a0 tx=0x7f8250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ca227dfc173a15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.827+0000 7f825fe00700 1 -- 172.21.15.52:0/3045895472 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8250004160 con 0x7f826007cde0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.827+0000 7f825fe00700 1 -- 172.21.15.52:0/3045895472 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8250004300 con 0x7f826007cde0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.827+0000 7f825fe00700 1 -- 172.21.15.52:0/3045895472 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8250011af0 con 0x7f826007cde0 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.827+0000 7f8267400700 1 -- 172.21.15.52:0/3045895472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 msgr2=0x7f826007b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 --2- 172.21.15.52:0/3045895472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f82500098a0 tx=0x7f8250009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 -- 172.21.15.52:0/3045895472 shutdown_connections 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 --2- 172.21.15.52:0/3045895472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826007b240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 -- 172.21.15.52:0/3045895472 >> 172.21.15.52:0/3045895472 conn(0x7f8260103b50 msgr2=0x7f8260105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 -- 172.21.15.52:0/3045895472 shutdown_connections 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.828+0000 7f8267400700 1 -- 172.21.15.52:0/3045895472 wait complete. 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.829+0000 7f8267400700 1 Processor -- start 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.829+0000 7f8267400700 1 -- start start 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.829+0000 7f8267400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.829+0000 7f8267400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f826010eaf0 con 0x7f826007cde0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.830+0000 7f8264a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.830+0000 7f8264a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38894/0 (socket says 172.21.15.52:38894) 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.830+0000 7f8264a00700 1 -- 172.21.15.52:0/1963827198 learned_addr learned my addr 172.21.15.52:0/1963827198 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.830+0000 7f8264a00700 1 -- 172.21.15.52:0/1963827198 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826019ebc0 con 0x7f826007cde0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.830+0000 7f8264a00700 1 --2- 172.21.15.52:0/1963827198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8250000c00 tx=0x7f8250011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8250003b20 con 0x7f826007cde0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f825001a8c0 con 0x7f826007cde0 2024-04-26T01:31:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826019ee50 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8250024940 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82601a1b40 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8250022020 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.831+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f825002ddd0 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.832+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82600640a0 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.835+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8250042e50 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.914+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f82601a2090 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.914+0000 7f825e000700 1 -- 172.21.15.52:0/1963827198 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f8250031030 con 0x7f826007cde0 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 msgr2=0x7f826019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 --2- 172.21.15.52:0/1963827198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8250000c00 tx=0x7f8250011b70 comp rx=0 tx=0).stop 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 shutdown_connections 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 --2- 172.21.15.52:0/1963827198 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f826007cde0 0x7f826019e680 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 >> 172.21.15.52:0/1963827198 conn(0x7f8260103b50 msgr2=0x7f8260104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:31.508 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 shutdown_connections 2024-04-26T01:31:31.509 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:30.916+0000 7f8267400700 1 -- 172.21.15.52:0/1963827198 wait complete. 2024-04-26T01:31:31.509 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2024-04-26T01:31:31.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:31 smithi052 systemd[1]: Stopping Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:31:38.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a[27484]: 2024-04-26T01:31:37.970+0000 7f3d8e200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T01:31:38.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a[27484]: 2024-04-26T01:31:37.970+0000 7f3d8e200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T01:31:38.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:38 smithi052 podman[27730]: 2024-04-26 01:31:38.063422857 +0000 UTC m=+6.141610878 container died 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:31:41.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 podman[27730]: 2024-04-26 01:31:41.118875792 +0000 UTC m=+9.197063859 container cleanup 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T01:31:41.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 bash[27730]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a 2024-04-26T01:31:41.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 podman[27744]: 2024-04-26 01:31:41.311470114 +0000 UTC m=+3.233601856 container remove 4a5d10976cc835ffc338fd8b4a3778cd8b1d975d96c5ac92902a55aaad4314b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:31:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 systemd[1]: ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T01:31:42.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 systemd[1]: Stopped Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:31:42.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 systemd[1]: ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.106s CPU time. 2024-04-26T01:31:42.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:41 smithi052 systemd[1]: Starting Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:31:42.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 podman[27833]: 2024-04-26 01:31:42.192896159 +0000 UTC m=+0.036900717 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:42.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 podman[27833]: 2024-04-26 01:31:42.379132943 +0000 UTC m=+0.223137528 container create d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-26T01:31:42.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 podman[27833]: 2024-04-26 01:31:42.607569823 +0000 UTC m=+0.451574414 container init d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True) 2024-04-26T01:31:42.696 INFO:teuthology.orchestra.run.smithi052.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T01:31:42.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 podman[27833]: 2024-04-26 01:31:42.618274236 +0000 UTC m=+0.462278828 container start d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: pidfile_write: ignore empty --pid-file 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: load: jerasure load: lrc 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: RocksDB version: 7.9.2 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Git sha 0 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: DB SUMMARY 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: DB Session ID: LSNK5YL5DXEDXYQWWCBB 2024-04-26T01:31:42.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: CURRENT file: CURRENT 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105448 ; 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.error_if_exists: 0 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.create_if_missing: 0 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.paranoid_checks: 1 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.env: 0x561b3a76dcc0 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.info_log: 0x561b3ea19c40 2024-04-26T01:31:42.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.statistics: (nil) 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.use_fsync: 0 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_log_file_size: 0 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_fallocate: 1 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T01:31:42.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.use_direct_reads: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.db_log_dir: 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.wal_dir: 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T01:31:42.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.write_buffer_manager: 0x561b3e9ad0e0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.unordered_write: 0 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T01:31:42.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.row_cache: None 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.wal_filter: None 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.two_write_queues: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.wal_compression: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.atomic_flush: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T01:31:42.967 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.log_readahead_size: 0 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_background_jobs: 2 2024-04-26T01:31:42.968 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_background_compactions: -1 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_subcompactions: 1 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_open_files: -1 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T01:31:42.969 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_background_flushes: -1 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Compression algorithms supported: 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kZSTD supported: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kXpressCompression supported: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kZlibCompression supported: 1 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kSnappyCompression supported: 1 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kLZ4Compression supported: 1 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: kBZip2Compression supported: 0 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T01:31:42.970 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T01:31:42.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T01:31:42.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T01:31:42.971 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.merge_operator: 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_filter: None 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561b3ea19e00) 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T01:31:42.972 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_type: 0 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_index_type: 0 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_shortening: 1 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: checksum: 4 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: no_block_cache: 0 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache: 0x561b3e9df1f0 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_options: 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: capacity : 536870912 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_shard_bits : 4 2024-04-26T01:31:42.973 INFO:journalctl@ceph.mon.a.smithi052.stdout: strict_capacity_limit : 0 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_compressed: (nil) 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: persistent_cache: (nil) 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size: 4096 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size_deviation: 10 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_restart_interval: 16 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_block_restart_interval: 1 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: metadata_block_size: 4096 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: partition_filters: 0 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: use_delta_encoding: 1 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: filter_policy: bloomfilter 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: whole_key_filtering: 1 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: verify_compression: 0 2024-04-26T01:31:42.974 INFO:journalctl@ceph.mon.a.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: format_version: 5 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_index_compression: 1 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_align: 0 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: max_auto_readahead_size: 262144 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: prepopulate_block_cache: 0 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: initial_auto_readahead_size: 8192 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression: NoCompression 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T01:31:42.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.num_levels: 7 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T01:31:42.976 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T01:31:42.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T01:31:42.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.inplace_update_support: 0 2024-04-26T01:31:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.bloom_locality: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.max_successive_merges: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.ttl: 2592000 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enable_blob_files: false 2024-04-26T01:31:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.min_blob_size: 0 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 299b321a-9203-4a1b-995d-7026ad5430e8 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095102679988, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095102682117, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98445, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90699, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714095102, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "299b321a-9203-4a1b-995d-7026ad5430e8", "db_session_id": "LSNK5YL5DXEDXYQWWCBB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T01:31:42.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714095102682308, "job": 1, "event": "recovery_finished"} 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561b3eaf4000 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: DB pointer 0x561b3eade000 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** DB Stats ** 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T01:31:42.982 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: L0 2/0 100.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: Sum 2/0 100.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 68.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.983 INFO:journalctl@ceph.mon.a.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative compaction: 0.00 GB write, 13.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval compaction: 0.00 GB write, 13.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache BinnedLRUCache@0x561b3e9df1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.984 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: starting mon.a rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???) e1 preinit fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).mds e1 new map 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).mds e1 print_map 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: e1 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: legacy client fscid: -1 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout: No filesystems configured 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:31:42.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 bash[27833]: d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 systemd[1]: Started Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: monmap epoch 1 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: last_changed 2024-04-26T01:31:23.102434+0000 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: min_mon_release 19 (squid) 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: election_strategy: 1 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: fsmap 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T01:31:42.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:42 smithi052 ceph-mon[27851]: mgrmap e1: no daemons active 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.330+0000 7fd6d4e00700 1 Processor -- start 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.330+0000 7fd6d4e00700 1 -- start start 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.331+0000 7fd6d4e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.331+0000 7fd6d4e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d007b810 con 0x7fd6d007cde0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.331+0000 7fd6cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.331+0000 7fd6cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47710/0 (socket says 172.21.15.52:47710) 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.332+0000 7fd6cde00700 1 -- 172.21.15.52:0/292875130 learned_addr learned my addr 172.21.15.52:0/292875130 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.332+0000 7fd6cde00700 1 -- 172.21.15.52:0/292875130 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d007b990 con 0x7fd6d007cde0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.332+0000 7fd6cde00700 1 --2- 172.21.15.52:0/292875130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00098a0 tx=0x7fd6c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60281530ce6f9c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.333+0000 7fd6cca00700 1 -- 172.21.15.52:0/292875130 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6c0004160 con 0x7fd6d007cde0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.333+0000 7fd6cca00700 1 -- 172.21.15.52:0/292875130 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6c0004300 con 0x7fd6d007cde0 2024-04-26T01:31:44.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.333+0000 7fd6cca00700 1 -- 172.21.15.52:0/292875130 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6c0011af0 con 0x7fd6d007cde0 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 -- 172.21.15.52:0/292875130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 msgr2=0x7fd6d007b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 --2- 172.21.15.52:0/292875130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00098a0 tx=0x7fd6c0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 -- 172.21.15.52:0/292875130 shutdown_connections 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 --2- 172.21.15.52:0/292875130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d007b240 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 -- 172.21.15.52:0/292875130 >> 172.21.15.52:0/292875130 conn(0x7fd6d0103b50 msgr2=0x7fd6d0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 -- 172.21.15.52:0/292875130 shutdown_connections 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.334+0000 7fd6d4e00700 1 -- 172.21.15.52:0/292875130 wait complete. 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.335+0000 7fd6d4e00700 1 Processor -- start 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.335+0000 7fd6d4e00700 1 -- start start 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6d4e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6d4e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010a6e0 con 0x7fd6d007cde0 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:44.161 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47722/0 (socket says 172.21.15.52:47722) 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6cde00700 1 -- 172.21.15.52:0/929710764 learned_addr learned my addr 172.21.15.52:0/929710764 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.336+0000 7fd6cde00700 1 -- 172.21.15.52:0/929710764 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d01a2fd0 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6cde00700 1 --2- 172.21.15.52:0/929710764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0000c00 tx=0x7fd6c0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6c0003b20 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6c001a8c0 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6c0024940 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d01a3260 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.337+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d01a5f50 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.338+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd6c0022020 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.338+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd6c002ddd0 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.339+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d00640a0 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.342+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd6c0042e50 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.420+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fd6d019bff0 con 0x7fd6d007cde0 2024-04-26T01:31:44.162 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.425+0000 7fd6c6a00700 1 -- 172.21.15.52:0/929710764 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fd6c0031020 con 0x7fd6d007cde0 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.426+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 msgr2=0x7fd6d01a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.426+0000 7fd6d4e00700 1 --2- 172.21.15.52:0/929710764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0000c00 tx=0x7fd6c0011b70 comp rx=0 tx=0).stop 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.426+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 shutdown_connections 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.426+0000 7fd6d4e00700 1 --2- 172.21.15.52:0/929710764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd6d007cde0 0x7fd6d01a2a90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.426+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 >> 172.21.15.52:0/929710764 conn(0x7fd6d0103b50 msgr2=0x7fd6d0105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.427+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 shutdown_connections 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:43.427+0000 7fd6d4e00700 1 -- 172.21.15.52:0/929710764 wait complete. 2024-04-26T01:31:44.163 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T01:31:44.189 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:31:44.189 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2024-04-26T01:31:44.190 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T01:31:44.191 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T01:31:44.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:44 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/929710764' entity='client.admin' 2024-04-26T01:31:44.643 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x 2024-04-26T01:31:44.643 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T01:31:44.888 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968.target.wants/ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-8b150f36-036c-11ef-bc93-c7b262605968@.service. 2024-04-26T01:31:45.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:44 smithi052 systemd[1]: Starting Ceph mgr.x for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:31:45.709 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:45 smithi052 podman[28075]: 2024-04-26 01:31:45.217570153 +0000 UTC m=+0.033797133 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-26T01:31:46.128 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:45 smithi052 podman[28075]: 2024-04-26 01:31:45.781377183 +0000 UTC m=+0.597604200 container create cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T01:31:46.128 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 podman[28075]: 2024-04-26 01:31:46.001285943 +0000 UTC m=+0.817513003 container init cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-26T01:31:46.128 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 podman[28075]: 2024-04-26 01:31:46.011133162 +0000 UTC m=+0.827360186 container start cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T01:31:46.128 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 bash[28075]: cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 2024-04-26T01:31:46.164 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:31:46.165 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-04-26T01:31:46.182 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:31:46.182 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:31:46.183 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-04-26T01:31:46.183 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T01:31:46.215 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:31:46.216 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2024-04-26T01:31:46.239 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2024-04-26T01:31:46.395 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 systemd[1]: Started Ceph mgr.x for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:31:46.396 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:46.260+0000 7fe92ec2a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:31:46.663 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:46.528+0000 7fe92ec2a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:31:46.663 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:46.666+0000 7fe92ec2a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:31:47.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:47 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2970979412' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:31:47.207 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:47 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:47.095+0000 7fe92ec2a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:31:47.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:47 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:47.212+0000 7fe92ec2a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:31:47.700 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:47.703 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:31:47.703 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-26T01:31:47.704 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:31:47.705 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:31:47.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:31:47.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:31:47.714 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:31:27.359437+0000", 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:47.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.856+0000 7f9340800700 1 Processor -- start 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.856+0000 7f9340800700 1 -- start start 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.856+0000 7f9340800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.856+0000 7f9340800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9338108f40 con 0x7f9338108550 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.857+0000 7f933de00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.857+0000 7f933de00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34106/0 (socket says 172.21.15.52:34106) 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.857+0000 7f933de00700 1 -- 172.21.15.52:0/3459054106 learned_addr learned my addr 172.21.15.52:0/3459054106 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.857+0000 7f933de00700 1 -- 172.21.15.52:0/3459054106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9338109700 con 0x7f9338108550 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.858+0000 7f933de00700 1 --2- 172.21.15.52:0/3459054106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f932c0098a0 tx=0x7f932c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f52b8244224fe5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.858+0000 7f933ca00700 1 -- 172.21.15.52:0/3459054106 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c004160 con 0x7f9338108550 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.858+0000 7f933ca00700 1 -- 172.21.15.52:0/3459054106 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f932c004300 con 0x7f9338108550 2024-04-26T01:31:47.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.858+0000 7f933ca00700 1 -- 172.21.15.52:0/3459054106 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c011af0 con 0x7f9338108550 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 -- 172.21.15.52:0/3459054106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 msgr2=0x7f9338108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 --2- 172.21.15.52:0/3459054106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f932c0098a0 tx=0x7f932c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 -- 172.21.15.52:0/3459054106 shutdown_connections 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 --2- 172.21.15.52:0/3459054106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f9338108970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 -- 172.21.15.52:0/3459054106 >> 172.21.15.52:0/3459054106 conn(0x7f9338103b10 msgr2=0x7f9338105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 -- 172.21.15.52:0/3459054106 shutdown_connections 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.859+0000 7f9340800700 1 -- 172.21.15.52:0/3459054106 wait complete. 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.860+0000 7f9340800700 1 Processor -- start 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.860+0000 7f9340800700 1 -- start start 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f9340800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f9340800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f933810cab0 con 0x7f9338108550 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f933de00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:47.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f933de00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34120/0 (socket says 172.21.15.52:34120) 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f933de00700 1 -- 172.21.15.52:0/2970979412 learned_addr learned my addr 172.21.15.52:0/2970979412 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.861+0000 7f933de00700 1 -- 172.21.15.52:0/2970979412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933807e0c0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f933de00700 1 --2- 172.21.15.52:0/2970979412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f932c0117b0 tx=0x7f932c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c003af0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f932c020070 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932c019630 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933807c2b0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933807c7f0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f932c019c30 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.862+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f932c00b720 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.863+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93380640a0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.866+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f932c01a9c0 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.946+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f933807cb30 con 0x7f9338108550 2024-04-26T01:31:47.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.947+0000 7f9332a00700 1 -- 172.21.15.52:0/2970979412 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f932c01a430 con 0x7f9338108550 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.949+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 msgr2=0x7f933807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.949+0000 7f9340800700 1 --2- 172.21.15.52:0/2970979412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f932c0117b0 tx=0x7f932c011b70 comp rx=0 tx=0).stop 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.950+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 shutdown_connections 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.950+0000 7f9340800700 1 --2- 172.21.15.52:0/2970979412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9338108550 0x7f933807db80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.950+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 >> 172.21.15.52:0/2970979412 conn(0x7f9338103b10 msgr2=0x7f93381046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.950+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 shutdown_connections 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:46.950+0000 7f9340800700 1 -- 172.21.15.52:0/2970979412 wait complete. 2024-04-26T01:31:47.720 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2024-04-26T01:31:49.209 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:48.839+0000 7fe92ec2a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:31:49.209 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:49.071+0000 7fe92ec2a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:31:49.675 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:49.288+0000 7fe92ec2a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:31:49.675 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:49.679+0000 7fe92ec2a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:31:49.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:49.807+0000 7fe92ec2a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:31:50.327 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:50.068+0000 7fe92ec2a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:31:50.328 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:50.332+0000 7fe92ec2a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:31:50.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:50 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/401401698' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:31:51.152 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:51.152 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:31:51.152 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:31:51.152 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:31:51.153 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:31:51.153 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:31:51.153 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:31:51.153 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.153 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:31:51.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:31:51.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:51.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:31:51.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:31:51.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:31:51.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:31:51.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:51.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:31:27.359437+0000", 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:31:51.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.333+0000 7fcc14e00700 1 Processor -- start 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.333+0000 7fcc14e00700 1 -- start start 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.333+0000 7fcc14e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.333+0000 7fcc14e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1007b7f0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.334+0000 7fcc0f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.334+0000 7fcc0f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34124/0 (socket says 172.21.15.52:34124) 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.334+0000 7fcc0f400700 1 -- 172.21.15.52:0/2250089382 learned_addr learned my addr 172.21.15.52:0/2250089382 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.334+0000 7fcc0f400700 1 -- 172.21.15.52:0/2250089382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1007b970 con 0x7fcc1007cdc0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.335+0000 7fcc0f400700 1 --2- 172.21.15.52:0/2250089382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc000098a0 tx=0x7fcc00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e3e103f00575db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.335+0000 7fcc0e000700 1 -- 172.21.15.52:0/2250089382 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc00004160 con 0x7fcc1007cdc0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.335+0000 7fcc0e000700 1 -- 172.21.15.52:0/2250089382 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc00004300 con 0x7fcc1007cdc0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.335+0000 7fcc0e000700 1 -- 172.21.15.52:0/2250089382 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc00011af0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 -- 172.21.15.52:0/2250089382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 msgr2=0x7fcc1007b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 --2- 172.21.15.52:0/2250089382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc000098a0 tx=0x7fcc00009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 -- 172.21.15.52:0/2250089382 shutdown_connections 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 --2- 172.21.15.52:0/2250089382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1007b220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 -- 172.21.15.52:0/2250089382 >> 172.21.15.52:0/2250089382 conn(0x7fcc10103990 msgr2=0x7fcc10105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 -- 172.21.15.52:0/2250089382 shutdown_connections 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.336+0000 7fcc14e00700 1 -- 172.21.15.52:0/2250089382 wait complete. 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc14e00700 1 Processor -- start 2024-04-26T01:31:51.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc14e00700 1 -- start start 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc14e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc14e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1010e8e0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc0f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.338+0000 7fcc0f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34132/0 (socket says 172.21.15.52:34132) 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc0f400700 1 -- 172.21.15.52:0/401401698 learned_addr learned my addr 172.21.15.52:0/401401698 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc0f400700 1 -- 172.21.15.52:0/401401698 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1019e9e0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc0f400700 1 --2- 172.21.15.52:0/401401698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc00003f40 tx=0x7fcc00011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc000038b0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc00003a50 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1019ec70 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.339+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc0001a5c0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.340+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc101a1960 con 0x7fcc1007cdc0 2024-04-26T01:31:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.340+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcc0001a8c0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.340+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc0002ddb0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.341+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc10069ba0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.344+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcc000248c0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.424+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcc101a1ea0 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.424+0000 7fcc07e00700 1 -- 172.21.15.52:0/401401698 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcc00032030 con 0x7fcc1007cdc0 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.425+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 msgr2=0x7fcc1019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.425+0000 7fcc14e00700 1 --2- 172.21.15.52:0/401401698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc00003f40 tx=0x7fcc00011890 comp rx=0 tx=0).stop 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.425+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 shutdown_connections 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.425+0000 7fcc14e00700 1 --2- 172.21.15.52:0/401401698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc1007cdc0 0x7fcc1019e4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.425+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 >> 172.21.15.52:0/401401698 conn(0x7fcc10103990 msgr2=0x7fcc10104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.426+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 shutdown_connections 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:50.426+0000 7fcc14e00700 1 -- 172.21.15.52:0/401401698 wait complete. 2024-04-26T01:31:51.171 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2024-04-26T01:31:51.459 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:51.177+0000 7fe92ec2a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:31:52.141 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:51.743+0000 7fe92ec2a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:31:52.141 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:51.873+0000 7fe92ec2a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:31:52.141 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:52.145+0000 7fe92ec2a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:31:53.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:52.808+0000 7fe92ec2a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:31:53.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:52.933+0000 7fe92ec2a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:31:53.587 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:53.267+0000 7fe92ec2a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:31:53.587 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:53.459+0000 7fe92ec2a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:31:53.587 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:53.590+0000 7fe92ec2a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:31:53.859 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:53.750+0000 7fe92ec2a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:31:54.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:53 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3300906446' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:31:54.458 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:54.136+0000 7fe92ec2a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:31:54.529 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:31:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:31:54.531 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:31:54.532 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:31:54.533 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:31:54.534 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.535 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:31:54.536 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:31:27.359437+0000", 2024-04-26T01:31:54.537 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.774+0000 7f44f8200700 1 Processor -- start 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.774+0000 7f44f8200700 1 -- start start 2024-04-26T01:31:54.538 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.774+0000 7f44f8200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.775+0000 7f44f8200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0063ce0 con 0x7f44f0106e20 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.775+0000 7f44f5800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.775+0000 7f44f5800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34134/0 (socket says 172.21.15.52:34134) 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.775+0000 7f44f5800700 1 -- 172.21.15.52:0/3476765096 learned_addr learned my addr 172.21.15.52:0/3476765096 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.776+0000 7f44f5800700 1 -- 172.21.15.52:0/3476765096 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f0107780 con 0x7f44f0106e20 2024-04-26T01:31:54.539 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.776+0000 7f44f5800700 1 --2- 172.21.15.52:0/3476765096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44e40098a0 tx=0x7f44e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9278fce14062df79 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.776+0000 7f44efe00700 1 -- 172.21.15.52:0/3476765096 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e4004160 con 0x7f44f0106e20 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.776+0000 7f44efe00700 1 -- 172.21.15.52:0/3476765096 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44e4004300 con 0x7f44f0106e20 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.776+0000 7f44efe00700 1 -- 172.21.15.52:0/3476765096 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e4011af0 con 0x7f44f0106e20 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 -- 172.21.15.52:0/3476765096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 msgr2=0x7f44f0107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 --2- 172.21.15.52:0/3476765096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44e40098a0 tx=0x7f44e4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 -- 172.21.15.52:0/3476765096 shutdown_connections 2024-04-26T01:31:54.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 --2- 172.21.15.52:0/3476765096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f0107240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 -- 172.21.15.52:0/3476765096 >> 172.21.15.52:0/3476765096 conn(0x7f44f0102a40 msgr2=0x7f44f0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 -- 172.21.15.52:0/3476765096 shutdown_connections 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.777+0000 7f44f8200700 1 -- 172.21.15.52:0/3476765096 wait complete. 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.778+0000 7f44f8200700 1 Processor -- start 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.778+0000 7f44f8200700 1 -- start start 2024-04-26T01:31:54.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f8200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f8200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01087b0 con 0x7f44f0106e20 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f5800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f5800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34146/0 (socket says 172.21.15.52:34146) 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f5800700 1 -- 172.21.15.52:0/3300906446 learned_addr learned my addr 172.21.15.52:0/3300906446 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.779+0000 7f44f5800700 1 -- 172.21.15.52:0/3300906446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f019eb90 con 0x7f44f0106e20 2024-04-26T01:31:54.542 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44f5800700 1 --2- 172.21.15.52:0/3300906446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f44e4011ac0 tx=0x7f44e4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e4003b20 con 0x7f44f0106e20 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44e401a8c0 con 0x7f44f0106e20 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44e40248d0 con 0x7f44f0106e20 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f019ee20 con 0x7f44f0106e20 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.780+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f01a1b10 con 0x7f44f0106e20 2024-04-26T01:31:54.543 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.781+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44e4022020 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.781+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f44e402dde0 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.781+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f00640a0 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.784+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f44e4042e50 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.863+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f44f002e190 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.864+0000 7f44ee000700 1 -- 172.21.15.52:0/3300906446 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f44e4031020 con 0x7f44f0106e20 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.865+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 msgr2=0x7f44f019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:54.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.865+0000 7f44f8200700 1 --2- 172.21.15.52:0/3300906446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f44e4011ac0 tx=0x7f44e4011ba0 comp rx=0 tx=0).stop 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.865+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 shutdown_connections 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.865+0000 7f44f8200700 1 --2- 172.21.15.52:0/3300906446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44f0106e20 0x7f44f019e650 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.865+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 >> 172.21.15.52:0/3300906446 conn(0x7f44f0102a40 msgr2=0x7f44f0195290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.866+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 shutdown_connections 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:53.866+0000 7f44f8200700 1 -- 172.21.15.52:0/3300906446 wait complete. 2024-04-26T01:31:54.545 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (3/15)... 2024-04-26T01:31:55.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:54.926+0000 7fe92ec2a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:31:55.217+0000 7fe92ec2a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: Activating manager daemon x 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: mgrmap e2: x(active, starting, since 0.0100277s) 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:31:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: Manager daemon x is now available 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' 2024-04-26T01:31:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:55 smithi052 ceph-mon[27851]: from='mgr.14100 172.21.15.52:0/3005858992' entity='mgr.x' 2024-04-26T01:31:57.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:57 smithi052 ceph-mon[27851]: mgrmap e3: x(active, since 1.01806s) 2024-04-26T01:31:58.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T01:31:58.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T01:31:58.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T01:31:58.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.169 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T01:31:58.172 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T01:31:58.172 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T01:31:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T01:31:27.359437+0000", 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-04-26T01:31:58.174 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.159+0000 7f7fee600700 1 Processor -- start 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.160+0000 7f7fee600700 1 -- start start 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.160+0000 7f7fee600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.160+0000 7f7fee600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8108f30 con 0x7f7fe8108540 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.160+0000 7f7fe7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:58.175 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.161+0000 7f7fe7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49364/0 (socket says 172.21.15.52:49364) 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.161+0000 7f7fe7400700 1 -- 172.21.15.52:0/2115257746 learned_addr learned my addr 172.21.15.52:0/2115257746 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.161+0000 7f7fe7400700 1 -- 172.21.15.52:0/2115257746 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe81096f0 con 0x7f7fe8108540 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.161+0000 7f7fe7400700 1 --2- 172.21.15.52:0/2115257746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80098a0 tx=0x7f7fd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2111dd8eb38a9be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.162+0000 7f7fe6000700 1 -- 172.21.15.52:0/2115257746 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8004160 con 0x7f7fe8108540 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.162+0000 7f7fe6000700 1 -- 172.21.15.52:0/2115257746 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7fd8004300 con 0x7f7fe8108540 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.162+0000 7f7fe6000700 1 -- 172.21.15.52:0/2115257746 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8011af0 con 0x7f7fe8108540 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.162+0000 7f7fee600700 1 -- 172.21.15.52:0/2115257746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 msgr2=0x7f7fe8108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:58.180 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.162+0000 7f7fee600700 1 --2- 172.21.15.52:0/2115257746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80098a0 tx=0x7f7fd8009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.163+0000 7f7fee600700 1 -- 172.21.15.52:0/2115257746 shutdown_connections 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.163+0000 7f7fee600700 1 --2- 172.21.15.52:0/2115257746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe8108960 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.163+0000 7f7fee600700 1 -- 172.21.15.52:0/2115257746 >> 172.21.15.52:0/2115257746 conn(0x7f7fe8103b40 msgr2=0x7f7fe8105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.163+0000 7f7fee600700 1 -- 172.21.15.52:0/2115257746 shutdown_connections 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.163+0000 7f7fee600700 1 -- 172.21.15.52:0/2115257746 wait complete. 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.164+0000 7f7fee600700 1 Processor -- start 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.164+0000 7f7fee600700 1 -- start start 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.164+0000 7f7fee600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.164+0000 7f7fee600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe810caa0 con 0x7f7fe8108540 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.165+0000 7f7fe7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.165+0000 7f7fe7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49372/0 (socket says 172.21.15.52:49372) 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.165+0000 7f7fe7400700 1 -- 172.21.15.52:0/1912704271 learned_addr learned my addr 172.21.15.52:0/1912704271 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.165+0000 7f7fe7400700 1 -- 172.21.15.52:0/1912704271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe819ec50 con 0x7f7fe8108540 2024-04-26T01:31:58.181 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.165+0000 7f7fe7400700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80117b0 tx=0x7f7fd8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.166+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8003af0 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.166+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7fd8020070 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.166+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8019630 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.166+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe819eee0 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.166+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe81a1bd0 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.167+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f7fd801a430 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.167+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe80640a0 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.168+0000 7f7fdfe00700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 0x7f7fcc03b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.168+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7fd804d880 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.168+0000 7f7fe6a00700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 0x7f7fcc03b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.169+0000 7f7fe6a00700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 0x7f7fcc03b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4006fd0 tx=0x7f7fd4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.174+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fd80042a0 con 0x7f7fe8108540 2024-04-26T01:31:58.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.242+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f7fd8019850 con 0x7f7fe8108540 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.494+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7fe81a1da0 con 0x7f7fe8108540 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.495+0000 7f7fdfe00700 1 -- 172.21.15.52:0/1912704271 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7fe81a1da0 con 0x7f7fe8108540 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 msgr2=0x7f7fcc03b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 0x7f7fcc03b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4006fd0 tx=0x7f7fd4006e40 comp rx=0 tx=0).stop 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 msgr2=0x7f7fe819e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7fd80117b0 tx=0x7f7fd8011b70 comp rx=0 tx=0).stop 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 shutdown_connections 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f7fcc039490 0x7f7fcc03b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 --2- 172.21.15.52:0/1912704271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fe8108540 0x7f7fe819e710 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 >> 172.21.15.52:0/1912704271 conn(0x7f7fe8103b40 msgr2=0x7f7fe8104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.499+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 shutdown_connections 2024-04-26T01:31:58.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:57.500+0000 7f7fee600700 1 -- 172.21.15.52:0/1912704271 wait complete. 2024-04-26T01:31:58.184 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2024-04-26T01:31:58.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:58 smithi052 ceph-mon[27851]: mgrmap e4: x(active, since 2s) 2024-04-26T01:31:58.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:58 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1912704271' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T01:31:59.387 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:31:59 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/855402085' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T01:31:59.710 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:59.710 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T01:31:59.711 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T01:31:59.715 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.817+0000 7f4cc4800700 1 Processor -- start 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.818+0000 7f4cc4800700 1 -- start start 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.818+0000 7f4cc4800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.818+0000 7f4cc4800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc10a390 con 0x7f4cbc10be00 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.819+0000 7f4cc1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.819+0000 7f4cc1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49384/0 (socket says 172.21.15.52:49384) 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.819+0000 7f4cc1e00700 1 -- 172.21.15.52:0/2237075092 learned_addr learned my addr 172.21.15.52:0/2237075092 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.819+0000 7f4cc1e00700 1 -- 172.21.15.52:0/2237075092 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc10a510 con 0x7f4cbc10be00 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.819+0000 7f4cc1e00700 1 --2- 172.21.15.52:0/2237075092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00098a0 tx=0x7f4cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69eb4094800e5d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.820+0000 7f4cc0a00700 1 -- 172.21.15.52:0/2237075092 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cb0004160 con 0x7f4cbc10be00 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.820+0000 7f4cc0a00700 1 -- 172.21.15.52:0/2237075092 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4cb0004300 con 0x7f4cbc10be00 2024-04-26T01:31:59.716 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.820+0000 7f4cc0a00700 1 -- 172.21.15.52:0/2237075092 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cb0011af0 con 0x7f4cbc10be00 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 -- 172.21.15.52:0/2237075092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 msgr2=0x7f4cbc10c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 --2- 172.21.15.52:0/2237075092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00098a0 tx=0x7f4cb0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 -- 172.21.15.52:0/2237075092 shutdown_connections 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 --2- 172.21.15.52:0/2237075092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc10c220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 -- 172.21.15.52:0/2237075092 >> 172.21.15.52:0/2237075092 conn(0x7f4cbc07dc00 msgr2=0x7f4cbc1084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 -- 172.21.15.52:0/2237075092 shutdown_connections 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.821+0000 7f4cc4800700 1 -- 172.21.15.52:0/2237075092 wait complete. 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.822+0000 7f4cc4800700 1 Processor -- start 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.822+0000 7f4cc4800700 1 -- start start 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc4800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc4800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc10ecd0 con 0x7f4cbc10be00 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:59.717 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49392/0 (socket says 172.21.15.52:49392) 2024-04-26T01:31:59.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc1e00700 1 -- 172.21.15.52:0/855402085 learned_addr learned my addr 172.21.15.52:0/855402085 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:31:59.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc1e00700 1 -- 172.21.15.52:0/855402085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc19ed20 con 0x7f4cbc10be00 2024-04-26T01:31:59.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.823+0000 7f4cc1e00700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00117b0 tx=0x7f4cb0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:59.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.824+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cb0003af0 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.824+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4cb0020070 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.824+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cb0019630 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.824+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc19efb0 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.824+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc1a1ca0 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.825+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cbc10ee50 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.825+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4cb001a9c0 con 0x7f4cbc10be00 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.826+0000 7f4cb6a00700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 0x7f4ca403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:31:59.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.826+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4cb004dae0 con 0x7f4cbc10be00 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.826+0000 7f4cc1400700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 0x7f4ca403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.827+0000 7f4cc1400700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 0x7f4ca403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:58.831+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4cb000e260 con 0x7f4cbc10be00 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.061+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4cbc1a1e70 con 0x7f4cbc10be00 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.064+0000 7f4cb6a00700 1 -- 172.21.15.52:0/855402085 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7f4cb00042a0 con 0x7f4cbc10be00 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.067+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 msgr2=0x7f4ca403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.067+0000 7f4cc4800700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 0x7f4ca403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac006e40 comp rx=0 tx=0).stop 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.067+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 msgr2=0x7f4cbc19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.067+0000 7f4cc4800700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00117b0 tx=0x7f4cb0011b70 comp rx=0 tx=0).stop 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 shutdown_connections 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f4ca40395b0 0x7f4ca403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 --2- 172.21.15.52:0/855402085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cbc10be00 0x7f4cbc19e7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 >> 172.21.15.52:0/855402085 conn(0x7f4cbc07dc00 msgr2=0x7f4cbc107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 shutdown_connections 2024-04-26T01:31:59.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:31:59.068+0000 7f4cc4800700 1 -- 172.21.15.52:0/855402085 wait complete. 2024-04-26T01:31:59.722 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2024-04-26T01:32:00.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:00 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3874873575' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T01:32:00.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setuser ceph since I am not root 2024-04-26T01:32:00.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setgroup ceph since I am not root 2024-04-26T01:32:01.371 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:01.024+0000 7f106e17f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:32:01.371 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:01.268+0000 7f106e17f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:01.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.380+0000 7f94f4a00700 1 Processor -- start 2024-04-26T01:32:01.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.380+0000 7f94f4a00700 1 -- start start 2024-04-26T01:32:01.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.381+0000 7f94f4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:01.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.381+0000 7f94f4a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f94f0108d40 con 0x7f94f0108350 2024-04-26T01:32:01.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.381+0000 7f94ee800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:01.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.381+0000 7f94ee800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49402/0 (socket says 172.21.15.52:49402) 2024-04-26T01:32:01.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.382+0000 7f94ee800700 1 -- 172.21.15.52:0/1265740018 learned_addr learned my addr 172.21.15.52:0/1265740018 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.382+0000 7f94ee800700 1 -- 172.21.15.52:0/1265740018 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f0109550 con 0x7f94f0108350 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.382+0000 7f94ee800700 1 --2- 172.21.15.52:0/1265740018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098a0 tx=0x7f94e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53af6b5b74a63763 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94ed400700 1 -- 172.21.15.52:0/1265740018 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94e0004160 con 0x7f94f0108350 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94ed400700 1 -- 172.21.15.52:0/1265740018 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f94e0004300 con 0x7f94f0108350 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94ed400700 1 -- 172.21.15.52:0/1265740018 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94e0011af0 con 0x7f94f0108350 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94f4a00700 1 -- 172.21.15.52:0/1265740018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 msgr2=0x7f94f0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94f4a00700 1 --2- 172.21.15.52:0/1265740018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098a0 tx=0x7f94e0009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94f4a00700 1 -- 172.21.15.52:0/1265740018 shutdown_connections 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.383+0000 7f94f4a00700 1 --2- 172.21.15.52:0/1265740018 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f0108770 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.384+0000 7f94f4a00700 1 -- 172.21.15.52:0/1265740018 >> 172.21.15.52:0/1265740018 conn(0x7f94f0103910 msgr2=0x7f94f0105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:01.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.384+0000 7f94f4a00700 1 -- 172.21.15.52:0/1265740018 shutdown_connections 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.384+0000 7f94f4a00700 1 -- 172.21.15.52:0/1265740018 wait complete. 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94f4a00700 1 Processor -- start 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94f4a00700 1 -- start start 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94f4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94f4a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f94f010c900 con 0x7f94f0108350 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94ee800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94ee800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49404/0 (socket says 172.21.15.52:49404) 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.385+0000 7f94ee800700 1 -- 172.21.15.52:0/3874873575 learned_addr learned my addr 172.21.15.52:0/3874873575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:01.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.386+0000 7f94ee800700 1 -- 172.21.15.52:0/3874873575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f019c860 con 0x7f94f0108350 2024-04-26T01:32:01.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.386+0000 7f94ee800700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94e0000c00 tx=0x7f94e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:01.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.386+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94e0003b20 con 0x7f94f0108350 2024-04-26T01:32:01.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.386+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f94e001a8c0 con 0x7f94f0108350 2024-04-26T01:32:01.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.386+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f019caf0 con 0x7f94f0108350 2024-04-26T01:32:01.390 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.387+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94e0024940 con 0x7f94f0108350 2024-04-26T01:32:01.390 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.387+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f019f7e0 con 0x7f94f0108350 2024-04-26T01:32:01.390 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.388+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f94e0022020 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.388+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f0069ba0 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.388+0000 7f94e7400700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 0x7f94d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.388+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f94e004d400 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.389+0000 7f94ede00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 0x7f94d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.389+0000 7f94ede00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 0x7f94d403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94dc006fd0 tx=0x7f94dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.394+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94e004d160 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.687+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f94f019f9b0 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.740+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f94f019f9b0 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.743+0000 7f94e7400700 1 -- 172.21.15.52:0/3874873575 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f94e002d3f0 con 0x7f94f0108350 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 msgr2=0x7f94d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 0x7f94d403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94dc006fd0 tx=0x7f94dc006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 msgr2=0x7f94f019c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f94e0000c00 tx=0x7f94e0011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:01.391 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 shutdown_connections 2024-04-26T01:32:01.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7f94d40395b0 0x7f94d403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:01.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 --2- 172.21.15.52:0/3874873575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108350 0x7f94f019c320 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:01.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 >> 172.21.15.52:0/3874873575 conn(0x7f94f0103910 msgr2=0x7f94f01044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:01.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 shutdown_connections 2024-04-26T01:32:01.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:00.745+0000 7f94f4a00700 1 -- 172.21.15.52:0/3874873575 wait complete. 2024-04-26T01:32:01.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:01.407+0000 7f106e17f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:32:02.007 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3874873575' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T01:32:02.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-mon[27851]: mgrmap e5: x(active, since 5s) 2024-04-26T01:32:02.008 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:01.819+0000 7f106e17f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:32:02.008 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:01.937+0000 7f106e17f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:32:02.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:02 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/142263628' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:32:02.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:02.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T01:32:02.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:32:02.973 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:32:02.973 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.011+0000 7fea87200700 1 Processor -- start 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.011+0000 7fea87200700 1 -- start start 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.012+0000 7fea87200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.012+0000 7fea87200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fea88108d30 con 0x7fea88108340 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.012+0000 7fea85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:02.974 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.012+0000 7fea85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49430/0 (socket says 172.21.15.52:49430) 2024-04-26T01:32:02.978 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.012+0000 7fea85e00700 1 -- 172.21.15.52:0/634479469 learned_addr learned my addr 172.21.15.52:0/634479469 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:02.978 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.013+0000 7fea85e00700 1 -- 172.21.15.52:0/634479469 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea88109540 con 0x7fea88108340 2024-04-26T01:32:02.978 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.013+0000 7fea85e00700 1 --2- 172.21.15.52:0/634479469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacb0aa34525387f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:02.978 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.013+0000 7fea84a00700 1 -- 172.21.15.52:0/634479469 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78004160 con 0x7fea88108340 2024-04-26T01:32:02.978 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea84a00700 1 -- 172.21.15.52:0/634479469 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea78004300 con 0x7fea88108340 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea84a00700 1 -- 172.21.15.52:0/634479469 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78011af0 con 0x7fea88108340 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 -- 172.21.15.52:0/634479469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 msgr2=0x7fea88108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 --2- 172.21.15.52:0/634479469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 -- 172.21.15.52:0/634479469 shutdown_connections 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 --2- 172.21.15.52:0/634479469 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea88108760 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 -- 172.21.15.52:0/634479469 >> 172.21.15.52:0/634479469 conn(0x7fea88103940 msgr2=0x7fea88105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 -- 172.21.15.52:0/634479469 shutdown_connections 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.014+0000 7fea87200700 1 -- 172.21.15.52:0/634479469 wait complete. 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.015+0000 7fea87200700 1 Processor -- start 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea87200700 1 -- start start 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea87200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea87200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810c8f0 con 0x7fea88108340 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:02.979 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49446/0 (socket says 172.21.15.52:49446) 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea85e00700 1 -- 172.21.15.52:0/142263628 learned_addr learned my addr 172.21.15.52:0/142263628 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.016+0000 7fea85e00700 1 -- 172.21.15.52:0/142263628 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8819c830 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea85e00700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea78000c00 tx=0x7fea78011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78003b20 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea7801a8c0 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea78024940 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8819cac0 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.017+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8819f7b0 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.018+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea88069ba0 con 0x7fea88108340 2024-04-26T01:32:02.980 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.019+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fea78022020 con 0x7fea88108340 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.019+0000 7fea7ea00700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 0x7fea6c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.019+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea7804d450 con 0x7fea88108340 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.019+0000 7fea85400700 1 -- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 msgr2=0x7fea6c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.019+0000 7fea85400700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 0x7fea6c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.025+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea7804d1b0 con 0x7fea88108340 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.220+0000 7fea85400700 1 -- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 msgr2=0x7fea6c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:02.981 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.220+0000 7fea85400700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 0x7fea6c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.314+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fea8819f980 con 0x7fea88108340 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.315+0000 7fea7ea00700 1 -- 172.21.15.52:0/142263628 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fea8819f980 con 0x7fea88108340 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.318+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 msgr2=0x7fea6c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.318+0000 7fea87200700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 0x7fea6c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 msgr2=0x7fea8819c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fea78000c00 tx=0x7fea78011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 shutdown_connections 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fea6c039600 0x7fea6c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 --2- 172.21.15.52:0/142263628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea88108340 0x7fea8819c2f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 >> 172.21.15.52:0/142263628 conn(0x7fea88103940 msgr2=0x7fea881044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 shutdown_connections 2024-04-26T01:32:02.983 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:02.319+0000 7fea87200700 1 -- 172.21.15.52:0/142263628 wait complete. 2024-04-26T01:32:02.984 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-04-26T01:32:02.984 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2024-04-26T01:32:03.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:03.536+0000 7f106e17f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:32:03.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:03.770+0000 7f106e17f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:04.379 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:03.990+0000 7f106e17f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:32:04.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:04.383+0000 7f106e17f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:32:04.709 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:04.510+0000 7f106e17f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:32:05.029 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:04.768+0000 7f106e17f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:32:05.029 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:05.033+0000 7f106e17f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:06.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:05.845+0000 7f106e17f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:32:06.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:06.415+0000 7f106e17f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:32:06.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:06.545+0000 7f106e17f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:32:07.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:06.821+0000 7f106e17f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:32:07.942 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:07.487+0000 7f106e17f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:32:07.943 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:07.613+0000 7f106e17f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:32:08.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:07.947+0000 7f106e17f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:32:08.209 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:08 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:08.129+0000 7f106e17f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:32:08.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:08 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:08.260+0000 7f106e17f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:32:08.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:08 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:08.416+0000 7f106e17f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:09.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:08 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:08.790+0000 7f106e17f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:32:09.866 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:09.578+0000 7f106e17f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:32:10.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:09.871+0000 7f106e17f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: Active manager daemon x restarted 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: Activating manager daemon x 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: mgrmap e6: x(active, starting, since 0.004916s) 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: Manager daemon x is now available 2024-04-26T01:32:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:09 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: Found migration_current of "None". Setting to last migration. 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:10 smithi052 ceph-mon[27851]: mgrmap e7: x(active, since 1.00947s) 2024-04-26T01:32:11.610 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:11.611 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T01:32:11.611 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:32:11.611 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:11.612 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.553+0000 7fc162400700 1 Processor -- start 2024-04-26T01:32:11.612 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.553+0000 7fc162400700 1 -- start start 2024-04-26T01:32:11.612 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.553+0000 7fc162400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:11.612 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.553+0000 7fc162400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c07b7b0 con 0x7fc15c07cd80 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.554+0000 7fc15b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.554+0000 7fc15b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49450/0 (socket says 172.21.15.52:49450) 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.554+0000 7fc15b400700 1 -- 172.21.15.52:0/2151368616 learned_addr learned my addr 172.21.15.52:0/2151368616 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.554+0000 7fc15b400700 1 -- 172.21.15.52:0/2151368616 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c07b930 con 0x7fc15c07cd80 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.555+0000 7fc15b400700 1 --2- 172.21.15.52:0/2151368616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098a0 tx=0x7fc14c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1928c63a233997df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.555+0000 7fc15a000700 1 -- 172.21.15.52:0/2151368616 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc14c004160 con 0x7fc15c07cd80 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.555+0000 7fc15a000700 1 -- 172.21.15.52:0/2151368616 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc14c004300 con 0x7fc15c07cd80 2024-04-26T01:32:11.613 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.555+0000 7fc15a000700 1 -- 172.21.15.52:0/2151368616 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc14c011af0 con 0x7fc15c07cd80 2024-04-26T01:32:11.616 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 -- 172.21.15.52:0/2151368616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 msgr2=0x7fc15c07b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:11.616 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 --2- 172.21.15.52:0/2151368616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098a0 tx=0x7fc14c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.616 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 -- 172.21.15.52:0/2151368616 shutdown_connections 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 --2- 172.21.15.52:0/2151368616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c07b1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 -- 172.21.15.52:0/2151368616 >> 172.21.15.52:0/2151368616 conn(0x7fc15c103b50 msgr2=0x7fc15c105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 -- 172.21.15.52:0/2151368616 shutdown_connections 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.556+0000 7fc162400700 1 -- 172.21.15.52:0/2151368616 wait complete. 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.557+0000 7fc162400700 1 Processor -- start 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.557+0000 7fc162400700 1 -- start start 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc162400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc162400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c10eb00 con 0x7fc15c07cd80 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc15b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc15b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49460/0 (socket says 172.21.15.52:49460) 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc15b400700 1 -- 172.21.15.52:0/2021435330 learned_addr learned my addr 172.21.15.52:0/2021435330 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc15b400700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c19eb70 con 0x7fc15c07cd80 2024-04-26T01:32:11.617 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.558+0000 7fc15b400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc14c011ac0 tx=0x7fc14c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.559+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc14c003b20 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.559+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc14c01a8c0 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.559+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc14c0248d0 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.559+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c19ee00 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.559+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c1a1af0 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.560+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc14c022020 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.560+0000 7fc153e00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.560+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc14003c1d0 con 0x7fc140039600 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.560+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc14c04d4e0 con 0x7fc15c07cd80 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.561+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.561+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:32:11.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.762+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:03.762+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:32:11.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:04.163+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:04.163+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:32:11.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:04.964+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:04.964+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:06.566+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:06.566+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:09.769+0000 7fc15aa00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/496114326 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:09.769+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:09.878+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fc14c02d3f0 con 0x7fc15c07cd80 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:09.878+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 msgr2=0x7fc14003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:09.878+0000 7fc153e00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.882+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fc14c04ee40 con 0x7fc15c07cd80 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.883+0000 7fc153e00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 0x7fc14003f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.883+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc14c007da0 con 0x7fc14003cef0 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.884+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 0x7fc14003f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.888+0000 7fc15aa00700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 0x7fc14003f2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc148003a10 tx=0x7fc1480092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:11.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.889+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fc14c007da0 con 0x7fc14003cef0 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.895+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc15c069ba0 con 0x7fc14003cef0 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc153e00700 1 -- 172.21.15.52:0/2021435330 <== mgr.14120 v2:172.21.15.52:6800/3772941101 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc15c069ba0 con 0x7fc14003cef0 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 msgr2=0x7fc14003f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 0x7fc14003f2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc148003a10 tx=0x7fc1480092b0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 msgr2=0x7fc15c19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc14c011ac0 tx=0x7fc14c011ba0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 shutdown_connections 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fc14003cef0 0x7fc14003f2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c07cd80 0x7fc15c19e630 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 --2- 172.21.15.52:0/2021435330 >> [v2:172.21.15.52:6800/496114326,v1:172.21.15.52:6801/496114326] conn(0x7fc140039600 0x7fc14003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 >> 172.21.15.52:0/2021435330 conn(0x7fc15c103b50 msgr2=0x7fc15c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.896+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 shutdown_connections 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:10.897+0000 7fc162400700 1 -- 172.21.15.52:0/2021435330 wait complete. 2024-04-26T01:32:11.624 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2024-04-26T01:32:11.625 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:10] ENGINE Bus STARTING 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:10] ENGINE Serving on https://172.21.15.52:7150 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:11] ENGINE Serving on http://172.21.15.52:8765 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:11] ENGINE Bus STARTED 2024-04-26T01:32:12.177 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:11 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:13.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:12 smithi052 ceph-mon[27851]: mgrmap e8: x(active, since 2s) 2024-04-26T01:32:13.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:12 smithi052 ceph-mon[27851]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:13.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:12 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:13.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:12 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:13.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.219+0000 7f81cf200700 1 Processor -- start 2024-04-26T01:32:13.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.220+0000 7f81cf200700 1 -- start start 2024-04-26T01:32:13.182 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.220+0000 7f81cf200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.220+0000 7f81cf200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f81d0108d40 con 0x7f81d0108350 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.220+0000 7f81cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.220+0000 7f81cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36916/0 (socket says 172.21.15.52:36916) 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.221+0000 7f81cde00700 1 -- 172.21.15.52:0/235011495 learned_addr learned my addr 172.21.15.52:0/235011495 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.221+0000 7f81cde00700 1 -- 172.21.15.52:0/235011495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d0109550 con 0x7f81d0108350 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.221+0000 7f81cde00700 1 --2- 172.21.15.52:0/235011495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e80 tx=0x7f81c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab830cc8a36f0893 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cca00700 1 -- 172.21.15.52:0/235011495 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c0017070 con 0x7f81d0108350 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cca00700 1 -- 172.21.15.52:0/235011495 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81c000b8c0 con 0x7f81d0108350 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cca00700 1 -- 172.21.15.52:0/235011495 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c0004490 con 0x7f81d0108350 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cf200700 1 -- 172.21.15.52:0/235011495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 msgr2=0x7f81d0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cf200700 1 --2- 172.21.15.52:0/235011495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e80 tx=0x7f81c000b330 comp rx=0 tx=0).stop 2024-04-26T01:32:13.183 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cf200700 1 -- 172.21.15.52:0/235011495 shutdown_connections 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cf200700 1 --2- 172.21.15.52:0/235011495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d0108770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.222+0000 7f81cf200700 1 -- 172.21.15.52:0/235011495 >> 172.21.15.52:0/235011495 conn(0x7f81d0103990 msgr2=0x7f81d0105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.223+0000 7f81cf200700 1 -- 172.21.15.52:0/235011495 shutdown_connections 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.223+0000 7f81cf200700 1 -- 172.21.15.52:0/235011495 wait complete. 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.224+0000 7f81cf200700 1 Processor -- start 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.224+0000 7f81cf200700 1 -- start start 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.224+0000 7f81cf200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.224+0000 7f81cf200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f81d010c900 con 0x7f81d0108350 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.224+0000 7f81cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.225+0000 7f81cde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36928/0 (socket says 172.21.15.52:36928) 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.225+0000 7f81cde00700 1 -- 172.21.15.52:0/1381704123 learned_addr learned my addr 172.21.15.52:0/1381704123 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:13.184 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.225+0000 7f81cde00700 1 -- 172.21.15.52:0/1381704123 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d019e9e0 con 0x7f81d0108350 2024-04-26T01:32:13.185 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.225+0000 7f81cde00700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f81c0004a60 tx=0x7f81c000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:13.185 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.225+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c0020020 con 0x7f81d0108350 2024-04-26T01:32:13.185 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.226+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81c0003c60 con 0x7f81d0108350 2024-04-26T01:32:13.185 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.226+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d019ec70 con 0x7f81d0108350 2024-04-26T01:32:13.185 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.226+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81c0017040 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.226+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d01a1960 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.227+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f81c0023020 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.227+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d0069ba0 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.227+0000 7f81c6a00700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 0x7f81b403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.227+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f81c000f050 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.228+0000 7f81cd400700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 0x7f81b403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.228+0000 7f81cd400700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 0x7f81b403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81bc006fd0 tx=0x7f81bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.235+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81c002a020 con 0x7f81d0108350 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.486+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f81d0104ae0 con 0x7f81b4039600 2024-04-26T01:32:13.186 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.495+0000 7f81c6a00700 1 -- 172.21.15.52:0/1381704123 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f81d0104ae0 con 0x7f81b4039600 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 msgr2=0x7f81b403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 0x7f81b403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f81bc006fd0 tx=0x7f81bc006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 msgr2=0x7f81d019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f81c0004a60 tx=0x7f81c000bd50 comp rx=0 tx=0).stop 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 shutdown_connections 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f81b4039600 0x7f81b403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.498+0000 7f81cf200700 1 --2- 172.21.15.52:0/1381704123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f81d0108350 0x7f81d019e4a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.499+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 >> 172.21.15.52:0/1381704123 conn(0x7f81d0103990 msgr2=0x7f81d01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.499+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 shutdown_connections 2024-04-26T01:32:13.187 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:12.499+0000 7f81cf200700 1 -- 172.21.15.52:0/1381704123 wait complete. 2024-04-26T01:32:14.699 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T01:32:14.700 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.797+0000 7f971f400700 1 Processor -- start 2024-04-26T01:32:14.700 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.797+0000 7f971f400700 1 -- start start 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.798+0000 7f971f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.798+0000 7f971f400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f971810b150 con 0x7f971810a760 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.798+0000 7f971ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.798+0000 7f971ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36938/0 (socket says 172.21.15.52:36938) 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.798+0000 7f971ca00700 1 -- 172.21.15.52:0/79807960 learned_addr learned my addr 172.21.15.52:0/79807960 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.799+0000 7f971ca00700 1 -- 172.21.15.52:0/79807960 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971810b910 con 0x7f971810a760 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.799+0000 7f971ca00700 1 --2- 172.21.15.52:0/79807960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f970c0098a0 tx=0x7f970c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914d2cd47e421383 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.799+0000 7f9717400700 1 -- 172.21.15.52:0/79807960 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f970c004160 con 0x7f971810a760 2024-04-26T01:32:14.701 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.799+0000 7f9717400700 1 -- 172.21.15.52:0/79807960 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f970c004300 con 0x7f971810a760 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f9717400700 1 -- 172.21.15.52:0/79807960 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f970c011af0 con 0x7f971810a760 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f971f400700 1 -- 172.21.15.52:0/79807960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 msgr2=0x7f971810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f971f400700 1 --2- 172.21.15.52:0/79807960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f970c0098a0 tx=0x7f970c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f971f400700 1 -- 172.21.15.52:0/79807960 shutdown_connections 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f971f400700 1 --2- 172.21.15.52:0/79807960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971810ab80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.800+0000 7f971f400700 1 -- 172.21.15.52:0/79807960 >> 172.21.15.52:0/79807960 conn(0x7f971807dc20 msgr2=0x7f97181084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.801+0000 7f971f400700 1 -- 172.21.15.52:0/79807960 shutdown_connections 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.801+0000 7f971f400700 1 -- 172.21.15.52:0/79807960 wait complete. 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971f400700 1 Processor -- start 2024-04-26T01:32:14.706 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971f400700 1 -- start start 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971f400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f971810ecc0 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36942/0 (socket says 172.21.15.52:36942) 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.802+0000 7f971ca00700 1 -- 172.21.15.52:0/3808508796 learned_addr learned my addr 172.21.15.52:0/3808508796 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f971ca00700 1 -- 172.21.15.52:0/3808508796 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971819ed30 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f971ca00700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f970c0117b0 tx=0x7f970c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f970c003af0 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f970c020070 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f970c019630 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.803+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971819efc0 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.804+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f971819fae0 con 0x7f971810a760 2024-04-26T01:32:14.707 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.805+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f970c01a9c0 con 0x7f971810a760 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.805+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9718069ba0 con 0x7f971810a760 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.805+0000 7f9715600700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 0x7f970003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.805+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f970c04dbf0 con 0x7f971810a760 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.806+0000 7f9717e00700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 0x7f970003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.806+0000 7f9717e00700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 0x7f970003bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:13.811+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f970c00e260 con 0x7f971810a760 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.045+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9718108390 con 0x7f9700039660 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.046+0000 7f9715600700 1 -- 172.21.15.52:0/3808508796 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9718108390 con 0x7f9700039660 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 msgr2=0x7f970003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 0x7f970003bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 msgr2=0x7f971819e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f970c0117b0 tx=0x7f970c011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:14.708 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 shutdown_connections 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f9700039660 0x7f970003bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 --2- 172.21.15.52:0/3808508796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f971810a760 0x7f971819e7f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.049+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 >> 172.21.15.52:0/3808508796 conn(0x7f971807dc20 msgr2=0x7f9718107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.051+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 shutdown_connections 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:14.051+0000 7f971f400700 1 -- 172.21.15.52:0/3808508796 wait complete. 2024-04-26T01:32:14.709 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2024-04-26T01:32:14.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:14 smithi052 ceph-mon[27851]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:15 smithi052 ceph-mon[27851]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:15 smithi052 ceph-mon[27851]: Generating ssh key... 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: Generating public/private rsa key pair. 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: Your identification has been saved in /tmp/tmpffu_qse6/key. 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: Your public key has been saved in /tmp/tmpffu_qse6/key.pub. 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: The key fingerprint is: 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: SHA256:sgZwMb/jYnXS4m3RFB6kA76G6yKFi/TswjOSb/ZvjTE ceph-8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: The key's randomart image is: 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: +---[RSA 3072]----+ 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: | o . .+ | 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: | = . o o | 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: | . . o o o | 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: | o . + + | 2024-04-26T01:32:16.946 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: | . o X S . | 2024-04-26T01:32:16.947 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: |... BEO . | 2024-04-26T01:32:16.947 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: |++o + ==o | 2024-04-26T01:32:16.947 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: |*=+= oo.. | 2024-04-26T01:32:16.947 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: |.=*+oo. | 2024-04-26T01:32:16.947 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: +----[SHA256]-----+ 2024-04-26T01:32:17.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.296+0000 7f3558000700 1 Processor -- start 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.296+0000 7f3558000700 1 -- start start 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.296+0000 7f3558000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.296+0000 7f3558000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3550108ef0 con 0x7f3550108500 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.297+0000 7f3555600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.297+0000 7f3555600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32826/0 (socket says 172.21.15.52:32826) 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.297+0000 7f3555600700 1 -- 172.21.15.52:0/2828822575 learned_addr learned my addr 172.21.15.52:0/2828822575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:17.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.297+0000 7f3555600700 1 -- 172.21.15.52:0/2828822575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3550109700 con 0x7f3550108500 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.298+0000 7f3555600700 1 --2- 172.21.15.52:0/2828822575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35440098a0 tx=0x7f3544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fc2d231201c185 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.298+0000 7f354fe00700 1 -- 172.21.15.52:0/2828822575 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544004160 con 0x7f3550108500 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.298+0000 7f354fe00700 1 -- 172.21.15.52:0/2828822575 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3544004300 con 0x7f3550108500 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.298+0000 7f354fe00700 1 -- 172.21.15.52:0/2828822575 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544011af0 con 0x7f3550108500 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 -- 172.21.15.52:0/2828822575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 msgr2=0x7f3550108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 --2- 172.21.15.52:0/2828822575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35440098a0 tx=0x7f3544009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 -- 172.21.15.52:0/2828822575 shutdown_connections 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 --2- 172.21.15.52:0/2828822575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f3550108920 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 -- 172.21.15.52:0/2828822575 >> 172.21.15.52:0/2828822575 conn(0x7f3550103b00 msgr2=0x7f3550105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:17.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 -- 172.21.15.52:0/2828822575 shutdown_connections 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.299+0000 7f3558000700 1 -- 172.21.15.52:0/2828822575 wait complete. 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.300+0000 7f3558000700 1 Processor -- start 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.300+0000 7f3558000700 1 -- start start 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3558000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3558000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f355010cab0 con 0x7f3550108500 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3555600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3555600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32834/0 (socket says 172.21.15.52:32834) 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3555600700 1 -- 172.21.15.52:0/4088379387 learned_addr learned my addr 172.21.15.52:0/4088379387 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.301+0000 7f3555600700 1 -- 172.21.15.52:0/4088379387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355019ca20 con 0x7f3550108500 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f3555600700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f35440117b0 tx=0x7f3544011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544003af0 con 0x7f3550108500 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3544020070 con 0x7f3550108500 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544019630 con 0x7f3550108500 2024-04-26T01:32:17.234 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355019ccb0 con 0x7f3550108500 2024-04-26T01:32:17.235 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.302+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355019f9a0 con 0x7f3550108500 2024-04-26T01:32:17.235 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.303+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f354401a9c0 con 0x7f3550108500 2024-04-26T01:32:17.235 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.303+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35500640a0 con 0x7f3550108500 2024-04-26T01:32:17.235 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.304+0000 7f354e000700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 0x7f353803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:17.235 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.304+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f354404db30 con 0x7f3550108500 2024-04-26T01:32:17.236 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.304+0000 7f3554c00700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 0x7f353803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:17.236 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.305+0000 7f3554c00700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 0x7f353803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f353c006fd0 tx=0x7f353c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:17.236 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.309+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f354400e260 con 0x7f3550108500 2024-04-26T01:32:17.236 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:15.536+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3550104db0 con 0x7f3538039600 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.615+0000 7f354e000700 1 -- 172.21.15.52:0/4088379387 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3550104db0 con 0x7f3538039600 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 msgr2=0x7f353803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 0x7f353803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f353c006fd0 tx=0x7f353c006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 msgr2=0x7f355019c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f35440117b0 tx=0x7f3544011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 shutdown_connections 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f3538039600 0x7f353803bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 --2- 172.21.15.52:0/4088379387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3550108500 0x7f355019c4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.618+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 >> 172.21.15.52:0/4088379387 conn(0x7f3550103b00 msgr2=0x7f35501046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.619+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 shutdown_connections 2024-04-26T01:32:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:16.619+0000 7f3558000700 1 -- 172.21.15.52:0/4088379387 wait complete. 2024-04-26T01:32:17.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:17 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:17.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:17 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:18.781 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:18.781 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.891+0000 7fb699800700 1 Processor -- start 2024-04-26T01:32:18.781 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.891+0000 7fb699800700 1 -- start start 2024-04-26T01:32:18.781 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.892+0000 7fb699800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:18.781 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.892+0000 7fb699800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb694108f20 con 0x7fb694108530 2024-04-26T01:32:18.782 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.892+0000 7fb692a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:18.782 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.892+0000 7fb692a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32846/0 (socket says 172.21.15.52:32846) 2024-04-26T01:32:18.782 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.892+0000 7fb692a00700 1 -- 172.21.15.52:0/1285531467 learned_addr learned my addr 172.21.15.52:0/1285531467 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:18.782 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.893+0000 7fb692a00700 1 -- 172.21.15.52:0/1285531467 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb694109730 con 0x7fb694108530 2024-04-26T01:32:18.783 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.893+0000 7fb692a00700 1 --2- 172.21.15.52:0/1285531467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098a0 tx=0x7fb684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49776b29d85c8873 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:18.783 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.894+0000 7fb692000700 1 -- 172.21.15.52:0/1285531467 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb684004160 con 0x7fb694108530 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.894+0000 7fb692000700 1 -- 172.21.15.52:0/1285531467 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb684004300 con 0x7fb694108530 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.894+0000 7fb692000700 1 -- 172.21.15.52:0/1285531467 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb684011af0 con 0x7fb694108530 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.894+0000 7fb699800700 1 -- 172.21.15.52:0/1285531467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 msgr2=0x7fb694108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.894+0000 7fb699800700 1 --2- 172.21.15.52:0/1285531467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098a0 tx=0x7fb684009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.895+0000 7fb699800700 1 -- 172.21.15.52:0/1285531467 shutdown_connections 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.895+0000 7fb699800700 1 --2- 172.21.15.52:0/1285531467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb694108950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.895+0000 7fb699800700 1 -- 172.21.15.52:0/1285531467 >> 172.21.15.52:0/1285531467 conn(0x7fb694103b50 msgr2=0x7fb694105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.895+0000 7fb699800700 1 -- 172.21.15.52:0/1285531467 shutdown_connections 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.895+0000 7fb699800700 1 -- 172.21.15.52:0/1285531467 wait complete. 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.896+0000 7fb699800700 1 Processor -- start 2024-04-26T01:32:18.784 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.896+0000 7fb699800700 1 -- start start 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.896+0000 7fb699800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.896+0000 7fb699800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb69410cae0 con 0x7fb694108530 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.897+0000 7fb692a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.897+0000 7fb692a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32858/0 (socket says 172.21.15.52:32858) 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.897+0000 7fb692a00700 1 -- 172.21.15.52:0/3727847390 learned_addr learned my addr 172.21.15.52:0/3727847390 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.897+0000 7fb692a00700 1 -- 172.21.15.52:0/3727847390 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69419c9c0 con 0x7fb694108530 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.897+0000 7fb692a00700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.898+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb684003b20 con 0x7fb694108530 2024-04-26T01:32:18.786 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.898+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb68401a8c0 con 0x7fb694108530 2024-04-26T01:32:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.898+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb684024940 con 0x7fb694108530 2024-04-26T01:32:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.898+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69419cc50 con 0x7fb694108530 2024-04-26T01:32:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.898+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69419f940 con 0x7fb694108530 2024-04-26T01:32:18.787 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.899+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb684022020 con 0x7fb694108530 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.899+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6940640a0 con 0x7fb694108530 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.900+0000 7fb68b400700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 0x7fb67803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.900+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb68404d920 con 0x7fb694108530 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.900+0000 7fb68be00700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 0x7fb67803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.901+0000 7fb68be00700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 0x7fb67803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb67c006fd0 tx=0x7fb67c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:17.906+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb68402c3a0 con 0x7fb694108530 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.136+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fb694104d00 con 0x7fb6780395b0 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.137+0000 7fb68b400700 1 -- 172.21.15.52:0/3727847390 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fb694104d00 con 0x7fb6780395b0 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 msgr2=0x7fb67803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 0x7fb67803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb67c006fd0 tx=0x7fb67c006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 msgr2=0x7fb69419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:18.789 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 shutdown_connections 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.140+0000 7fb699800700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fb6780395b0 0x7fb67803ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.141+0000 7fb699800700 1 --2- 172.21.15.52:0/3727847390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb694108530 0x7fb69419c480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.141+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 >> 172.21.15.52:0/3727847390 conn(0x7fb694103b50 msgr2=0x7fb6941045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.141+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 shutdown_connections 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:18.141+0000 7fb699800700 1 -- 172.21.15.52:0/3727847390 wait complete. 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T01:32:18.790 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T01:32:18.799 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2024-04-26T01:32:19.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:18 smithi052 ceph-mon[27851]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:20.026 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:19 smithi052 ceph-mon[27851]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:22.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:21 smithi052 ceph-mon[27851]: Deploying cephadm binary to smithi052 2024-04-26T01:32:26.718 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.453+0000 7fe9f9200700 1 Processor -- start 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.454+0000 7fe9f9200700 1 -- start start 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.454+0000 7fe9f9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.454+0000 7fe9f9200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f410b170 con 0x7fe9f410a780 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.455+0000 7fe9f1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.455+0000 7fe9f1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32874/0 (socket says 172.21.15.52:32874) 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.455+0000 7fe9f1e00700 1 -- 172.21.15.52:0/1401224987 learned_addr learned my addr 172.21.15.52:0/1401224987 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.455+0000 7fe9f1e00700 1 -- 172.21.15.52:0/1401224987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f410b930 con 0x7fe9f410a780 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.456+0000 7fe9f1e00700 1 --2- 172.21.15.52:0/1401224987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40098a0 tx=0x7fe9e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30615a59cd95b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.456+0000 7fe9f0a00700 1 -- 172.21.15.52:0/1401224987 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4004160 con 0x7fe9f410a780 2024-04-26T01:32:26.719 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.456+0000 7fe9f0a00700 1 -- 172.21.15.52:0/1401224987 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe9e4004300 con 0x7fe9f410a780 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.456+0000 7fe9f0a00700 1 -- 172.21.15.52:0/1401224987 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4011af0 con 0x7fe9f410a780 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 -- 172.21.15.52:0/1401224987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 msgr2=0x7fe9f410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 --2- 172.21.15.52:0/1401224987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40098a0 tx=0x7fe9e4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 -- 172.21.15.52:0/1401224987 shutdown_connections 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 --2- 172.21.15.52:0/1401224987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f410aba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 -- 172.21.15.52:0/1401224987 >> 172.21.15.52:0/1401224987 conn(0x7fe9f4105d40 msgr2=0x7fe9f4108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 -- 172.21.15.52:0/1401224987 shutdown_connections 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.457+0000 7fe9f9200700 1 -- 172.21.15.52:0/1401224987 wait complete. 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.458+0000 7fe9f9200700 1 Processor -- start 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f9200700 1 -- start start 2024-04-26T01:32:26.720 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f9200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f410ece0 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32878/0 (socket says 172.21.15.52:32878) 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.459+0000 7fe9f1e00700 1 -- 172.21.15.52:0/507309325 learned_addr learned my addr 172.21.15.52:0/507309325 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9f1e00700 1 -- 172.21.15.52:0/507309325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f419ebc0 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9f1e00700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4003b20 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe9e401a8c0 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f419ee50 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4024940 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.460+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f41a1b40 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.461+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f40640a0 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.462+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe9e4022020 con 0x7fe9f410a780 2024-04-26T01:32:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.462+0000 7fe9eaa00700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 0x7fe9d803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.462+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe9e404d8a0 con 0x7fe9f410a780 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.462+0000 7fe9f1400700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 0x7fe9d803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.463+0000 7fe9f1400700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 0x7fe9d803bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0006fd0 tx=0x7fe9e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.468+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9e400be50 con 0x7fe9f410a780 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:19.700+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}) -- 0x7fe9f4106ef0 con 0x7fe9d8039600 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.024+0000 7fe9eaa00700 1 -- 172.21.15.52:0/507309325 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fe9f4106ef0 con 0x7fe9d8039600 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 msgr2=0x7fe9d803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 0x7fe9d803bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0006fd0 tx=0x7fe9e0006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 msgr2=0x7fe9f419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4011b70 comp rx=0 tx=0).stop 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 shutdown_connections 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7fe9d8039600 0x7fe9d803bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 --2- 172.21.15.52:0/507309325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9f410a780 0x7fe9f419e680 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:26.723 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 >> 172.21.15.52:0/507309325 conn(0x7fe9f4105d40 msgr2=0x7fe9f41067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:26.723 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 shutdown_connections 2024-04-26T01:32:26.723 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:26.029+0000 7fe9f9200700 1 -- 172.21.15.52:0/507309325 wait complete. 2024-04-26T01:32:26.723 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mon service... 2024-04-26T01:32:27.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:27 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:27.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:27 smithi052 ceph-mon[27851]: Added host smithi052 2024-04-26T01:32:27.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:27 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T01:32:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.379+0000 7f1865600700 1 Processor -- start 2024-04-26T01:32:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.379+0000 7f1865600700 1 -- start start 2024-04-26T01:32:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.380+0000 7f1865600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.380+0000 7f1865600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f186007bec0 con 0x7f1860108f90 2024-04-26T01:32:28.568 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.380+0000 7f185ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:28.568 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.380+0000 7f185ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43118/0 (socket says 172.21.15.52:43118) 2024-04-26T01:32:28.568 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.380+0000 7f185ea00700 1 -- 172.21.15.52:0/3493241833 learned_addr learned my addr 172.21.15.52:0/3493241833 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:28.569 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.381+0000 7f185ea00700 1 -- 172.21.15.52:0/3493241833 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186010b8c0 con 0x7f1860108f90 2024-04-26T01:32:28.569 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.381+0000 7f185ea00700 1 --2- 172.21.15.52:0/3493241833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1850009e80 tx=0x7f185000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8ea31c74b4e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:28.569 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.381+0000 7f185e000700 1 -- 172.21.15.52:0/3493241833 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1850017070 con 0x7f1860108f90 2024-04-26T01:32:28.569 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.381+0000 7f185e000700 1 -- 172.21.15.52:0/3493241833 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f185000b8c0 con 0x7f1860108f90 2024-04-26T01:32:28.569 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.381+0000 7f185e000700 1 -- 172.21.15.52:0/3493241833 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1850004490 con 0x7f1860108f90 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 -- 172.21.15.52:0/3493241833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 msgr2=0x7f186010b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 --2- 172.21.15.52:0/3493241833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1850009e80 tx=0x7f185000b330 comp rx=0 tx=0).stop 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 -- 172.21.15.52:0/3493241833 shutdown_connections 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 --2- 172.21.15.52:0/3493241833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186010b380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 -- 172.21.15.52:0/3493241833 >> 172.21.15.52:0/3493241833 conn(0x7f1860102a00 msgr2=0x7f1860104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.382+0000 7f1865600700 1 -- 172.21.15.52:0/3493241833 shutdown_connections 2024-04-26T01:32:28.571 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.383+0000 7f1865600700 1 -- 172.21.15.52:0/3493241833 wait complete. 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.384+0000 7f1865600700 1 Processor -- start 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.384+0000 7f1865600700 1 -- start start 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.384+0000 7f1865600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.384+0000 7f1865600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f186010c890 con 0x7f1860108f90 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.384+0000 7f185ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f185ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43126/0 (socket says 172.21.15.52:43126) 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f185ea00700 1 -- 172.21.15.52:0/1421354126 learned_addr learned my addr 172.21.15.52:0/1421354126 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f185ea00700 1 -- 172.21.15.52:0/1421354126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186019ebb0 con 0x7f1860108f90 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f185ea00700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f185000b7b0 tx=0x7f185000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1850020020 con 0x7f1860108f90 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1850003d10 con 0x7f1860108f90 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.385+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1850017040 con 0x7f1860108f90 2024-04-26T01:32:28.572 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.386+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186019ee40 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.386+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18601a1b30 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.387+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1850023020 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.387+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18600640a0 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.387+0000 7f1857e00700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 0x7f184403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.387+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f185000f050 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.388+0000 7f1856000700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 0x7f184403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.388+0000 7f1856000700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 0x7f184403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.393+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f185002a020 con 0x7f1860108f90 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.666+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1860103bd0 con 0x7f18440395b0 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.676+0000 7f1857e00700 1 -- 172.21.15.52:0/1421354126 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1860103bd0 con 0x7f18440395b0 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 msgr2=0x7f184403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 0x7f184403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1848006fd0 tx=0x7f1848006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:28.575 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 msgr2=0x7f186019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f185000b7b0 tx=0x7f185000be90 comp rx=0 tx=0).stop 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 shutdown_connections 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f18440395b0 0x7f184403ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 --2- 172.21.15.52:0/1421354126 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1860108f90 0x7f186019e670 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 >> 172.21.15.52:0/1421354126 conn(0x7f1860102a00 msgr2=0x7f18601034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 shutdown_connections 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:27.679+0000 7f1865600700 1 -- 172.21.15.52:0/1421354126 wait complete. 2024-04-26T01:32:28.576 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mgr service... 2024-04-26T01:32:28.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:28 smithi052 ceph-mon[27851]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:28.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:28 smithi052 ceph-mon[27851]: Saving service mon spec with placement count:5 2024-04-26T01:32:28.957 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:28 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:30.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:30 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:30.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:30 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:30.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:30 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.815+0000 7f24fac00700 1 Processor -- start 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.816+0000 7f24fac00700 1 -- start start 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.816+0000 7f24fac00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.816+0000 7f24fac00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f24f407b810 con 0x7f24f407cde0 2024-04-26T01:32:30.768 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.817+0000 7f24f3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:30.773 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.817+0000 7f24f3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43128/0 (socket says 172.21.15.52:43128) 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.817+0000 7f24f3e00700 1 -- 172.21.15.52:0/2547946619 learned_addr learned my addr 172.21.15.52:0/2547946619 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.817+0000 7f24f3e00700 1 -- 172.21.15.52:0/2547946619 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f407b990 con 0x7f24f407cde0 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.818+0000 7f24f3e00700 1 --2- 172.21.15.52:0/2547946619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f24e40098a0 tx=0x7f24e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794287e798c8840b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.818+0000 7f24f2a00700 1 -- 172.21.15.52:0/2547946619 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e4004160 con 0x7f24f407cde0 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.818+0000 7f24f2a00700 1 -- 172.21.15.52:0/2547946619 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24e4004300 con 0x7f24f407cde0 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.818+0000 7f24f2a00700 1 -- 172.21.15.52:0/2547946619 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e4011a20 con 0x7f24f407cde0 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 -- 172.21.15.52:0/2547946619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 msgr2=0x7f24f407b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 --2- 172.21.15.52:0/2547946619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f24e40098a0 tx=0x7f24e4009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 -- 172.21.15.52:0/2547946619 shutdown_connections 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 --2- 172.21.15.52:0/2547946619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f407b240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 -- 172.21.15.52:0/2547946619 >> 172.21.15.52:0/2547946619 conn(0x7f24f4103b50 msgr2=0x7f24f4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:30.774 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 -- 172.21.15.52:0/2547946619 shutdown_connections 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.819+0000 7f24fac00700 1 -- 172.21.15.52:0/2547946619 wait complete. 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.820+0000 7f24fac00700 1 Processor -- start 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.820+0000 7f24fac00700 1 -- start start 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24fac00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24fac00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f24f410eb40 con 0x7f24f407cde0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24f3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24f3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43134/0 (socket says 172.21.15.52:43134) 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24f3e00700 1 -- 172.21.15.52:0/873523473 learned_addr learned my addr 172.21.15.52:0/873523473 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.821+0000 7f24f3e00700 1 -- 172.21.15.52:0/873523473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f419eb70 con 0x7f24f407cde0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24f3e00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009520 tx=0x7f24e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e4003af0 con 0x7f24f407cde0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24e401a8e0 con 0x7f24f407cde0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f419ee00 con 0x7f24f407cde0 2024-04-26T01:32:30.775 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e40247a0 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.822+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f41a1af0 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.823+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f40640a0 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.824+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f24e4022020 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.824+0000 7f24f0c00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 0x7f24d803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.824+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f24e404d8b0 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.824+0000 7f24f3400700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 0x7f24d803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.825+0000 7f24f3400700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 0x7f24d803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f24e0006fd0 tx=0x7f24e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:29.830+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24e402dde0 con 0x7f24f407cde0 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.082+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f24f4104ce0 con 0x7f24d8039600 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.088+0000 7f24f0c00700 1 -- 172.21.15.52:0/873523473 <== mgr.14120 v2:172.21.15.52:6800/3772941101 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f24f4104ce0 con 0x7f24d8039600 2024-04-26T01:32:30.776 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.090+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 msgr2=0x7f24d803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.090+0000 7f24fac00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 0x7f24d803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f24e0006fd0 tx=0x7f24e0006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.090+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 msgr2=0x7f24f419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.090+0000 7f24fac00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009520 tx=0x7f24e4011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 shutdown_connections 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f24d8039600 0x7f24d803bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 --2- 172.21.15.52:0/873523473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24f407cde0 0x7f24f419e630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:30.777 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 >> 172.21.15.52:0/873523473 conn(0x7f24f4103b50 msgr2=0x7f24f41045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 shutdown_connections 2024-04-26T01:32:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:30.091+0000 7f24fac00700 1 -- 172.21.15.52:0/873523473 wait complete. 2024-04-26T01:32:31.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:31 smithi052 ceph-mon[27851]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:32:31.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:31 smithi052 ceph-mon[27851]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:31.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:31 smithi052 ceph-mon[27851]: Saving service mgr spec with placement count:2 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.405+0000 7f8170600700 1 Processor -- start 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.405+0000 7f8170600700 1 -- start start 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.405+0000 7f8170600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.405+0000 7f8170600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f816807be00 con 0x7f8168109080 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.406+0000 7f816dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.406+0000 7f816dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43150/0 (socket says 172.21.15.52:43150) 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.406+0000 7f816dc00700 1 -- 172.21.15.52:0/140472918 learned_addr learned my addr 172.21.15.52:0/140472918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.406+0000 7f816dc00700 1 -- 172.21.15.52:0/140472918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816810b9b0 con 0x7f8168109080 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.407+0000 7f816dc00700 1 --2- 172.21.15.52:0/140472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f815c0098a0 tx=0x7f815c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a096584ddaa7e0d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.407+0000 7f8167e00700 1 -- 172.21.15.52:0/140472918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f815c004160 con 0x7f8168109080 2024-04-26T01:32:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.407+0000 7f8167e00700 1 -- 172.21.15.52:0/140472918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f815c004300 con 0x7f8168109080 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.407+0000 7f8167e00700 1 -- 172.21.15.52:0/140472918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f815c011a20 con 0x7f8168109080 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 -- 172.21.15.52:0/140472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 msgr2=0x7f816810b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 --2- 172.21.15.52:0/140472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f815c0098a0 tx=0x7f815c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 -- 172.21.15.52:0/140472918 shutdown_connections 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 --2- 172.21.15.52:0/140472918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816810b470 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 -- 172.21.15.52:0/140472918 >> 172.21.15.52:0/140472918 conn(0x7f8168102a50 msgr2=0x7f8168104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 -- 172.21.15.52:0/140472918 shutdown_connections 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.408+0000 7f8170600700 1 -- 172.21.15.52:0/140472918 wait complete. 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.409+0000 7f8170600700 1 Processor -- start 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.409+0000 7f8170600700 1 -- start start 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f8170600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f8170600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f816810c900 con 0x7f8168109080 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f816dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f816dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43164/0 (socket says 172.21.15.52:43164) 2024-04-26T01:32:32.933 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f816dc00700 1 -- 172.21.15.52:0/1475406899 learned_addr learned my addr 172.21.15.52:0/1475406899 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.410+0000 7f816dc00700 1 -- 172.21.15.52:0/1475406899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816819c960 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f816dc00700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f815c004660 tx=0x7f815c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f815c003aa0 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f815c01b8f0 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f8170600700 1 -- 172.21.15.52:0/1475406899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816819cbf0 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f815c023900 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.411+0000 7f8170600700 1 -- 172.21.15.52:0/1475406899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f816819f8e0 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.413+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f815c023b70 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.413+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8144005330 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.413+0000 7f8166000700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 0x7f815003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.414+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f815c04dea0 con 0x7f8168109080 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.414+0000 7f816d200700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 0x7f815003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:32.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.415+0000 7f816d200700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 0x7f815003bb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8154006fd0 tx=0x7f8154006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.424+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f815c01f030 con 0x7f8168109080 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.677+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f8144005620 con 0x7f8168109080 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.681+0000 7f8166000700 1 -- 172.21.15.52:0/1475406899 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f815c0202b0 con 0x7f8168109080 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 msgr2=0x7f815003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 0x7f815003bb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8154006fd0 tx=0x7f8154006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 msgr2=0x7f816819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f815c004660 tx=0x7f815c011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 shutdown_connections 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f8150039660 0x7f815003bb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 --2- 172.21.15.52:0/1475406899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8168109080 0x7f816819c420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.684+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 >> 172.21.15.52:0/1475406899 conn(0x7f8168102a50 msgr2=0x7f8168103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.685+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 shutdown_connections 2024-04-26T01:32:32.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:31.685+0000 7f815b400700 1 -- 172.21.15.52:0/1475406899 wait complete. 2024-04-26T01:32:32.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:32 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1475406899' entity='client.admin' 2024-04-26T01:32:33.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:33 smithi052 ceph-mon[27851]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:32:33.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:33 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:34.519 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.520+0000 7f0ba7400700 1 Processor -- start 2024-04-26T01:32:34.519 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.520+0000 7f0ba7400700 1 -- start start 2024-04-26T01:32:34.519 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.521+0000 7f0ba7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:34.519 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.521+0000 7f0ba7400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba007b7d0 con 0x7f0ba007cda0 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.521+0000 7f0ba4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.521+0000 7f0ba4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43174/0 (socket says 172.21.15.52:43174) 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.521+0000 7f0ba4a00700 1 -- 172.21.15.52:0/186860113 learned_addr learned my addr 172.21.15.52:0/186860113 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.522+0000 7f0ba4a00700 1 -- 172.21.15.52:0/186860113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba007b950 con 0x7f0ba007cda0 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.522+0000 7f0ba4a00700 1 --2- 172.21.15.52:0/186860113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b940098a0 tx=0x7f0b94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8d65fa7a11191d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.522+0000 7f0b9f400700 1 -- 172.21.15.52:0/186860113 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b94004160 con 0x7f0ba007cda0 2024-04-26T01:32:34.520 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.522+0000 7f0b9f400700 1 -- 172.21.15.52:0/186860113 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b94004300 con 0x7f0ba007cda0 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.523+0000 7f0b9f400700 1 -- 172.21.15.52:0/186860113 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b94011a20 con 0x7f0ba007cda0 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.523+0000 7f0ba7400700 1 -- 172.21.15.52:0/186860113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 msgr2=0x7f0ba007b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.523+0000 7f0ba7400700 1 --2- 172.21.15.52:0/186860113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b940098a0 tx=0x7f0b94009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.524+0000 7f0ba7400700 1 -- 172.21.15.52:0/186860113 shutdown_connections 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.524+0000 7f0ba7400700 1 --2- 172.21.15.52:0/186860113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba007b200 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.524+0000 7f0ba7400700 1 -- 172.21.15.52:0/186860113 >> 172.21.15.52:0/186860113 conn(0x7f0ba0103b50 msgr2=0x7f0ba0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:34.521 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.524+0000 7f0ba7400700 1 -- 172.21.15.52:0/186860113 shutdown_connections 2024-04-26T01:32:34.522 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.524+0000 7f0ba7400700 1 -- 172.21.15.52:0/186860113 wait complete. 2024-04-26T01:32:34.522 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.525+0000 7f0ba7400700 1 Processor -- start 2024-04-26T01:32:34.522 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.525+0000 7f0ba7400700 1 -- start start 2024-04-26T01:32:34.522 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.525+0000 7f0ba7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:34.523 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.525+0000 7f0ba7400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba010eb30 con 0x7f0ba007cda0 2024-04-26T01:32:34.523 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.526+0000 7f0ba4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:34.523 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.526+0000 7f0ba4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43176/0 (socket says 172.21.15.52:43176) 2024-04-26T01:32:34.523 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.526+0000 7f0ba4a00700 1 -- 172.21.15.52:0/189660214 learned_addr learned my addr 172.21.15.52:0/189660214 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:34.523 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.526+0000 7f0ba4a00700 1 -- 172.21.15.52:0/189660214 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba019eb40 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0ba4a00700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009520 tx=0x7f0b94011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b940039e0 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9401b8f0 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b94019770 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba019edd0 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.527+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba01a1ac0 con 0x7f0ba007cda0 2024-04-26T01:32:34.524 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.528+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba00640a0 con 0x7f0ba007cda0 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.529+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0b9401b440 con 0x7f0ba007cda0 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.530+0000 7f0b9d600700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 0x7f0b8803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.530+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b9404dfb0 con 0x7f0ba007cda0 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.530+0000 7f0b9fe00700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 0x7f0b8803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.531+0000 7f0b9fe00700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 0x7f0b8803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:34.525 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.541+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b9401f030 con 0x7f0ba007cda0 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.773+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0ba01a1c90 con 0x7f0ba007cda0 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.777+0000 7f0b9d600700 1 -- 172.21.15.52:0/189660214 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0b9400b3e0 con 0x7f0ba007cda0 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 msgr2=0x7f0b8803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 0x7f0b8803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 msgr2=0x7f0ba019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009520 tx=0x7f0b94011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 shutdown_connections 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0b88039600 0x7f0b8803bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 --2- 172.21.15.52:0/189660214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ba007cda0 0x7f0ba019e600 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 >> 172.21.15.52:0/189660214 conn(0x7f0ba0103b50 msgr2=0x7f0ba01045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 shutdown_connections 2024-04-26T01:32:34.526 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:33.780+0000 7f0ba7400700 1 -- 172.21.15.52:0/189660214 wait complete. 2024-04-26T01:32:34.527 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2024-04-26T01:32:35.067 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:34 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/189660214' entity='client.admin' 2024-04-26T01:32:36.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:35 smithi052 ceph-mon[27851]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:32:36.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:35 smithi052 ceph-mon[27851]: from='mgr.14120 172.21.15.52:0/3573998164' entity='mgr.x' 2024-04-26T01:32:36.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:35 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1468379132' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T01:32:36.459 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setuser ceph since I am not root 2024-04-26T01:32:36.459 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setgroup ceph since I am not root 2024-04-26T01:32:36.924 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:36.543+0000 7f9309c47200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:32:36.924 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:36.790+0000 7f9309c47200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:36.925 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:36.928+0000 7f9309c47200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:32:37.469 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:37.351+0000 7f9309c47200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:32:37.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:37 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1468379132' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T01:32:37.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:37 smithi052 ceph-mon[27851]: mgrmap e9: x(active, since 26s) 2024-04-26T01:32:37.776 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:37.473+0000 7f9309c47200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.115+0000 7f7387a00700 1 Processor -- start 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.115+0000 7f7387a00700 1 -- start start 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.115+0000 7f7387a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.115+0000 7f7387a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f738007b7b0 con 0x7f738007cd80 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.116+0000 7f7385000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.116+0000 7f7385000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41712/0 (socket says 172.21.15.52:41712) 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.116+0000 7f7385000700 1 -- 172.21.15.52:0/1303107119 learned_addr learned my addr 172.21.15.52:0/1303107119 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.116+0000 7f7385000700 1 -- 172.21.15.52:0/1303107119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738007b930 con 0x7f738007cd80 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.117+0000 7f7385000700 1 --2- 172.21.15.52:0/1303107119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73740098a0 tx=0x7f7374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5594792936eca0e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:37.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.117+0000 7f737f400700 1 -- 172.21.15.52:0/1303107119 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7374004160 con 0x7f738007cd80 2024-04-26T01:32:37.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.117+0000 7f737f400700 1 -- 172.21.15.52:0/1303107119 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7374004300 con 0x7f738007cd80 2024-04-26T01:32:37.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.117+0000 7f737f400700 1 -- 172.21.15.52:0/1303107119 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7374011a20 con 0x7f738007cd80 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 -- 172.21.15.52:0/1303107119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 msgr2=0x7f738007b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 --2- 172.21.15.52:0/1303107119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73740098a0 tx=0x7f7374009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 -- 172.21.15.52:0/1303107119 shutdown_connections 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 --2- 172.21.15.52:0/1303107119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738007b1e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 -- 172.21.15.52:0/1303107119 >> 172.21.15.52:0/1303107119 conn(0x7f7380103b50 msgr2=0x7f7380105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:37.801 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 -- 172.21.15.52:0/1303107119 shutdown_connections 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.118+0000 7f7387a00700 1 -- 172.21.15.52:0/1303107119 wait complete. 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.119+0000 7f7387a00700 1 Processor -- start 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7387a00700 1 -- start start 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7387a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7387a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f738010eb50 con 0x7f738007cd80 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7385000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7385000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41718/0 (socket says 172.21.15.52:41718) 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.120+0000 7f7385000700 1 -- 172.21.15.52:0/1468379132 learned_addr learned my addr 172.21.15.52:0/1468379132 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f7385000700 1 -- 172.21.15.52:0/1468379132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738019eb00 con 0x7f738007cd80 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f7385000700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7374000c00 tx=0x7f7374011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7374003aa0 con 0x7f738007cd80 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f737401b8f0 con 0x7f738007cd80 2024-04-26T01:32:37.802 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7374023870 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738019ed90 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.121+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73801a1a80 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.123+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f737401b440 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.123+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73800640a0 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.123+0000 7f737d600700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 0x7f736803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.123+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f737404de30 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.124+0000 7f737fe00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 0x7f736803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.124+0000 7f737fe00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 0x7f736803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f736c006fd0 tx=0x7f736c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.129+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f737401f030 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:35.424+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f73801a2070 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.218+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f737401a3d0 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.223+0000 7f737d600700 1 -- 172.21.15.52:0/1468379132 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f737400b3e0 con 0x7f738007cd80 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.224+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 msgr2=0x7f736803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:37.803 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.224+0000 7f7387a00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 0x7f736803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f736c006fd0 tx=0x7f736c006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.224+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 msgr2=0x7f738019e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.224+0000 7f7387a00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7374000c00 tx=0x7f7374011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.225+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 shutdown_connections 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.225+0000 7f7387a00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f7368039600 0x7f736803bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.225+0000 7f7387a00700 1 --2- 172.21.15.52:0/1468379132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f738007cd80 0x7f738019e5c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.225+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 >> 172.21.15.52:0/1468379132 conn(0x7f7380103b50 msgr2=0x7f73801045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.226+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 shutdown_connections 2024-04-26T01:32:37.804 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:36.226+0000 7f7387a00700 1 -- 172.21.15.52:0/1468379132 wait complete. 2024-04-26T01:32:38.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:38 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1041877503' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T01:32:39.325 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:39.093+0000 7f9309c47200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:32:39.336 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.375+0000 7f0ddbc00700 1 Processor -- start 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.375+0000 7f0ddbc00700 1 -- start start 2024-04-26T01:32:39.337 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.375+0000 7f0ddbc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:39.344 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.375+0000 7f0ddbc00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd4063ce0 con 0x7f0dd4106e20 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.376+0000 7f0dd9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.376+0000 7f0dd9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41750/0 (socket says 172.21.15.52:41750) 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.376+0000 7f0dd9200700 1 -- 172.21.15.52:0/790373213 learned_addr learned my addr 172.21.15.52:0/790373213 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.376+0000 7f0dd9200700 1 -- 172.21.15.52:0/790373213 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd4107780 con 0x7f0dd4106e20 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.377+0000 7f0dd9200700 1 --2- 172.21.15.52:0/790373213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80098a0 tx=0x7f0dc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa44a130d655255c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:39.345 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.377+0000 7f0dd3400700 1 -- 172.21.15.52:0/790373213 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc8004160 con 0x7f0dd4106e20 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.377+0000 7f0dd3400700 1 -- 172.21.15.52:0/790373213 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dc8004300 con 0x7f0dd4106e20 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.377+0000 7f0dd3400700 1 -- 172.21.15.52:0/790373213 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc8011a20 con 0x7f0dd4106e20 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 -- 172.21.15.52:0/790373213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 msgr2=0x7f0dd4107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/790373213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80098a0 tx=0x7f0dc8009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 -- 172.21.15.52:0/790373213 shutdown_connections 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/790373213 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd4107240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 -- 172.21.15.52:0/790373213 >> 172.21.15.52:0/790373213 conn(0x7f0dd4102a40 msgr2=0x7f0dd4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 -- 172.21.15.52:0/790373213 shutdown_connections 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.378+0000 7f0ddbc00700 1 -- 172.21.15.52:0/790373213 wait complete. 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.379+0000 7f0ddbc00700 1 Processor -- start 2024-04-26T01:32:39.348 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.379+0000 7f0ddbc00700 1 -- start start 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0ddbc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0ddbc00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd41087f0 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0dd9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0dd9200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41756/0 (socket says 172.21.15.52:41756) 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0dd9200700 1 -- 172.21.15.52:0/1041877503 learned_addr learned my addr 172.21.15.52:0/1041877503 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0dd9200700 1 -- 172.21.15.52:0/1041877503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd419eaf0 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.380+0000 7f0dd9200700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.381+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc8003aa0 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.381+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dc801b8f0 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.381+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dc8023870 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.381+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd419ed80 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.381+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd41a1a70 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.382+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f0dc801b440 con 0x7f0dd4106e20 2024-04-26T01:32:39.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.382+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd40640a0 con 0x7f0dd4106e20 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.383+0000 7f0dd1600700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 0x7f0dbc03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.383+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0dc804df80 con 0x7f0dd4106e20 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.383+0000 7f0dd3e00700 1 -- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 msgr2=0x7f0dbc03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.383+0000 7f0dd3e00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 0x7f0dbc03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.389+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dc801f030 con 0x7f0dd4106e20 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.584+0000 7f0dd3e00700 1 -- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 msgr2=0x7f0dbc03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.584+0000 7f0dd3e00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 0x7f0dbc03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.676+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0dd402e1f0 con 0x7f0dd4106e20 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.677+0000 7f0dd1600700 1 -- 172.21.15.52:0/1041877503 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0dc80292a0 con 0x7f0dd4106e20 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 msgr2=0x7f0dbc03bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 0x7f0dbc03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 msgr2=0x7f0dd419e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 shutdown_connections 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f0dbc039660 0x7f0dbc03bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 --2- 172.21.15.52:0/1041877503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0dd4106e20 0x7f0dd419e5b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.680+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 >> 172.21.15.52:0/1041877503 conn(0x7f0dd4102a40 msgr2=0x7f0dd4195240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.681+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 shutdown_connections 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:38.681+0000 7f0ddbc00700 1 -- 172.21.15.52:0/1041877503 wait complete. 2024-04-26T01:32:39.351 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-04-26T01:32:39.352 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2024-04-26T01:32:39.644 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:39.332+0000 7f9309c47200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:39.679 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:39.572+0000 7f9309c47200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:32:40.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:39.984+0000 7f9309c47200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:32:40.209 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:40.109+0000 7f9309c47200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:32:40.626 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:40.368+0000 7f9309c47200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:32:40.626 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:40.630+0000 7f9309c47200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:32:41.709 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:41.437+0000 7f9309c47200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:32:42.406 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:42.005+0000 7f9309c47200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:32:42.406 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:42.136+0000 7f9309c47200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:32:42.406 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:42.409+0000 7f9309c47200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:32:43.202 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:43.078+0000 7f9309c47200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:32:43.458 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:43.206+0000 7f9309c47200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:32:43.873 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:43.549+0000 7f9309c47200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:32:43.873 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:43.742+0000 7f9309c47200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:32:43.873 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:43.877+0000 7f9309c47200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:32:44.208 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:44.040+0000 7f9309c47200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:32:44.708 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:44.420+0000 7f9309c47200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:32:45.458 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:45.212+0000 7f9309c47200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: Active manager daemon x restarted 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: Activating manager daemon x 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: mgrmap e10: x(active, starting, since 0.00976859s) 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-mon[27851]: Manager daemon x is now available 2024-04-26T01:32:45.959 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:32:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:32:45.505+0000 7f9309c47200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:46] ENGINE Bus STARTING 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:46] ENGINE Serving on https://172.21.15.52:7150 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:46] ENGINE Serving on http://172.21.15.52:8765 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: [26/Apr/2024:01:32:46] ENGINE Bus STARTED 2024-04-26T01:32:46.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:46 smithi052 ceph-mon[27851]: mgrmap e11: x(active, since 1.01198s) 2024-04-26T01:32:47.142 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-04-26T01:32:47.143 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T01:32:47.143 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T01:32:47.143 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-04-26T01:32:47.144 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.937+0000 7f5277000700 1 Processor -- start 2024-04-26T01:32:47.144 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.937+0000 7f5277000700 1 -- start start 2024-04-26T01:32:47.144 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.937+0000 7f5277000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:47.148 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.937+0000 7f5277000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5270063ca0 con 0x7f5270109070 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.938+0000 7f526fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.938+0000 7f526fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41764/0 (socket says 172.21.15.52:41764) 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.938+0000 7f526fe00700 1 -- 172.21.15.52:0/2049441455 learned_addr learned my addr 172.21.15.52:0/2049441455 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.938+0000 7f526fe00700 1 -- 172.21.15.52:0/2049441455 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527010b9a0 con 0x7f5270109070 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.938+0000 7f526fe00700 1 --2- 172.21.15.52:0/2049441455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5260009e80 tx=0x7f526000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea08da582e71098 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.939+0000 7f526ea00700 1 -- 172.21.15.52:0/2049441455 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260017070 con 0x7f5270109070 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.939+0000 7f526ea00700 1 -- 172.21.15.52:0/2049441455 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f526000b8c0 con 0x7f5270109070 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.939+0000 7f526ea00700 1 -- 172.21.15.52:0/2049441455 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260004490 con 0x7f5270109070 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 -- 172.21.15.52:0/2049441455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 msgr2=0x7f527010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 --2- 172.21.15.52:0/2049441455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5260009e80 tx=0x7f526000b330 comp rx=0 tx=0).stop 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 -- 172.21.15.52:0/2049441455 shutdown_connections 2024-04-26T01:32:47.149 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 --2- 172.21.15.52:0/2049441455 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f527010b460 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 -- 172.21.15.52:0/2049441455 >> 172.21.15.52:0/2049441455 conn(0x7f5270102a40 msgr2=0x7f5270104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 -- 172.21.15.52:0/2049441455 shutdown_connections 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.940+0000 7f5277000700 1 -- 172.21.15.52:0/2049441455 wait complete. 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.941+0000 7f5277000700 1 Processor -- start 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.941+0000 7f5277000700 1 -- start start 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.941+0000 7f5277000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.941+0000 7f5277000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f527010c8f0 con 0x7f5270109070 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.942+0000 7f526fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.942+0000 7f526fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41776/0 (socket says 172.21.15.52:41776) 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.942+0000 7f526fe00700 1 -- 172.21.15.52:0/502697739 learned_addr learned my addr 172.21.15.52:0/502697739 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.942+0000 7f526fe00700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5270108e30 con 0x7f5270109070 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.942+0000 7f526fe00700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5260004220 tx=0x7f52600093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.943+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260020020 con 0x7f5270109070 2024-04-26T01:32:47.150 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.943+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5260003b50 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.943+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5260017040 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.943+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5270107020 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.943+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5270107560 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.944+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f5260023020 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.944+0000 7f526cc00700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.944+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f525403c220 con 0x7f5254039650 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.944+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f526000f050 con 0x7f5270109070 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.945+0000 7f526f400700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:39.945+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:40.146+0000 7f526f400700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:40.146+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:40.547+0000 7f526f400700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:40.547+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T01:32:47.151 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:41.348+0000 7f526f400700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:47.154 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:41.348+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T01:32:47.154 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:42.949+0000 7f526f400700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3772941101 2024-04-26T01:32:47.154 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:42.949+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T01:32:47.154 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.516+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f5260012950 con 0x7f5270109070 2024-04-26T01:32:47.154 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.516+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 msgr2=0x7f525403bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:45.516+0000 7f526cc00700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.517+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f526004f0a0 con 0x7f5270109070 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.518+0000 7f526cc00700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 0x7f525403f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.518+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f526001de80 con 0x7f525403cfe0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.520+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 0x7f525403f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.522+0000 7f526f400700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 0x7f525403f3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f525c003a10 tx=0x7f525c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.524+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f526001de80 con 0x7f525403cfe0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.532+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5270069ba0 con 0x7f525403cfe0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.533+0000 7f526cc00700 1 -- 172.21.15.52:0/502697739 <== mgr.14152 v2:172.21.15.52:6800/2415060450 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f5270069ba0 con 0x7f525403cfe0 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 msgr2=0x7f525403f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 0x7f525403f3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f525c003a10 tx=0x7f525c009270 comp rx=0 tx=0).stop 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 msgr2=0x7f52701088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5260004220 tx=0x7f52600093e0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 shutdown_connections 2024-04-26T01:32:47.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f525403cfe0 0x7f525403f3d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5270109070 0x7f52701088f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 --2- 172.21.15.52:0/502697739 >> [v2:172.21.15.52:6800/3772941101,v1:172.21.15.52:6801/3772941101] conn(0x7f5254039650 0x7f525403bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 >> 172.21.15.52:0/502697739 conn(0x7f5270102a40 msgr2=0x7f5270103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 shutdown_connections 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:46.534+0000 7f5277000700 1 -- 172.21.15.52:0/502697739 wait complete. 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2024-04-26T01:32:47.156 INFO:teuthology.orchestra.run.smithi052.stdout:Generating a dashboard self-signed certificate... 2024-04-26T01:32:47.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:47 smithi052 ceph-mon[27851]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T01:32:47.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:47 smithi052 ceph-mon[27851]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T01:32:48.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:48 smithi052 ceph-mon[27851]: mgrmap e12: x(active, since 2s) 2024-04-26T01:32:48.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:32:49.461 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.505+0000 7f595e400700 1 Processor -- start 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.505+0000 7f595e400700 1 -- start start 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.506+0000 7f595e400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.506+0000 7f595e400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5958063d60 con 0x7f5958106da0 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.506+0000 7f5957400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.506+0000 7f5957400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40904/0 (socket says 172.21.15.52:40904) 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.506+0000 7f5957400700 1 -- 172.21.15.52:0/1058988005 learned_addr learned my addr 172.21.15.52:0/1058988005 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.507+0000 7f5957400700 1 -- 172.21.15.52:0/1058988005 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5958109710 con 0x7f5958106da0 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.507+0000 7f5957400700 1 --2- 172.21.15.52:0/1058988005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5948009e80 tx=0x7f594800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8ee6ebed7b39c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:49.462 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.507+0000 7f5956a00700 1 -- 172.21.15.52:0/1058988005 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5948017070 con 0x7f5958106da0 2024-04-26T01:32:49.463 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f5956a00700 1 -- 172.21.15.52:0/1058988005 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f594800b8c0 con 0x7f5958106da0 2024-04-26T01:32:49.463 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f5956a00700 1 -- 172.21.15.52:0/1058988005 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5948004490 con 0x7f5958106da0 2024-04-26T01:32:49.463 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f595e400700 1 -- 172.21.15.52:0/1058988005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 msgr2=0x7f59581091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:49.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f595e400700 1 --2- 172.21.15.52:0/1058988005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5948009e80 tx=0x7f594800b330 comp rx=0 tx=0).stop 2024-04-26T01:32:49.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f595e400700 1 -- 172.21.15.52:0/1058988005 shutdown_connections 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f595e400700 1 --2- 172.21.15.52:0/1058988005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581091d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.508+0000 7f595e400700 1 -- 172.21.15.52:0/1058988005 >> 172.21.15.52:0/1058988005 conn(0x7f5958102a40 msgr2=0x7f5958104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.509+0000 7f595e400700 1 -- 172.21.15.52:0/1058988005 shutdown_connections 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.509+0000 7f595e400700 1 -- 172.21.15.52:0/1058988005 wait complete. 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f595e400700 1 Processor -- start 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f595e400700 1 -- start start 2024-04-26T01:32:49.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f595e400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f595e400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f595810a780 con 0x7f5958106da0 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f5957400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f5957400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40920/0 (socket says 172.21.15.52:40920) 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.510+0000 7f5957400700 1 -- 172.21.15.52:0/349282290 learned_addr learned my addr 172.21.15.52:0/349282290 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.511+0000 7f5957400700 1 -- 172.21.15.52:0/349282290 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59581a2f50 con 0x7f5958106da0 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.511+0000 7f5957400700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5948004220 tx=0x7f59480093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:49.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.512+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5948020020 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.512+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5948003b50 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.512+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59581a31e0 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.512+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5948017040 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.512+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59581a5ed0 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.513+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59580640a0 con 0x7f5958106da0 2024-04-26T01:32:49.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.513+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5948023020 con 0x7f5958106da0 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.513+0000 7f5954c00700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 0x7f593c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.514+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f594800f050 con 0x7f5958106da0 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.514+0000 7f594ea00700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 0x7f593c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.515+0000 7f594ea00700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 0x7f593c03bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5940006fd0 tx=0x7f5940006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.520+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f594802a020 con 0x7f5958106da0 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.760+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f595802e330 con 0x7f593c039650 2024-04-26T01:32:49.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.843+0000 7f5954c00700 1 -- 172.21.15.52:0/349282290 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f595802e330 con 0x7f593c039650 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 msgr2=0x7f593c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 0x7f593c03bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5940006fd0 tx=0x7f5940006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 msgr2=0x7f59581a2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5948004220 tx=0x7f59480093e0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 shutdown_connections 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f593c039650 0x7f593c03bb10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 --2- 172.21.15.52:0/349282290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5958106da0 0x7f59581a2a10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:49.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 >> 172.21.15.52:0/349282290 conn(0x7f5958102a40 msgr2=0x7f5958104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:49.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.846+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 shutdown_connections 2024-04-26T01:32:49.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:48.847+0000 7f595e400700 1 -- 172.21.15.52:0/349282290 wait complete. 2024-04-26T01:32:49.475 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2024-04-26T01:32:50.033 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:49 smithi052 ceph-mon[27851]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:50.033 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:32:50.033 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:32:51.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8dYRtN/PTBlHzfuhRFcL9.2d7IwXIvIlMZLM15So/sQRFWa.I3vKC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714095170, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T01:32:51.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.079+0000 7f6bd9600700 1 Processor -- start 2024-04-26T01:32:51.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.079+0000 7f6bd9600700 1 -- start start 2024-04-26T01:32:51.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.080+0000 7f6bd9600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:51.374 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.080+0000 7f6bd9600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd407c920 con 0x7f6bd407dea0 2024-04-26T01:32:51.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.080+0000 7f6bd2a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:51.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.080+0000 7f6bd2a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40922/0 (socket says 172.21.15.52:40922) 2024-04-26T01:32:51.375 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.080+0000 7f6bd2a00700 1 -- 172.21.15.52:0/2496639318 learned_addr learned my addr 172.21.15.52:0/2496639318 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:51.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.081+0000 7f6bd2a00700 1 -- 172.21.15.52:0/2496639318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd407caa0 con 0x7f6bd407dea0 2024-04-26T01:32:51.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.081+0000 7f6bd2a00700 1 --2- 172.21.15.52:0/2496639318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009e80 tx=0x7f6bc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115bbd4f7664baf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:51.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.081+0000 7f6bd2000700 1 -- 172.21.15.52:0/2496639318 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4017070 con 0x7f6bd407dea0 2024-04-26T01:32:51.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.081+0000 7f6bd2000700 1 -- 172.21.15.52:0/2496639318 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc400b8c0 con 0x7f6bd407dea0 2024-04-26T01:32:51.376 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.081+0000 7f6bd2000700 1 -- 172.21.15.52:0/2496639318 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4004490 con 0x7f6bd407dea0 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.082+0000 7f6bd9600700 1 -- 172.21.15.52:0/2496639318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 msgr2=0x7f6bd407c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.082+0000 7f6bd9600700 1 --2- 172.21.15.52:0/2496639318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009e80 tx=0x7f6bc400b330 comp rx=0 tx=0).stop 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.082+0000 7f6bd9600700 1 -- 172.21.15.52:0/2496639318 shutdown_connections 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.082+0000 7f6bd9600700 1 --2- 172.21.15.52:0/2496639318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd407c350 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.082+0000 7f6bd9600700 1 -- 172.21.15.52:0/2496639318 >> 172.21.15.52:0/2496639318 conn(0x7f6bd4103b70 msgr2=0x7f6bd4105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:51.377 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.083+0000 7f6bd9600700 1 -- 172.21.15.52:0/2496639318 shutdown_connections 2024-04-26T01:32:51.378 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.083+0000 7f6bd9600700 1 -- 172.21.15.52:0/2496639318 wait complete. 2024-04-26T01:32:51.378 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.084+0000 7f6bd9600700 1 Processor -- start 2024-04-26T01:32:51.378 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.084+0000 7f6bd9600700 1 -- start start 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.084+0000 7f6bd9600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.084+0000 7f6bd9600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd410eb30 con 0x7f6bd407dea0 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.085+0000 7f6bd2a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.085+0000 7f6bd2a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40936/0 (socket says 172.21.15.52:40936) 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.085+0000 7f6bd2a00700 1 -- 172.21.15.52:0/3754974129 learned_addr learned my addr 172.21.15.52:0/3754974129 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:51.379 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.085+0000 7f6bd2a00700 1 -- 172.21.15.52:0/3754974129 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd419eb80 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.085+0000 7f6bd2a00700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400b7b0 tx=0x7f6bc400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.086+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4020020 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.086+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc4003c60 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.086+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd419ee10 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.086+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bc4017040 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.086+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd41a1b00 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.087+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd40640a0 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.087+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6bc4023020 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.088+0000 7f6bcb400700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 0x7f6bb803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.088+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6bc400f050 con 0x7f6bd407dea0 2024-04-26T01:32:51.381 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.088+0000 7f6bcbe00700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 0x7f6bb803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:51.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.089+0000 7f6bcbe00700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 0x7f6bb803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc006fd0 tx=0x7f6bbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:51.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.094+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bc402a020 con 0x7f6bd407dea0 2024-04-26T01:32:51.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.345+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f6bd4104d70 con 0x7f6bb8039600 2024-04-26T01:32:51.382 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.647+0000 7f6bcb400700 1 -- 172.21.15.52:0/3754974129 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f6bd4104d70 con 0x7f6bb8039600 2024-04-26T01:32:51.383 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 msgr2=0x7f6bb803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:51.383 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 0x7f6bb803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc006fd0 tx=0x7f6bbc006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:51.383 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 msgr2=0x7f6bd419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:51.383 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400b7b0 tx=0x7f6bc400bd50 comp rx=0 tx=0).stop 2024-04-26T01:32:51.383 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 shutdown_connections 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6bb8039600 0x7f6bb803bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 --2- 172.21.15.52:0/3754974129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bd407dea0 0x7f6bd419e640 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 >> 172.21.15.52:0/3754974129 conn(0x7f6bd4103b70 msgr2=0x7f6bd4104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 shutdown_connections 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:50.651+0000 7f6bd9600700 1 -- 172.21.15.52:0/3754974129 wait complete. 2024-04-26T01:32:51.384 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2024-04-26T01:32:51.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:51 smithi052 ceph-mon[27851]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:32:51.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:32:51.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:32:53.217 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.219+0000 7f0013e00700 1 Processor -- start 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.220+0000 7f0013e00700 1 -- start start 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.220+0000 7f0013e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.220+0000 7f0013e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0014108d30 con 0x7f0014108340 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.221+0000 7f0012a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.221+0000 7f0012a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40942/0 (socket says 172.21.15.52:40942) 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.221+0000 7f0012a00700 1 -- 172.21.15.52:0/3680365335 learned_addr learned my addr 172.21.15.52:0/3680365335 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.221+0000 7f0012a00700 1 -- 172.21.15.52:0/3680365335 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0014109540 con 0x7f0014108340 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.222+0000 7f0012a00700 1 --2- 172.21.15.52:0/3680365335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00040098a0 tx=0x7f0004009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11be62198d3bbbb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.222+0000 7f0011600700 1 -- 172.21.15.52:0/3680365335 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0004004160 con 0x7f0014108340 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.222+0000 7f0011600700 1 -- 172.21.15.52:0/3680365335 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0004004300 con 0x7f0014108340 2024-04-26T01:32:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.222+0000 7f0011600700 1 -- 172.21.15.52:0/3680365335 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0004011a20 con 0x7f0014108340 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 -- 172.21.15.52:0/3680365335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 msgr2=0x7f0014108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 --2- 172.21.15.52:0/3680365335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00040098a0 tx=0x7f0004009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 -- 172.21.15.52:0/3680365335 shutdown_connections 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 --2- 172.21.15.52:0/3680365335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f0014108760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 -- 172.21.15.52:0/3680365335 >> 172.21.15.52:0/3680365335 conn(0x7f0014103940 msgr2=0x7f0014105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 -- 172.21.15.52:0/3680365335 shutdown_connections 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.223+0000 7f0013e00700 1 -- 172.21.15.52:0/3680365335 wait complete. 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7f0013e00700 1 Processor -- start 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7f0013e00700 1 -- start start 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.224+0000 7f0013e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0013e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f001410c8f0 con 0x7f0014108340 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0012a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0012a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40956/0 (socket says 172.21.15.52:40956) 2024-04-26T01:32:53.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0012a00700 1 -- 172.21.15.52:0/985413986 learned_addr learned my addr 172.21.15.52:0/985413986 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0012a00700 1 -- 172.21.15.52:0/985413986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001407cfb0 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.225+0000 7f0012a00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0004009520 tx=0x7f0004011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.226+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00040039e0 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.226+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f000401b8f0 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.226+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001407b1a0 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.226+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0004019770 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.226+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001407b6e0 con 0x7f0014108340 2024-04-26T01:32:53.223 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.227+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f000401b440 con 0x7f0014108340 2024-04-26T01:32:53.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.227+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0014069ba0 con 0x7f0014108340 2024-04-26T01:32:53.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.228+0000 7f0018a00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 0x7efff803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:53.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.228+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f000404df90 con 0x7f0014108340 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.228+0000 7f0012000700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 0x7efff803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.229+0000 7f0012000700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 0x7efff803bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0000006fd0 tx=0x7f0000006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.234+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f000401f030 con 0x7f0014108340 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.350+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0004017cf0 con 0x7f0014108340 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.471+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f001407bfc0 con 0x7f0014108340 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.472+0000 7f0018a00700 1 -- 172.21.15.52:0/985413986 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f0004038e20 con 0x7f0014108340 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 msgr2=0x7efff803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 0x7efff803bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0000006fd0 tx=0x7f0000006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 msgr2=0x7f001407ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0004009520 tx=0x7f0004011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 shutdown_connections 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efff8039650 0x7efff803bb10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 --2- 172.21.15.52:0/985413986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0014108340 0x7f001407ca70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:53.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.475+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 >> 172.21.15.52:0/985413986 conn(0x7f0014103940 msgr2=0x7f0014104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:53.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.476+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 shutdown_connections 2024-04-26T01:32:53.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:52.476+0000 7f0013e00700 1 -- 172.21.15.52:0/985413986 wait complete. 2024-04-26T01:32:53.256 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T01:32:53.256 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-04-26T01:32:53.280 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T01:32:53.280 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-04-26T01:32:53.280 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-04-26T01:32:53.281 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T01:32:53.287 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2024-04-26T01:32:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-04-26T01:32:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2024-04-26T01:32:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: Password: z2e6dwd974 2024-04-26T01:32:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:53.289 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config directory 2024-04-26T01:32:53.291 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling autotune for osd_memory_target 2024-04-26T01:32:53.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:53 smithi052 ceph-mon[27851]: mgrmap e13: x(active, since 6s) 2024-04-26T01:32:53.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:53 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/985413986' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T01:32:55.108 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.248+0000 7fd978600700 1 Processor -- start 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.248+0000 7fd978600700 1 -- start start 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.249+0000 7fd978600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.249+0000 7fd978600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd970110c10 con 0x7fd970110220 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.249+0000 7fd975c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.249+0000 7fd975c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40964/0 (socket says 172.21.15.52:40964) 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.249+0000 7fd975c00700 1 -- 172.21.15.52:0/3289502045 learned_addr learned my addr 172.21.15.52:0/3289502045 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:55.109 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.250+0000 7fd975c00700 1 -- 172.21.15.52:0/3289502045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97010ece0 con 0x7fd970110220 2024-04-26T01:32:55.110 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.250+0000 7fd975c00700 1 --2- 172.21.15.52:0/3289502045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9640098a0 tx=0x7fd964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c60e6bb8a3d68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:55.110 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.251+0000 7fd96fe00700 1 -- 172.21.15.52:0/3289502045 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964004160 con 0x7fd970110220 2024-04-26T01:32:55.110 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.251+0000 7fd96fe00700 1 -- 172.21.15.52:0/3289502045 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd964004300 con 0x7fd970110220 2024-04-26T01:32:55.114 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.251+0000 7fd96fe00700 1 -- 172.21.15.52:0/3289502045 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964011a20 con 0x7fd970110220 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.251+0000 7fd978600700 1 -- 172.21.15.52:0/3289502045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 msgr2=0x7fd970110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.251+0000 7fd978600700 1 --2- 172.21.15.52:0/3289502045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9640098a0 tx=0x7fd964009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.252+0000 7fd978600700 1 -- 172.21.15.52:0/3289502045 shutdown_connections 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.252+0000 7fd978600700 1 --2- 172.21.15.52:0/3289502045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd970110640 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.252+0000 7fd978600700 1 -- 172.21.15.52:0/3289502045 >> 172.21.15.52:0/3289502045 conn(0x7fd97007de20 msgr2=0x7fd970080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.252+0000 7fd978600700 1 -- 172.21.15.52:0/3289502045 shutdown_connections 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.252+0000 7fd978600700 1 -- 172.21.15.52:0/3289502045 wait complete. 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.253+0000 7fd978600700 1 Processor -- start 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.253+0000 7fd978600700 1 -- start start 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.254+0000 7fd978600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.254+0000 7fd978600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd970113140 con 0x7fd970110220 2024-04-26T01:32:55.115 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.254+0000 7fd975c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.254+0000 7fd975c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40978/0 (socket says 172.21.15.52:40978) 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.254+0000 7fd975c00700 1 -- 172.21.15.52:0/610332971 learned_addr learned my addr 172.21.15.52:0/610332971 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd975c00700 1 -- 172.21.15.52:0/610332971 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9701a3120 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd975c00700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd964000c00 tx=0x7fd964011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964003aa0 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd96401b8f0 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd964023890 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.255+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9701a33b0 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.256+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9701a3810 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.257+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd96401b440 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.257+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9700640a0 con 0x7fd970110220 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.257+0000 7fd96e000700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 0x7fd95803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:55.116 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.257+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd96404e020 con 0x7fd970110220 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.258+0000 7fd975200700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 0x7fd95803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.258+0000 7fd975200700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 0x7fd95803bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd960006fd0 tx=0x7fd960006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.265+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd96401f030 con 0x7fd970110220 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.499+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fd9701a65e0 con 0x7fd970110220 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.500+0000 7fd96e000700 1 -- 172.21.15.52:0/610332971 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fd9640185b0 con 0x7fd970110220 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.504+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 msgr2=0x7fd95803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.504+0000 7fd978600700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 0x7fd95803bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd960006fd0 tx=0x7fd960006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.504+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 msgr2=0x7fd9701a2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.504+0000 7fd978600700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd964000c00 tx=0x7fd964011aa0 comp rx=0 tx=0).stop 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 shutdown_connections 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd958039700 0x7fd95803bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 --2- 172.21.15.52:0/610332971 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd970110220 0x7fd9701a2be0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 >> 172.21.15.52:0/610332971 conn(0x7fd97007de20 msgr2=0x7fd97007f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 shutdown_connections 2024-04-26T01:32:55.117 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:54.505+0000 7fd978600700 1 -- 172.21.15.52:0/610332971 wait complete. 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.776+0000 7f1442c00700 1 Processor -- start 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.776+0000 7f1442c00700 1 -- start start 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.776+0000 7f1442c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.776+0000 7f1442c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f143c110c10 con 0x7f143c110220 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.777+0000 7f143be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.777+0000 7f143be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38454/0 (socket says 172.21.15.52:38454) 2024-04-26T01:32:56.738 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.777+0000 7f143be00700 1 -- 172.21.15.52:0/681627643 learned_addr learned my addr 172.21.15.52:0/681627643 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.777+0000 7f143be00700 1 -- 172.21.15.52:0/681627643 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c10ece0 con 0x7f143c110220 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.778+0000 7f143be00700 1 --2- 172.21.15.52:0/681627643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f142c0098a0 tx=0x7f142c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3571d9af354e755a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.778+0000 7f143aa00700 1 -- 172.21.15.52:0/681627643 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f142c004160 con 0x7f143c110220 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.778+0000 7f143aa00700 1 -- 172.21.15.52:0/681627643 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f142c004300 con 0x7f143c110220 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.778+0000 7f143aa00700 1 -- 172.21.15.52:0/681627643 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f142c011a20 con 0x7f143c110220 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 -- 172.21.15.52:0/681627643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 msgr2=0x7f143c110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 --2- 172.21.15.52:0/681627643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f142c0098a0 tx=0x7f142c009bb0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 -- 172.21.15.52:0/681627643 shutdown_connections 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 --2- 172.21.15.52:0/681627643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c110640 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 -- 172.21.15.52:0/681627643 >> 172.21.15.52:0/681627643 conn(0x7f143c07de20 msgr2=0x7f143c080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:56.744 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 -- 172.21.15.52:0/681627643 shutdown_connections 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.779+0000 7f1442c00700 1 -- 172.21.15.52:0/681627643 wait complete. 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.780+0000 7f1442c00700 1 Processor -- start 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.780+0000 7f1442c00700 1 -- start start 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f1442c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f1442c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f143c113140 con 0x7f143c110220 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f143be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:56.745 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f143be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38460/0 (socket says 172.21.15.52:38460) 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f143be00700 1 -- 172.21.15.52:0/672956447 learned_addr learned my addr 172.21.15.52:0/672956447 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.781+0000 7f143be00700 1 -- 172.21.15.52:0/672956447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c1a3120 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f143be00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f142c0119f0 tx=0x7f142c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f142c0039e0 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f142c01b8f0 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143c1a33b0 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f142c0237a0 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.782+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f143c1a3810 con 0x7f143c110220 2024-04-26T01:32:56.746 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.783+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f142c01b440 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.783+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f143c0640a0 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.784+0000 7f1438c00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 0x7f142003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.784+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f142c04e010 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.784+0000 7f143b400700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 0x7f142003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.785+0000 7f143b400700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 0x7f142003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1428006fd0 tx=0x7f1428006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:55.789+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f142c01f030 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.141+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f143c1a60a0 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.147+0000 7f1438c00700 1 -- 172.21.15.52:0/672956447 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f143c1a60a0 con 0x7f143c110220 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 msgr2=0x7f142003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 0x7f142003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1428006fd0 tx=0x7f1428006e40 comp rx=0 tx=0).stop 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 msgr2=0x7f143c1a2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f142c0119f0 tx=0x7f142c011ad0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.747 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 shutdown_connections 2024-04-26T01:32:56.749 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14200396b0 0x7f142003bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.749 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 --2- 172.21.15.52:0/672956447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f143c110220 0x7f143c1a2be0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:32:56.749 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 >> 172.21.15.52:0/672956447 conn(0x7f143c07de20 msgr2=0x7f143c07f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:32:56.749 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 shutdown_connections 2024-04-26T01:32:56.749 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-04-26T01:32:56.150+0000 7f1442c00700 1 -- 172.21.15.52:0/672956447 wait complete. 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T01:32:56.750 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:32:56.751 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2024-04-26T01:32:56.839 INFO:tasks.cephadm:Fetching config... 2024-04-26T01:32:56.840 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:32:56.840 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T01:32:56.894 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T01:32:56.894 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:32:56.894 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T01:32:56.937 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T01:32:56.937 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:32:56.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T01:32:57.054 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T01:32:57.054 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:32:57.054 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T01:32:57.154 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T01:32:57.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:32:57.270 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:57.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:32:57 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/672956447' entity='client.admin' 2024-04-26T01:32:57.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:32:57.361 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:57.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T01:32:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxSWmHlDFknPs+LVmDiTB4LZF7WUdTPpb5X97ndZaCewnpX4DIeEsdb8cc6Hn+tElCRbgaoQCi7IJwmHYjxqc29ttmxr3zUv8oFaLENJ7yy0wGbS4gFUKDlf9gxUvo2mJs4C2Q9l0qH2S4UBXVeK1FwimMwxTt+B5j7dN5rbzWKAOVPfAl8q7R3W/iCiaYWQAw4UKKBFnQc06o4b022rLwdd79EfkQKqN2vXLq/0y4JaqJLZSyReLtiTURECJ/ZBqN9tW3DZZc4WDYrBL1NXFzREXs10y9WXhMB6PajBR4YykUp/ut5HAGgDfdb98mamfKPNR9ehJLqqLfLhcOzTDkGhG7+AZdKXheYGaciG3f/ZubAmBbyWm0WuZD1bcrcZbGLGMx8eHqyu81HRWYK0CV3wzbX4osJFw+Ua+I/s2DUkVxsHbsYq3TLdkTCkomK7MRpEP2cDVUCMBLVggo6Rf92XucbGa8zzKTjImyQnanf9RBgliElIarhfvT2xZCtCk= ceph-8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:32:57.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T01:32:57.852 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:00.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.237+0000 7fbae7e00700 1 -- 172.21.15.52:0/1321213089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 msgr2=0x7fbae01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.237+0000 7fbae7e00700 1 --2- 172.21.15.52:0/1321213089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae01028c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009910 tx=0x7fbad4009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.238+0000 7fbae7e00700 1 -- 172.21.15.52:0/1321213089 shutdown_connections 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.238+0000 7fbae7e00700 1 --2- 172.21.15.52:0/1321213089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae01028c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.238+0000 7fbae7e00700 1 -- 172.21.15.52:0/1321213089 >> 172.21.15.52:0/1321213089 conn(0x7fbae00fda40 msgr2=0x7fbae00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.238+0000 7fbae7e00700 1 -- 172.21.15.52:0/1321213089 shutdown_connections 2024-04-26T01:33:00.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.238+0000 7fbae7e00700 1 -- 172.21.15.52:0/1321213089 wait complete. 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.239+0000 7fbae7e00700 1 Processor -- start 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.239+0000 7fbae7e00700 1 -- start start 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.239+0000 7fbae7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.239+0000 7fbae7e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01083e0 con 0x7fbae01024a0 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.240+0000 7fbae5400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.240+0000 7fbae5400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38476/0 (socket says 172.21.15.52:38476) 2024-04-26T01:33:00.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.240+0000 7fbae5400700 1 -- 172.21.15.52:0/2772249933 learned_addr learned my addr 172.21.15.52:0/2772249933 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.240+0000 7fbae5400700 1 -- 172.21.15.52:0/2772249933 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae019a240 con 0x7fbae01024a0 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbae5400700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbad4005950 tx=0x7fbad4004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbad4026070 con 0x7fbae01024a0 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad400bc00 con 0x7fbae01024a0 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae019a4d0 con 0x7fbae01024a0 2024-04-26T01:33:00.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbad4003a10 con 0x7fbae01024a0 2024-04-26T01:33:00.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.241+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae019d1b0 con 0x7fbae01024a0 2024-04-26T01:33:00.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.242+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbad4029070 con 0x7fbae01024a0 2024-04-26T01:33:00.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.242+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae0108b60 con 0x7fbae01024a0 2024-04-26T01:33:00.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.243+0000 7fbade000700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 0x7fbac803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:00.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.243+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbad401e070 con 0x7fbae01024a0 2024-04-26T01:33:00.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.243+0000 7fbae4a00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 0x7fbac803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:00.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.244+0000 7fbae4a00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 0x7fbac803bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbacc006fd0 tx=0x7fbacc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:00.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.249+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbad4013070 con 0x7fbae01024a0 2024-04-26T01:33:00.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.487+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbae002e090 con 0x7fbae01024a0 2024-04-26T01:33:00.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.493+0000 7fbade000700 1 -- 172.21.15.52:0/2772249933 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbad4030020 con 0x7fbae01024a0 2024-04-26T01:33:00.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 msgr2=0x7fbac803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 0x7fbac803bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbacc006fd0 tx=0x7fbacc006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 msgr2=0x7fbae0199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbad4005950 tx=0x7fbad4004fa0 comp rx=0 tx=0).stop 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 shutdown_connections 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbac80396b0 0x7fbac803bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 --2- 172.21.15.52:0/2772249933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbae01024a0 0x7fbae0199d00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.496+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 >> 172.21.15.52:0/2772249933 conn(0x7fbae00fda40 msgr2=0x7fbae00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.497+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 shutdown_connections 2024-04-26T01:33:00.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:00.497+0000 7fbae7e00700 1 -- 172.21.15.52:0/2772249933 wait complete. 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2772249933' entity='client.admin' 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:01.418 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T01:33:01.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T01:33:01.849 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 -- 172.21.15.52:0/3834835273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 msgr2=0x7f94bc109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 --2- 172.21.15.52:0/3834835273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc109070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009910 tx=0x7f94ac009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 -- 172.21.15.52:0/3834835273 shutdown_connections 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 --2- 172.21.15.52:0/3834835273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc109070 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 -- 172.21.15.52:0/3834835273 >> 172.21.15.52:0/3834835273 conn(0x7f94bc077b30 msgr2=0x7f94bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 -- 172.21.15.52:0/3834835273 shutdown_connections 2024-04-26T01:33:04.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.411+0000 7f94c1400700 1 -- 172.21.15.52:0/3834835273 wait complete. 2024-04-26T01:33:04.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.413+0000 7f94c1400700 1 Processor -- start 2024-04-26T01:33:04.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.413+0000 7f94c1400700 1 -- start start 2024-04-26T01:33:04.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.413+0000 7f94c1400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:04.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.413+0000 7f94c1400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10ea20 con 0x7f94bc108c50 2024-04-26T01:33:04.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.414+0000 7f94ba800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:04.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.414+0000 7f94ba800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38492/0 (socket says 172.21.15.52:38492) 2024-04-26T01:33:04.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.414+0000 7f94ba800700 1 -- 172.21.15.52:0/1569393840 learned_addr learned my addr 172.21.15.52:0/1569393840 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:04.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.414+0000 7f94ba800700 1 -- 172.21.15.52:0/1569393840 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc19e650 con 0x7f94bc108c50 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.414+0000 7f94ba800700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94ac005800 tx=0x7f94ac009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.415+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac027070 con 0x7f94bc108c50 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.415+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94ac00be10 con 0x7f94bc108c50 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.415+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac00f460 con 0x7f94bc108c50 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.415+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc19e8e0 con 0x7f94bc108c50 2024-04-26T01:33:04.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.415+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1a15d0 con 0x7f94bc108c50 2024-04-26T01:33:04.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.416+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f94ac02a070 con 0x7f94bc108c50 2024-04-26T01:33:04.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.416+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc10f1a0 con 0x7f94bc108c50 2024-04-26T01:33:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.417+0000 7f94b3400700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 0x7f94a003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.417+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f94ac01e030 con 0x7f94bc108c50 2024-04-26T01:33:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.417+0000 7f94b9e00700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 0x7f94a003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.418+0000 7f94b9e00700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 0x7f94a003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006fd0 tx=0x7f94a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:04.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.422+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94ac013070 con 0x7f94bc108c50 2024-04-26T01:33:04.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.659+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f94bc02e090 con 0x7f94a00396b0 2024-04-26T01:33:04.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.666+0000 7f94b3400700 1 -- 172.21.15.52:0/1569393840 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f94bc02e090 con 0x7f94a00396b0 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 msgr2=0x7f94a003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 0x7f94a003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f94a8006fd0 tx=0x7f94a8006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 msgr2=0x7f94bc19e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94ac005800 tx=0x7f94ac009f60 comp rx=0 tx=0).stop 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 shutdown_connections 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f94a00396b0 0x7f94a003bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 --2- 172.21.15.52:0/1569393840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94bc108c50 0x7f94bc19e110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.669+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 >> 172.21.15.52:0/1569393840 conn(0x7f94bc077b30 msgr2=0x7f94bc078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.670+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 shutdown_connections 2024-04-26T01:33:04.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:04.670+0000 7f94c1400700 1 -- 172.21.15.52:0/1569393840 wait complete. 2024-04-26T01:33:05.420 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-04-26T01:33:05.420 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:33:05.420 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:33:05.439 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:33:05.440 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:05.496 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-04-26T01:33:05.497 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch host add smithi183 2024-04-26T01:33:05.931 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:05.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:05.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:05.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:05.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:05.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:05.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:33:05.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:05 smithi052 ceph-mon[27851]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:06 smithi052 ceph-mon[27851]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:06 smithi052 ceph-mon[27851]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:07.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:07 smithi052 ceph-mon[27851]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:33:07.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:07.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:07.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:08.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 -- 172.21.15.52:0/143775826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 msgr2=0x7faae410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:08.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 --2- 172.21.15.52:0/143775826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae410a960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faad8009910 tx=0x7faad8009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 -- 172.21.15.52:0/143775826 shutdown_connections 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 --2- 172.21.15.52:0/143775826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae410a960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 -- 172.21.15.52:0/143775826 >> 172.21.15.52:0/143775826 conn(0x7faae40fffc0 msgr2=0x7faae41023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 -- 172.21.15.52:0/143775826 shutdown_connections 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.108+0000 7faaea400700 1 -- 172.21.15.52:0/143775826 wait complete. 2024-04-26T01:33:08.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.109+0000 7faaea400700 1 Processor -- start 2024-04-26T01:33:08.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faaea400700 1 -- start start 2024-04-26T01:33:08.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faaea400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:08.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faaea400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faae410e9f0 con 0x7faae410a580 2024-04-26T01:33:08.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faae9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:08.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faae9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45976/0 (socket says 172.21.15.52:45976) 2024-04-26T01:33:08.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.110+0000 7faae9000700 1 -- 172.21.15.52:0/1134170163 learned_addr learned my addr 172.21.15.52:0/1134170163 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:08.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faae9000700 1 -- 172.21.15.52:0/1134170163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae419a3d0 con 0x7faae410a580 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faae9000700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faad8006010 tx=0x7faad8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faad8026070 con 0x7faae410a580 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae419a660 con 0x7faae410a580 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad800bc00 con 0x7faae410a580 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.111+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faad8003910 con 0x7faae410a580 2024-04-26T01:33:08.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.112+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae419e760 con 0x7faae410a580 2024-04-26T01:33:08.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.113+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7faad8029070 con 0x7faae410a580 2024-04-26T01:33:08.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.113+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faae4105bf0 con 0x7faae410a580 2024-04-26T01:33:08.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.113+0000 7faae1600700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 0x7faacc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:08.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.113+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faad801e070 con 0x7faae410a580 2024-04-26T01:33:08.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.114+0000 7faae3e00700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 0x7faacc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:08.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.114+0000 7faae3e00700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 0x7faacc03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:08.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.119+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faad8013070 con 0x7faae410a580 2024-04-26T01:33:08.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:08.354+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}) -- 0x7faae419b1f0 con 0x7faacc0396b0 2024-04-26T01:33:08.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:08 smithi052 ceph-mon[27851]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:08.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:08 smithi052 ceph-mon[27851]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:09.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:09 smithi052 ceph-mon[27851]: Deploying cephadm binary to smithi183 2024-04-26T01:33:10.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:10 smithi052 ceph-mon[27851]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:12.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:12 smithi052 ceph-mon[27851]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:13.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.229+0000 7faae1600700 1 -- 172.21.15.52:0/1134170163 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7faae419b1f0 con 0x7faacc0396b0 2024-04-26T01:33:13.225 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 msgr2=0x7faacc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 0x7faacc03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 msgr2=0x7faae4199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faad8006010 tx=0x7faad8005780 comp rx=0 tx=0).stop 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 shutdown_connections 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faacc0396b0 0x7faacc03bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 --2- 172.21.15.52:0/1134170163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faae410a580 0x7faae4199e90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.233+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 >> 172.21.15.52:0/1134170163 conn(0x7faae40fffc0 msgr2=0x7faae4101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:13.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.234+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 shutdown_connections 2024-04-26T01:33:13.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:13.234+0000 7faaea400700 1 -- 172.21.15.52:0/1134170163 wait complete. 2024-04-26T01:33:13.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:33:14.433 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:14.581 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:14.581 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:14 smithi052 ceph-mon[27851]: Added host smithi183 2024-04-26T01:33:14.582 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:14.582 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:15.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:15 smithi052 ceph-mon[27851]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:15.601 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:15 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 -- 172.21.15.52:0/3716816709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 msgr2=0x7f964c077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 --2- 172.21.15.52:0/3716816709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c077350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f963c009910 tx=0x7f963c009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 -- 172.21.15.52:0/3716816709 shutdown_connections 2024-04-26T01:33:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 --2- 172.21.15.52:0/3716816709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c077350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 -- 172.21.15.52:0/3716816709 >> 172.21.15.52:0/3716816709 conn(0x7f964c100190 msgr2=0x7f964c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.493+0000 7f9651400700 1 -- 172.21.15.52:0/3716816709 shutdown_connections 2024-04-26T01:33:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.494+0000 7f9651400700 1 -- 172.21.15.52:0/3716816709 wait complete. 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f9651400700 1 Processor -- start 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f9651400700 1 -- start start 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f9651400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f9651400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f964c112fa0 con 0x7f964c076f70 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f964a800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f964a800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37076/0 (socket says 172.21.15.52:37076) 2024-04-26T01:33:16.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.495+0000 7f964a800700 1 -- 172.21.15.52:0/1309424351 learned_addr learned my addr 172.21.15.52:0/1309424351 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:16.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.496+0000 7f964a800700 1 -- 172.21.15.52:0/1309424351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c19e9b0 con 0x7f964c076f70 2024-04-26T01:33:16.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.496+0000 7f964a800700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f963c006010 tx=0x7f963c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:16.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.496+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f963c026070 con 0x7f964c076f70 2024-04-26T01:33:16.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.497+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f963c00bc00 con 0x7f964c076f70 2024-04-26T01:33:16.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.497+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964c19ec40 con 0x7f964c076f70 2024-04-26T01:33:16.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.497+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f963c003910 con 0x7f964c076f70 2024-04-26T01:33:16.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.497+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964c19f0a0 con 0x7f964c076f70 2024-04-26T01:33:16.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.498+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f963c029070 con 0x7f964c076f70 2024-04-26T01:33:16.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.498+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964c0786b0 con 0x7f964c076f70 2024-04-26T01:33:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.499+0000 7f9643400700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 0x7f963003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.499+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f963c01e070 con 0x7f964c076f70 2024-04-26T01:33:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.499+0000 7f9649e00700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 0x7f963003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:16.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.500+0000 7f9649e00700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 0x7f963003bbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9638006fd0 tx=0x7f9638006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.505+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f963c013070 con 0x7f964c076f70 2024-04-26T01:33:16.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.751+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f964c19f840 con 0x7f9630039700 2024-04-26T01:33:16.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.753+0000 7f9643400700 1 -- 172.21.15.52:0/1309424351 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f964c19f840 con 0x7f9630039700 2024-04-26T01:33:16.748 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:33:16.749 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-04-26T01:33:16.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.756+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 msgr2=0x7f963003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.756+0000 7f9651400700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 0x7f963003bbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9638006fd0 tx=0x7f9638006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:16.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.756+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 msgr2=0x7f964c19e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:16.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.756+0000 7f9651400700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f963c006010 tx=0x7f963c005780 comp rx=0 tx=0).stop 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 shutdown_connections 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9630039700 0x7f963003bbc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 --2- 172.21.15.52:0/1309424351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f964c076f70 0x7f964c19e470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 >> 172.21.15.52:0/1309424351 conn(0x7f964c100190 msgr2=0x7f964c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 shutdown_connections 2024-04-26T01:33:16.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:16.757+0000 7f9651400700 1 -- 172.21.15.52:0/1309424351 wait complete. 2024-04-26T01:33:17.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:16 smithi052 ceph-mon[27851]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:17.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:16 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:17.432 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-04-26T01:33:17.432 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:33:17.433 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-04-26T01:33:17.451 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:33:17.452 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:17.509 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-04-26T01:33:17.509 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch host add smithi191 2024-04-26T01:33:17.927 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:18.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:17 smithi052 ceph-mon[27851]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:33:19.105 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:18 smithi052 ceph-mon[27851]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 -- 172.21.15.52:0/1186852503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4010d960 msgr2=0x7f8d4010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 --2- 172.21.15.52:0/1186852503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4010d960 0x7f8d4010dd40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009910 tx=0x7f8d34009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 -- 172.21.15.52:0/1186852503 shutdown_connections 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 --2- 172.21.15.52:0/1186852503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4010d960 0x7f8d4010dd40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 -- 172.21.15.52:0/1186852503 >> 172.21.15.52:0/1186852503 conn(0x7f8d400781e0 msgr2=0x7f8d4007a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.015+0000 7f8d48600700 1 -- 172.21.15.52:0/1186852503 shutdown_connections 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.016+0000 7f8d48600700 1 -- 172.21.15.52:0/1186852503 wait complete. 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d48600700 1 Processor -- start 2024-04-26T01:33:20.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d48600700 1 -- start start 2024-04-26T01:33:20.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d48600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:20.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d48600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8d40112f60 con 0x7f8d4019e690 2024-04-26T01:33:20.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d45c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d45c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37092/0 (socket says 172.21.15.52:37092) 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.017+0000 7f8d45c00700 1 -- 172.21.15.52:0/2896573480 learned_addr learned my addr 172.21.15.52:0/2896573480 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.018+0000 7f8d45c00700 1 -- 172.21.15.52:0/2896573480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4019efb0 con 0x7f8d4019e690 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.018+0000 7f8d45c00700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400b5c0 tx=0x7f8d34005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.018+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d34026070 con 0x7f8d4019e690 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.019+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d3400b6f0 con 0x7f8d4019e690 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.019+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4019f2a0 con 0x7f8d4019e690 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.019+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d401a3130 con 0x7f8d4019e690 2024-04-26T01:33:20.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.019+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d340037d0 con 0x7f8d4019e690 2024-04-26T01:33:20.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.020+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8d34029070 con 0x7f8d4019e690 2024-04-26T01:33:20.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.020+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d20005360 con 0x7f8d4019e690 2024-04-26T01:33:20.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.020+0000 7f8d3e000700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 0x7f8d2803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:20.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.021+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8d3401e070 con 0x7f8d4019e690 2024-04-26T01:33:20.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.023+0000 7f8d45200700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 0x7f8d2803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:20.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.024+0000 7f8d45200700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 0x7f8d2803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c006fd0 tx=0x7f8d2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:20.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.027+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d34013070 con 0x7f8d4019e690 2024-04-26T01:33:20.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:20.262+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}) -- 0x7f8d20000bf0 con 0x7f8d280396b0 2024-04-26T01:33:20.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:20.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:20.257 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:33:20.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:19 smithi052 ceph-mon[27851]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:21 smithi052 ceph-mon[27851]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:22.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:22 smithi052 ceph-mon[27851]: Deploying cephadm binary to smithi191 2024-04-26T01:33:23.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:23 smithi052 ceph-mon[27851]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:25.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.137+0000 7f8d3e000700 1 -- 172.21.15.52:0/2896573480 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8d20000bf0 con 0x7f8d280396b0 2024-04-26T01:33:25.135 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-04-26T01:33:25.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 msgr2=0x7f8d2803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:25.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 0x7f8d2803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c006fd0 tx=0x7f8d2c006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:25.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 msgr2=0x7f8d4019ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:25.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400b5c0 tx=0x7f8d34005780 comp rx=0 tx=0).stop 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 shutdown_connections 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8d280396b0 0x7f8d2803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 --2- 172.21.15.52:0/2896573480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d4019e690 0x7f8d4019ea70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 >> 172.21.15.52:0/2896573480 conn(0x7f8d400781e0 msgr2=0x7f8d4007a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 shutdown_connections 2024-04-26T01:33:25.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:25.142+0000 7f8d48600700 1 -- 172.21.15.52:0/2896573480 wait complete. 2024-04-26T01:33:25.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:25 smithi052 ceph-mon[27851]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:25.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:25.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:25.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T01:33:26.349 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:26.497 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:26 smithi052 ceph-mon[27851]: Added host smithi191 2024-04-26T01:33:26.497 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:26.497 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:27.543 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:27 smithi052 ceph-mon[27851]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:28.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 -- 172.21.15.52:0/2474044431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 msgr2=0x7faa581088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:28.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 --2- 172.21.15.52:0/2474044431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa581088c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faa48009910 tx=0x7faa48009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:28.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 -- 172.21.15.52:0/2474044431 shutdown_connections 2024-04-26T01:33:28.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 --2- 172.21.15.52:0/2474044431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa581088c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:28.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 -- 172.21.15.52:0/2474044431 >> 172.21.15.52:0/2474044431 conn(0x7faa580fdf80 msgr2=0x7faa581003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:28.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 -- 172.21.15.52:0/2474044431 shutdown_connections 2024-04-26T01:33:28.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.373+0000 7faa5e000700 1 -- 172.21.15.52:0/2474044431 wait complete. 2024-04-26T01:33:28.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.375+0000 7faa5e000700 1 Processor -- start 2024-04-26T01:33:28.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.375+0000 7faa5e000700 1 -- start start 2024-04-26T01:33:28.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa5e000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:28.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa5e000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faa5810c940 con 0x7faa581084e0 2024-04-26T01:33:28.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa57400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:28.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa57400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36338/0 (socket says 172.21.15.52:36338) 2024-04-26T01:33:28.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa57400700 1 -- 172.21.15.52:0/308736611 learned_addr learned my addr 172.21.15.52:0/308736611 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:28.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.376+0000 7faa57400700 1 -- 172.21.15.52:0/308736611 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa58076c30 con 0x7faa581084e0 2024-04-26T01:33:28.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa57400700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa48006010 tx=0x7faa48005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:28.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa48026070 con 0x7faa581084e0 2024-04-26T01:33:28.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa4800bc00 con 0x7faa581084e0 2024-04-26T01:33:28.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa48003910 con 0x7faa581084e0 2024-04-26T01:33:28.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa580752e0 con 0x7faa581084e0 2024-04-26T01:33:28.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.377+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa580757a0 con 0x7faa581084e0 2024-04-26T01:33:28.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.379+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa58103b90 con 0x7faa581084e0 2024-04-26T01:33:28.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.379+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7faa48029070 con 0x7faa581084e0 2024-04-26T01:33:28.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.380+0000 7faa54c00700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 0x7faa3c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:28.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.380+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faa4801e070 con 0x7faa581084e0 2024-04-26T01:33:28.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.380+0000 7faa4fe00700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 0x7faa3c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:28.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.381+0000 7faa4fe00700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 0x7faa3c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:28.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.386+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa48013070 con 0x7faa581084e0 2024-04-26T01:33:28.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.624+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa58076270 con 0x7faa3c0396b0 2024-04-26T01:33:28.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.626+0000 7faa54c00700 1 -- 172.21.15.52:0/308736611 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7faa58076270 con 0x7faa3c0396b0 2024-04-26T01:33:28.621 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:33:28.621 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 msgr2=0x7faa3c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 0x7faa3c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 msgr2=0x7faa58074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa48006010 tx=0x7faa48005780 comp rx=0 tx=0).stop 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 shutdown_connections 2024-04-26T01:33:28.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa3c0396b0 0x7faa3c03bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:28.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 --2- 172.21.15.52:0/308736611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa581084e0 0x7faa58074d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:28.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 >> 172.21.15.52:0/308736611 conn(0x7faa580fdf80 msgr2=0x7faa5810a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:28.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.629+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 shutdown_connections 2024-04-26T01:33:28.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:28.630+0000 7faa5e000700 1 -- 172.21.15.52:0/308736611 wait complete. 2024-04-26T01:33:29.300 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:28 smithi052 ceph-mon[27851]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:29.300 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:29.361 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T01:33:29.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T01:33:29.814 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:30.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:29 smithi052 ceph-mon[27851]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:33:31.123 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:30 smithi052 ceph-mon[27851]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:31.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.002+0000 7ff1fd200700 1 -- 172.21.15.52:0/3774083517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 msgr2=0x7ff1f810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:31.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.002+0000 7ff1fd200700 1 --2- 172.21.15.52:0/3774083517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f810dd40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009910 tx=0x7ff1e8009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:31.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.002+0000 7ff1fd200700 1 -- 172.21.15.52:0/3774083517 shutdown_connections 2024-04-26T01:33:31.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.002+0000 7ff1fd200700 1 --2- 172.21.15.52:0/3774083517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f810dd40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:31.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.002+0000 7ff1fd200700 1 -- 172.21.15.52:0/3774083517 >> 172.21.15.52:0/3774083517 conn(0x7ff1f80781e0 msgr2=0x7ff1f807a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:31.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.003+0000 7ff1fd200700 1 -- 172.21.15.52:0/3774083517 shutdown_connections 2024-04-26T01:33:31.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.003+0000 7ff1fd200700 1 -- 172.21.15.52:0/3774083517 wait complete. 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.004+0000 7ff1fd200700 1 Processor -- start 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.004+0000 7ff1fd200700 1 -- start start 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1fd200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1fd200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f8112f60 con 0x7ff1f810d960 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1f5e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1f5e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36360/0 (socket says 172.21.15.52:36360) 2024-04-26T01:33:32.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1f5e00700 1 -- 172.21.15.52:0/1212280095 learned_addr learned my addr 172.21.15.52:0/1212280095 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.005+0000 7ff1f5e00700 1 -- 172.21.15.52:0/1212280095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f819ea40 con 0x7ff1f810d960 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1f5e00700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006010 tx=0x7ff1e8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1e8026070 con 0x7ff1f810d960 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e800bba0 con 0x7ff1f810d960 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f819ed30 con 0x7ff1f810d960 2024-04-26T01:33:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1e8003930 con 0x7ff1f810d960 2024-04-26T01:33:32.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.006+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f81a2dd0 con 0x7ff1f810d960 2024-04-26T01:33:32.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.007+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff1e8029070 con 0x7ff1f810d960 2024-04-26T01:33:32.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.007+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f810a180 con 0x7ff1f810d960 2024-04-26T01:33:32.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.008+0000 7ff1eea00700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 0x7ff1dc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:32.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.008+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff1e801e070 con 0x7ff1f810d960 2024-04-26T01:33:32.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.008+0000 7ff1f5400700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 0x7ff1dc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:32.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.009+0000 7ff1f5400700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 0x7ff1dc03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4006fd0 tx=0x7ff1e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:32.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.013+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1e8013070 con 0x7ff1f810d960 2024-04-26T01:33:32.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:32.254+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff1f80640a0 con 0x7ff1f810d960 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:32.250 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:32.251 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:32 smithi052 ceph-mon[27851]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:33:33.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.172+0000 7ff1eea00700 1 -- 172.21.15.52:0/1212280095 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7ff1e800be00 con 0x7ff1f810d960 2024-04-26T01:33:33.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 msgr2=0x7ff1dc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 0x7ff1dc03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4006fd0 tx=0x7ff1e4006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 msgr2=0x7ff1f819e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006010 tx=0x7ff1e8005780 comp rx=0 tx=0).stop 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 shutdown_connections 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1dc0396b0 0x7ff1dc03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 --2- 172.21.15.52:0/1212280095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f810d960 0x7ff1f819e500 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 >> 172.21.15.52:0/1212280095 conn(0x7ff1f80781e0 msgr2=0x7ff1f8110580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 shutdown_connections 2024-04-26T01:33:33.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:33.176+0000 7ff1fd200700 1 -- 172.21.15.52:0/1212280095 wait complete. 2024-04-26T01:33:33.173 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1212280095' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:33.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:33.911 INFO:tasks.cephadm:Adding mon.a on smithi052 2024-04-26T01:33:33.911 INFO:tasks.cephadm:Adding mon.b on smithi183 2024-04-26T01:33:33.911 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-04-26T01:33:33.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi052:172.21.15.52=a;smithi183:172.21.15.183=b;smithi191:172.21.15.191=c' 2024-04-26T01:33:34.147 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:34.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:34 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1212280095' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T01:33:34.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:34 smithi052 ceph-mon[27851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:35.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:35 smithi052 ceph-mon[27851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:35.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2464129825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 msgr2=0x7fa9ac075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2464129825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac075140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009910 tx=0x7fa99c009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2464129825 shutdown_connections 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2464129825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac075140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2464129825 >> 172.21.15.191:0/2464129825 conn(0x7fa9ac0fdf80 msgr2=0x7fa9ac1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2464129825 shutdown_connections 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.672+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2464129825 wait complete. 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.673+0000 7fa9b0ee8700 1 Processor -- start 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.673+0000 7fa9b0ee8700 1 -- start start 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.673+0000 7fa9b0ee8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.673+0000 7fa9b0ee8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac10eb90 con 0x7fa9ac074d60 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.674+0000 7fa9aa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.674+0000 7fa9aa59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.191:44880/0 (socket says 172.21.15.191:44880) 2024-04-26T01:33:35.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.674+0000 7fa9aa59c700 1 -- 172.21.15.191:0/2215001987 learned_addr learned my addr 172.21.15.191:0/2215001987 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.674+0000 7fa9aa59c700 1 -- 172.21.15.191:0/2215001987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac19a630 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa9aa59c700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0056d0 tx=0x7fa99c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa99c026070 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ac19a8c0 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa99c004370 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ac19ad20 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.675+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa99c0039b0 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.676+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ac0764a0 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.677+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa99c029070 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.677+0000 7fa99b7fe700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 0x7fa99403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.677+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa99c01e070 con 0x7fa9ac074d60 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.678+0000 7fa9a9d9b700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 0x7fa99403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:35.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.679+0000 7fa9a9d9b700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 0x7fa99403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0006fd0 tx=0x7fa9a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:35.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.693+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa99c030020 con 0x7fa9ac074d60 2024-04-26T01:33:35.898 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.903+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi183:172.21.15.183=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}) -- 0x7fa9ac19b510 con 0x7fa994039650 2024-04-26T01:33:35.913 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.919+0000 7fa99b7fe700 1 -- 172.21.15.191:0/2215001987 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa9ac19b510 con 0x7fa994039650 2024-04-26T01:33:35.913 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 msgr2=0x7fa99403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 0x7fa99403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0006fd0 tx=0x7fa9a0006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 msgr2=0x7fa9ac19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0056d0 tx=0x7fa99c0057b0 comp rx=0 tx=0).stop 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 shutdown_connections 2024-04-26T01:33:35.916 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa994039650 0x7fa99403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:35.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 --2- 172.21.15.191:0/2215001987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9ac074d60 0x7fa9ac19a0f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:35.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 >> 172.21.15.191:0/2215001987 conn(0x7fa9ac0fdf80 msgr2=0x7fa9ac106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:35.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 shutdown_connections 2024-04-26T01:33:35.917 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:35.920+0000 7fa9b0ee8700 1 -- 172.21.15.191:0/2215001987 wait complete. 2024-04-26T01:33:36.524 DEBUG:teuthology.orchestra.run.smithi183:mon.b> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T01:33:36.527 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T01:33:36.529 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:33:36.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi183:172.21.15.183=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi183:172.21.15.183=b;smithi191:172.21.15.191=c;count:3 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:37.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:36 smithi052 ceph-mon[27851]: Deploying daemon mon.c on smithi191 2024-04-26T01:33:37.240 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:38 smithi052 ceph-mon[27851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/4002024615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 msgr2=0x7f1b88101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/4002024615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88101970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009910 tx=0x7f1b78009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/4002024615 shutdown_connections 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/4002024615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88101970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/4002024615 >> 172.21.15.191:0/4002024615 conn(0x7f1b880fce40 msgr2=0x7f1b880ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/4002024615 shutdown_connections 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.208+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/4002024615 wait complete. 2024-04-26T01:33:40.205 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.209+0000 7f1b8ea3a700 1 Processor -- start 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.209+0000 7f1b8ea3a700 1 -- start start 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.209+0000 7f1b8ea3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.209+0000 7f1b8ea3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8810b810 con 0x7f1b88101590 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.210+0000 7f1b87fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.210+0000 7f1b87fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.191:42170/0 (socket says 172.21.15.191:42170) 2024-04-26T01:33:40.206 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.210+0000 7f1b87fff700 1 -- 172.21.15.191:0/1250404446 learned_addr learned my addr 172.21.15.191:0/1250404446 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.210+0000 7f1b87fff700 1 -- 172.21.15.191:0/1250404446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b88199230 con 0x7f1b88101590 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b87fff700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1b780056d0 tx=0x7f1b780057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b78026070 con 0x7f1b88101590 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b88199520 con 0x7f1b88101590 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b881999e0 con 0x7f1b88101590 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b7800b6f0 con 0x7f1b88101590 2024-04-26T01:33:40.207 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.211+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b78030060 con 0x7f1b88101590 2024-04-26T01:33:40.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.212+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b74005330 con 0x7f1b88101590 2024-04-26T01:33:40.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.212+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1b78029070 con 0x7f1b88101590 2024-04-26T01:33:40.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.213+0000 7f1b857fa700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 0x7f1b7003baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:40.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.213+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1b7801e070 con 0x7f1b88101590 2024-04-26T01:33:40.208 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.213+0000 7f1b877fe700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 0x7f1b7003baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:40.209 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.214+0000 7f1b877fe700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 0x7f1b7003baf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c006fd0 tx=0x7f1b7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:40.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.229+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b78013070 con 0x7f1b88101590 2024-04-26T01:33:40.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.533+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1b74005620 con 0x7f1b88101590 2024-04-26T01:33:40.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.534+0000 7f1b857fa700 1 -- 172.21.15.191:0/1250404446 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f1b7800b8a0 con 0x7f1b88101590 2024-04-26T01:33:40.529 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:33:40.529 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:31:23.102434Z","created":"2024-04-26T01:31:23.102434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T01:33:40.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 msgr2=0x7f1b7003baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:40.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 0x7f1b7003baf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c006fd0 tx=0x7f1b7c006e40 comp rx=0 tx=0).stop 2024-04-26T01:33:40.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 msgr2=0x7f1b88198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:40.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1b780056d0 tx=0x7f1b780057b0 comp rx=0 tx=0).stop 2024-04-26T01:33:40.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 shutdown_connections 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b70039630 0x7f1b7003baf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 --2- 172.21.15.191:0/1250404446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88101590 0x7f1b88198cf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 >> 172.21.15.191:0/1250404446 conn(0x7f1b880fce40 msgr2=0x7f1b88102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 shutdown_connections 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:40.536+0000 7f1b8ea3a700 1 -- 172.21.15.191:0/1250404446 wait complete. 2024-04-26T01:33:40.532 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-04-26T01:33:41.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:40 smithi191 systemd[1]: Started Ceph mon.c for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:33:42.508 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:33:42.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:33:42.752 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: mon.a calling monitor election 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: mon.c calling monitor election 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.091 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: monmap epoch 2 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: last_changed 2024-04-26T01:33:40.746403+0000 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: min_mon_release 19 (squid) 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: election_strategy: 1 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: fsmap 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: mgrmap e13: x(active, since 60s) 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T01:33:46.092 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:46.093 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: mon.a calling monitor election 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: mon.c calling monitor election 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: monmap epoch 2 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: last_changed 2024-04-26T01:33:40.746403+0000 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: min_mon_release 19 (squid) 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: election_strategy: 1 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: fsmap 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: mgrmap e13: x(active, since 60s) 2024-04-26T01:33:46.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T01:33:46.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:46.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:46.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:47.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:46 smithi191 ceph-mon[32745]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:47.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:46 smithi191 ceph-mon[32745]: Deploying daemon mon.b on smithi183 2024-04-26T01:33:47.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:47.208 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:46 smithi052 ceph-mon[27851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:47.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:46 smithi052 ceph-mon[27851]: Deploying daemon mon.b on smithi183 2024-04-26T01:33:47.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:47.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 -- 172.21.15.191:0/1229059781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 msgr2=0x7faa18005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:47.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 --2- 172.21.15.191:0/1229059781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa18005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faa2c06ad30 tx=0x7faa2800feb0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 -- 172.21.15.191:0/1229059781 shutdown_connections 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 --2- 172.21.15.191:0/1229059781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa18005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 -- 172.21.15.191:0/1229059781 >> 172.21.15.191:0/1229059781 conn(0x7faa2c0fcf70 msgr2=0x7faa2c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 -- 172.21.15.191:0/1229059781 shutdown_connections 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.288+0000 7faa335f4700 1 -- 172.21.15.191:0/1229059781 wait complete. 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 Processor -- start 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 -- start start 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa2c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c10a490 con 0x7faa180035c0 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa335f4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c10a310 con 0x7faa2c1018a0 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:57500/0 (socket says 172.21.15.191:57500) 2024-04-26T01:33:47.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 -- 172.21.15.191:0/2302620522 learned_addr learned my addr 172.21.15.191:0/2302620522 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 -- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 msgr2=0x7faa2c101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 -- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 msgr2=0x7faa2c101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.289+0000 7faa30b8f700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.290+0000 7faa31390700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa2c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.290+0000 7faa31390700 1 -- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 msgr2=0x7faa2c101d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.290+0000 7faa31390700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.290+0000 7faa31390700 1 -- 172.21.15.191:0/2302620522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c103280 con 0x7faa180035c0 2024-04-26T01:33:47.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa31390700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa2c101360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa28010010 tx=0x7faa28005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:47.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7faa28004aa0 con 0x7faa180035c0 2024-04-26T01:33:47.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c1a42c0 con 0x7faa180035c0 2024-04-26T01:33:47.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c1a4800 con 0x7faa180035c0 2024-04-26T01:33:47.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa28004c40 con 0x7faa180035c0 2024-04-26T01:33:47.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.291+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7faa280044f0 con 0x7faa180035c0 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.292+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa18002160 con 0x7faa180035c0 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.292+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7faa28023070 con 0x7faa180035c0 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.293+0000 7faa227fc700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 0x7faa1003b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.293+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7faa28020050 con 0x7faa180035c0 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.293+0000 7faa30b8f700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 0x7faa1003b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:47.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.294+0000 7faa30b8f700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 0x7faa1003b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009180 tx=0x7faa1c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:47.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.308+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa28028020 con 0x7faa180035c0 2024-04-26T01:33:47.585 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.590+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7faa18002c50 con 0x7faa180035c0 2024-04-26T01:33:47.586 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.591+0000 7faa227fc700 1 -- 172.21.15.191:0/2302620522 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7faa2800c2d0 con 0x7faa180035c0 2024-04-26T01:33:47.586 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:33:47.586 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:33:40.746403Z","created":"2024-04-26T01:31:23.102434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 msgr2=0x7faa1003b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 0x7faa1003b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009180 tx=0x7faa1c008040 comp rx=0 tx=0).stop 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 msgr2=0x7faa2c101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa2c101360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa28010010 tx=0x7faa28005020 comp rx=0 tx=0).stop 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 shutdown_connections 2024-04-26T01:33:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faa100393f0 0x7faa1003b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa180035c0 0x7faa2c101360 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 --2- 172.21.15.191:0/2302620522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa2c1018a0 0x7faa2c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 >> 172.21.15.191:0/2302620522 conn(0x7faa2c0fcf70 msgr2=0x7faa2c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 shutdown_connections 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:47.593+0000 7faa335f4700 1 -- 172.21.15.191:0/2302620522 wait complete. 2024-04-26T01:33:47.589 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-04-26T01:33:47.935 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:47 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2302620522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:33:48.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:47 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2302620522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:33:49.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:48 smithi191 ceph-mon[32745]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:49.202 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T01:33:49.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T01:33:49.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:48 smithi052 ceph-mon[27851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:49.447 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:33:50.171 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:49 smithi183 systemd[1]: Started Ceph mon.b for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:33:55.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: mon.c calling monitor election 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: mon.a calling monitor election 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:33:55.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: monmap epoch 3 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: min_mon_release 19 (squid) 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: election_strategy: 1 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: fsmap 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: mgrmap e13: x(active, since 69s) 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: mon.c calling monitor election 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: mon.a calling monitor election 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: monmap epoch 3 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: min_mon_release 19 (squid) 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: election_strategy: 1 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: fsmap 2024-04-26T01:33:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: mgrmap e13: x(active, since 69s) 2024-04-26T01:33:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T01:33:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 -- 172.21.15.191:0/3471451158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 msgr2=0x7f4180005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 --2- 172.21.15.191:0/3471451158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f4180005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f419006beb0 tx=0x7f418400fe60 comp rx=0 tx=0).stop 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 -- 172.21.15.191:0/3471451158 shutdown_connections 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 --2- 172.21.15.191:0/3471451158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f4180005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 -- 172.21.15.191:0/3471451158 >> 172.21.15.191:0/3471451158 conn(0x7f4190077ab0 msgr2=0x7f4190077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 -- 172.21.15.191:0/3471451158 shutdown_connections 2024-04-26T01:33:55.262 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.266+0000 7f4196e1c700 1 -- 172.21.15.191:0/3471451158 wait complete. 2024-04-26T01:33:55.263 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 Processor -- start 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 -- start start 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f419019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f419010a480 con 0x7f41800035c0 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f419010a300 con 0x7f419019a610 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.267+0000 7f4196e1c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f419010a600 con 0x7f419019eee0 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.191:42742/0 (socket says 172.21.15.191:42742) 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:34536/0 (socket says 172.21.15.191:34536) 2024-04-26T01:33:55.264 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 -- 172.21.15.191:0/2606402106 learned_addr learned my addr 172.21.15.191:0/2606402106 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f419019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 msgr2=0x7f419019f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 msgr2=0x7f419019f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 msgr2=0x7f419019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 msgr2=0x7f419019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f418ffff700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f41953b9700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 msgr2=0x7f419019f390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 msgr2=0x7f419019e9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.268+0000 7f4194bb8700 1 -- 172.21.15.191:0/2606402106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41901a6c50 con 0x7f41800035c0 2024-04-26T01:33:55.265 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f4194bb8700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f419019a0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f418401f000 tx=0x7f4184003ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418402b020 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41901a6e20 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41840056c0 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184022040 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.269+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41901a7350 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.270+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41900640a0 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.270+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f418402e020 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.271+0000 7f418dffb700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 0x7f417403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.271+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f418404f3b0 con 0x7f41800035c0 2024-04-26T01:33:55.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.271+0000 7f418ffff700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 0x7f417403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:55.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.272+0000 7f418ffff700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 0x7f417403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f417c009180 tx=0x7f417c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:55.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.279+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4184035020 con 0x7f41800035c0 2024-04-26T01:33:55.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.564+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f419002df80 con 0x7f41800035c0 2024-04-26T01:33:55.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.565+0000 7f418dffb700 1 -- 172.21.15.191:0/2606402106 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f418402e310 con 0x7f41800035c0 2024-04-26T01:33:55.561 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-04-26T01:33:55.561 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","modified":"2024-04-26T01:33:49.758682Z","created":"2024-04-26T01:31:23.102434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 msgr2=0x7f417403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 0x7f417403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f417c009180 tx=0x7f417c008040 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 msgr2=0x7f419019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f419019a0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f418401f000 tx=0x7f4184003ca0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 shutdown_connections 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4174039ae0 0x7f417403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f419019eee0 0x7f419019f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41800035c0 0x7f419019a0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 --2- 172.21.15.191:0/2606402106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f419019a610 0x7f419019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 >> 172.21.15.191:0/2606402106 conn(0x7f4190077ab0 msgr2=0x7f41900ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:55.564 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 shutdown_connections 2024-04-26T01:33:55.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:33:55.568+0000 7f4196e1c700 1 -- 172.21.15.191:0/2606402106 wait complete. 2024-04-26T01:33:55.565 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-04-26T01:33:56.083 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T01:33:56.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2606402106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:56.161 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.161 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.161 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:56.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2606402106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:33:56.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:56.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:56.544 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:33:57.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:57.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: mon.c calling monitor election 2024-04-26T01:33:57.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:33:57.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: mon.a calling monitor election 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: monmap epoch 3 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: min_mon_release 19 (squid) 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: election_strategy: 1 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: fsmap 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: mgrmap e13: x(active, since 69s) 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: overall HEALTH_OK 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:33:57.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2606402106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T01:33:57.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:57.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mon.b calling monitor election 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mon.b calling monitor election 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mon.c calling monitor election 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mon.a calling monitor election 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: monmap epoch 3 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: min_mon_release 19 (squid) 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: election_strategy: 1 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: fsmap 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: mgrmap e13: x(active, since 71s) 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mon.b calling monitor election 2024-04-26T01:33:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mon.b calling monitor election 2024-04-26T01:33:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mon.c calling monitor election 2024-04-26T01:33:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mon.a calling monitor election 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: monmap epoch 3 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: min_mon_release 19 (squid) 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: election_strategy: 1 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: fsmap 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: mgrmap e13: x(active, since 71s) 2024-04-26T01:33:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: overall HEALTH_OK 2024-04-26T01:33:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:57.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mon.b calling monitor election 2024-04-26T01:33:57.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mon.b calling monitor election 2024-04-26T01:33:57.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mon.c calling monitor election 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mon.a calling monitor election 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: monmap epoch 3 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: last_changed 2024-04-26T01:33:49.758682+0000 2024-04-26T01:33:57.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: created 2024-04-26T01:31:23.102434+0000 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: min_mon_release 19 (squid) 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: election_strategy: 1 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: 2: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: fsmap 2024-04-26T01:33:57.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: mgrmap e13: x(active, since 71s) 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:57.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 -- 172.21.15.52:0/2151770470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41084e0 msgr2=0x7fcae41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 --2- 172.21.15.52:0/2151770470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41084e0 0x7fcae41088c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009910 tx=0x7fcad4009c20 comp rx=0 tx=0).stop 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 -- 172.21.15.52:0/2151770470 shutdown_connections 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 --2- 172.21.15.52:0/2151770470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41084e0 0x7fcae41088c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 -- 172.21.15.52:0/2151770470 >> 172.21.15.52:0/2151770470 conn(0x7fcae4076ea0 msgr2=0x7fcae40772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 -- 172.21.15.52:0/2151770470 shutdown_connections 2024-04-26T01:33:58.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.546+0000 7fcae9c00700 1 -- 172.21.15.52:0/2151770470 wait complete. 2024-04-26T01:33:58.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.547+0000 7fcae9c00700 1 Processor -- start 2024-04-26T01:33:58.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 -- start start 2024-04-26T01:33:58.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 0x7fcae4101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:58.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcae41a42c0 0x7fcae41a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcae410c920 con 0x7fcae41019a0 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcae410c7a0 con 0x7fcae41084e0 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae9c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcae410caa0 con 0x7fcae41a42c0 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae2000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae2a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 0x7fcae4101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae2000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56396/0 (socket says 172.21.15.52:56396) 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae2000700 1 -- 172.21.15.52:0/1638309774 learned_addr learned my addr 172.21.15.52:0/1638309774 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:33:58.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.548+0000 7fcae3400700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcae41a42c0 0x7fcae41a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:58.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 -- 172.21.15.52:0/1638309774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcae41a42c0 msgr2=0x7fcae41a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:58.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcae41a42c0 0x7fcae41a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 -- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 msgr2=0x7fcae4101e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:58.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 -- 172.21.15.52:0/1638309774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae41a6de0 con 0x7fcae41084e0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2000700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.549+0000 7fcae2a00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 0x7fcae4101460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcad4005950 tx=0x7fcad4004f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.550+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4026070 con 0x7fcae41084e0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.550+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae41a70d0 con 0x7fcae41084e0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.550+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcad400b6f0 con 0x7fcae41084e0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.550+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4009e60 con 0x7fcae41084e0 2024-04-26T01:33:58.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.550+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae41a7660 con 0x7fcae41084e0 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.551+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fcad4029070 con 0x7fcae41084e0 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.551+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae40640a0 con 0x7fcae41084e0 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.551+0000 7fcadb400700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 0x7fcac803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.551+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcad404d4f0 con 0x7fcae41084e0 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.552+0000 7fcae2000700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 0x7fcac803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:33:58.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.552+0000 7fcae2000700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 0x7fcac803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009f70 tx=0x7fcad00092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:33:58.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.557+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcad4013070 con 0x7fcae41084e0 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:58.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:33:58.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:33:58.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:58.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.801+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fcae41a78c0 con 0x7fcae41084e0 2024-04-26T01:33:58.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.802+0000 7fcadb400700 1 -- 172.21.15.52:0/1638309774 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fcad4030020 con 0x7fcae41084e0 2024-04-26T01:33:58.797 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:58.797 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-04-26T01:33:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:33:58.797 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-04-26T01:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 msgr2=0x7fcac803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 0x7fcac803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009f70 tx=0x7fcad00092c0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 msgr2=0x7fcae4101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 0x7fcae4101460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcad4005950 tx=0x7fcad4004f80 comp rx=0 tx=0).stop 2024-04-26T01:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 shutdown_connections 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcac8039ae0 0x7fcac803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcae41a42c0 0x7fcae41a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcae41084e0 0x7fcae4101460 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 --2- 172.21.15.52:0/1638309774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcae41019a0 0x7fcae4101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 >> 172.21.15.52:0/1638309774 conn(0x7fcae4076ea0 msgr2=0x7fcae40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 shutdown_connections 2024-04-26T01:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:33:58.805+0000 7fcae9c00700 1 -- 172.21.15.52:0/1638309774 wait complete. 2024-04-26T01:33:59.024 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:33:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:33:58.758+0000 7f92c6200700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T01:33:59.467 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T01:33:59.467 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:33:59.467 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:33:59.548 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:33:59.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:59.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:33:59.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:33:59 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1638309774' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:59.654 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:33:59.654 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.654 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:33:59 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1638309774' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.664 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:33:59.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:33:59.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:33:59.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:33:59.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:33:59 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1638309774' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:33:59.694 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:33:59.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:59.759 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:33:59.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T01:33:59.793 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:33:59.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:33:59.860 INFO:tasks.cephadm:Adding mgr.x on smithi052 2024-04-26T01:33:59.861 INFO:tasks.cephadm:Adding mgr.y on smithi183 2024-04-26T01:33:59.861 INFO:tasks.cephadm:Adding mgr.z on smithi191 2024-04-26T01:33:59.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi052=x;smithi183=y;smithi191=z' 2024-04-26T01:34:00.133 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:34:00.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:00 smithi191 ceph-mon[32745]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:34:00.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:00 smithi191 ceph-mon[32745]: Reconfiguring daemon mon.a on smithi052 2024-04-26T01:34:00.659 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:00 smithi052 ceph-mon[27851]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:34:00.659 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:00 smithi052 ceph-mon[27851]: Reconfiguring daemon mon.a on smithi052 2024-04-26T01:34:00.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:00 smithi183 ceph-mon[32752]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T01:34:00.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:00 smithi183 ceph-mon[32752]: Reconfiguring daemon mon.a on smithi052 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:01.418 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:01.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2138253981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac1008b0 msgr2=0x7fc4ac100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2138253981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac1008b0 0x7fc4ac100cd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00b0a0 tx=0x7fc49c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2138253981 shutdown_connections 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2138253981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac1008b0 0x7fc4ac100cd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2138253981 >> 172.21.15.191:0/2138253981 conn(0x7fc4ac0fc1f0 msgr2=0x7fc4ac0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2138253981 shutdown_connections 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.629+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2138253981 wait complete. 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 Processor -- start 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 -- start start 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4ac1008b0 0x7fc4ac197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac198660 0x7fc4ac19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac108460 con 0x7fc4ac198660 2024-04-26T01:34:01.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1082e0 con 0x7fc4ac1008b0 2024-04-26T01:34:01.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4b1dec700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1085e0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4aaffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:01.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4aaffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:39776/0 (socket says 172.21.15.191:39776) 2024-04-26T01:34:01.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.630+0000 7fc4aaffd700 1 -- 172.21.15.191:0/2461745439 learned_addr learned my addr 172.21.15.191:0/2461745439 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:34:01.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 -- 172.21.15.191:0/2461745439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4ac1008b0 msgr2=0x7fc4ac197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4ac1008b0 0x7fc4ac197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 -- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac198660 msgr2=0x7fc4ac19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac198660 0x7fc4ac19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 -- 172.21.15.191:0/2461745439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac1a00f0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4aaffd700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b780 tx=0x7fc4a000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0004230 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4a00043d0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0011bd0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac1a03e0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.631+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac1a0950 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.632+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc4a001a400 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.632+0000 7fc4a8ff9700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 0x7fc49403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.632+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc4a004c440 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.632+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ac0640a0 con 0x7fc4ac1981e0 2024-04-26T01:34:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.634+0000 7fc4ab7fe700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 0x7fc49403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:01.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.634+0000 7fc4ab7fe700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 0x7fc49403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc49c006130 tx=0x7fc49c006060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:01.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.638+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4a0014160 con 0x7fc4ac1981e0 2024-04-26T01:34:01.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:01.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:01.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:01.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:01.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:01.842 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.847+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi183=y;smithi191=z", "target": ["mon-mgr", ""]}) -- 0x7fc4ac062f10 con 0x7fc494039ae0 2024-04-26T01:34:01.857 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.863+0000 7fc4a8ff9700 1 -- 172.21.15.191:0/2461745439 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc4ac062f10 con 0x7fc494039ae0 2024-04-26T01:34:01.858 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 msgr2=0x7fc49403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 0x7fc49403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc49c006130 tx=0x7fc49c006060 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 msgr2=0x7fc4ac19d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000b780 tx=0x7fc4a000bb40 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 shutdown_connections 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc494039ae0 0x7fc49403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4ac198660 0x7fc4ac19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4ac1008b0 0x7fc4ac197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 --2- 172.21.15.191:0/2461745439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4ac1981e0 0x7fc4ac19d240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.865+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 >> 172.21.15.191:0/2461745439 conn(0x7fc4ac0fc1f0 msgr2=0x7fc4ac0fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.866+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 shutdown_connections 2024-04-26T01:34:01.860 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:34:01.866+0000 7fc4b1dec700 1 -- 172.21.15.191:0/2461745439 wait complete. 2024-04-26T01:34:02.442 DEBUG:teuthology.orchestra.run.smithi183:mgr.y> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T01:34:02.445 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T01:34:02.447 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T01:34:02.447 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:34:02.447 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:34:02.509 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:34:02.509 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-04-26T01:34:02.567 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:34:02.567 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:02.567 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T01:34:02.567 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:02.567 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:02.568 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-26 01:33:01.718617328 +0000 2024-04-26T01:34:02.568 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-26 01:27:50.450612313 +0000 2024-04-26T01:34:02.568 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-26 01:27:50.450612313 +0000 2024-04-26T01:34:02.568 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-26 01:27:50.450612313 +0000 2024-04-26T01:34:02.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:34:02.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:02 smithi052 ceph-mon[27851]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:34:02.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:02 smithi052 ceph-mon[27851]: Reconfiguring daemon mon.b on smithi183 2024-04-26T01:34:02.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:02.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:02 smithi191 ceph-mon[32745]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:34:02.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:02 smithi191 ceph-mon[32745]: Reconfiguring daemon mon.b on smithi183 2024-04-26T01:34:02.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:02.657 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-26T01:34:02.657 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-26T01:34:02.657 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000255605 s, 2.0 MB/s 2024-04-26T01:34:02.663 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:34:02.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:02 smithi183 ceph-mon[32752]: Reconfiguring mon.b (monmap changed)... 2024-04-26T01:34:02.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:02 smithi183 ceph-mon[32752]: Reconfiguring daemon mon.b on smithi183 2024-04-26T01:34:02.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:02.719 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-26 01:33:01.772616645 +0000 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-26 01:27:51.045604791 +0000 2024-04-26T01:34:02.770 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-26 01:27:51.045604791 +0000 2024-04-26T01:34:02.771 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-26 01:27:51.045604791 +0000 2024-04-26T01:34:02.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:34:02.850 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-26T01:34:02.850 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-26T01:34:02.850 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000297451 s, 1.7 MB/s 2024-04-26T01:34:02.854 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:34:02.911 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-04-26T01:34:02.962 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:34:02.962 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:02.962 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-26 01:33:01.819616052 +0000 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-26 01:27:51.681596750 +0000 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-26 01:27:51.681596750 +0000 2024-04-26T01:34:02.963 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-26 01:27:51.681596750 +0000 2024-04-26T01:34:02.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:34:03.043 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-26T01:34:03.043 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-26T01:34:03.043 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000264903 s, 1.9 MB/s 2024-04-26T01:34:03.047 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:34:03.107 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-26 01:33:01.866615458 +0000 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-26 01:27:52.300588924 +0000 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-26 01:27:52.300588924 +0000 2024-04-26T01:34:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-26 01:27:52.300588924 +0000 2024-04-26T01:34:03.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:34:03.239 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-04-26T01:34:03.240 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-04-26T01:34:03.240 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000279037 s, 1.8 MB/s 2024-04-26T01:34:03.243 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:34:03.301 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-04-26T01:34:03.301 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:34:03.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:34:03.321 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:03.380 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-26 01:33:49.189960036 +0000 2024-04-26T01:34:03.381 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-26 01:26:36.910972995 +0000 2024-04-26T01:34:03.381 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-26 01:26:36.910972995 +0000 2024-04-26T01:34:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-04-26 01:26:36.910972995 +0000 2024-04-26T01:34:03.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:34:03.446 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-26T01:34:03.446 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-26T01:34:03.446 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000233994 s, 2.2 MB/s 2024-04-26T01:34:03.447 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:34:03.506 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-04-26T01:34:03.510 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi183=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: Saving service mgr spec with placement smithi052=x;smithi183=y;smithi191=z;count:3 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:03.511 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:34:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-26 01:33:49.189960036 +0000 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-26 01:26:37.249964153 +0000 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-26 01:26:37.249964153 +0000 2024-04-26T01:34:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-04-26 01:26:37.249964153 +0000 2024-04-26T01:34:03.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:34:03.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:03.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi183=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:03.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: Saving service mgr spec with placement smithi052=x;smithi183=y;smithi191=z;count:3 2024-04-26T01:34:03.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.626 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:03.626 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:03.626 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:03.627 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-26T01:34:03.627 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-26T01:34:03.627 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000197617 s, 2.6 MB/s 2024-04-26T01:34:03.629 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:34:03.686 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-04-26T01:34:03.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:03.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi183=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:03.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: Saving service mgr spec with placement smithi052=x;smithi183=y;smithi191=z;count:3 2024-04-26T01:34:03.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T01:34:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T01:34:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:03.741 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-26 01:33:49.190960011 +0000 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-26 01:26:37.585955390 +0000 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-26 01:26:37.585955390 +0000 2024-04-26T01:34:03.742 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-04-26 01:26:37.585955390 +0000 2024-04-26T01:34:03.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:34:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-26T01:34:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-26T01:34:03.805 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000214407 s, 2.4 MB/s 2024-04-26T01:34:03.806 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:34:03.863 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-04-26 01:33:49.190960011 +0000 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-04-26 01:26:37.934946287 +0000 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-04-26 01:26:37.934946287 +0000 2024-04-26T01:34:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-04-26 01:26:37.934946287 +0000 2024-04-26T01:34:03.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:34:03.981 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-04-26T01:34:03.981 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-04-26T01:34:03.981 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000211305 s, 2.4 MB/s 2024-04-26T01:34:03.983 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:34:04.038 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-04-26T01:34:04.038 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:34:04.055 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:34:04.056 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-26 01:33:40.094488912 +0000 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-26 01:26:36.938813612 +0000 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-26 01:26:36.938813612 +0000 2024-04-26T01:34:04.111 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-26 01:26:36.938813612 +0000 2024-04-26T01:34:04.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:34:04.175 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-26T01:34:04.175 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-26T01:34:04.175 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000318961 s, 1.6 MB/s 2024-04-26T01:34:04.177 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:34:04.233 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-26 01:33:40.095488896 +0000 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-26 01:26:37.272808048 +0000 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-26 01:26:37.272808048 +0000 2024-04-26T01:34:04.288 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-26 01:26:37.272808048 +0000 2024-04-26T01:34:04.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:34:04.350 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-26T01:34:04.350 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-26T01:34:04.351 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000203992 s, 2.5 MB/s 2024-04-26T01:34:04.352 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:34:04.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:34:04.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: Reconfiguring daemon mon.c on smithi191 2024-04-26T01:34:04.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.409 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-26 01:33:40.095488896 +0000 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-26 01:26:37.614802351 +0000 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-26 01:26:37.614802351 +0000 2024-04-26T01:34:04.465 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-26 01:26:37.614802351 +0000 2024-04-26T01:34:04.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:34:04.528 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-26T01:34:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-26T01:34:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000211083 s, 2.4 MB/s 2024-04-26T01:34:04.529 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:34:04.586 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-04-26 01:33:40.095488896 +0000 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-04-26 01:26:37.949796770 +0000 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-04-26 01:26:37.949796770 +0000 2024-04-26T01:34:04.641 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-04-26 01:26:37.949796770 +0000 2024-04-26T01:34:04.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: Reconfiguring daemon mon.c on smithi191 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:04.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:04.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:04.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.703 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-04-26T01:34:04.703 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-04-26T01:34:04.703 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000201475 s, 2.5 MB/s 2024-04-26T01:34:04.704 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: Reconfiguring mon.c (monmap changed)... 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: Reconfiguring daemon mon.c on smithi191 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:04.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:04.761 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/vg_nvme/lv_4... 2024-04-26T01:34:04.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:34:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:05.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:05 smithi191 ceph-mon[32745]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:05.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:05 smithi191 ceph-mon[32745]: Deploying daemon mgr.z on smithi191 2024-04-26T01:34:05.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:05 smithi183 ceph-mon[32752]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:05.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:05 smithi183 ceph-mon[32752]: Deploying daemon mgr.z on smithi191 2024-04-26T01:34:05.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:05 smithi052 ceph-mon[27851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:05.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:05 smithi052 ceph-mon[27851]: Deploying daemon mgr.z on smithi191 2024-04-26T01:34:06.596 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 systemd[1]: Starting Ceph mgr.z for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:34:06.882 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 podman[34591]: 2024-04-26 01:34:06.504572947 +0000 UTC m=+0.024596884 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:34:06.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 podman[34591]: 2024-04-26 01:34:06.60243085 +0000 UTC m=+0.122454779 container create 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T01:34:06.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 podman[34591]: 2024-04-26 01:34:06.81298377 +0000 UTC m=+0.333007698 container init 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-26T01:34:06.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 podman[34591]: 2024-04-26 01:34:06.815472844 +0000 UTC m=+0.335496780 container start 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-26T01:34:06.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 ceph-mgr[34612]: -- 172.21.15.191:0/1173847903 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e43a9805a0 con 0x55e43a954800 2024-04-26T01:34:06.883 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 bash[34591]: 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 2024-04-26T01:34:07.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 systemd[1]: Started Ceph mgr.z for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:34:07.150 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:06.975+0000 7f9ee4e45200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:07.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:07.556 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:07.157+0000 7f9ee4e45200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:34:07.556 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:07.256+0000 7f9ee4e45200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:34:07.556 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:07.560+0000 7f9ee4e45200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:07.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:07.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:07.899 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:07.644+0000 7f9ee4e45200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0348554 s, 301 MB/s 2024-04-26T01:34:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-26T01:34:08.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:08 smithi052 ceph-mon[27851]: Deploying daemon mgr.y on smithi183 2024-04-26T01:34:08.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:08 smithi191 ceph-mon[32745]: Deploying daemon mgr.y on smithi183 2024-04-26T01:34:08.666 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:08 smithi183 ceph-mon[32752]: Deploying daemon mgr.y on smithi183 2024-04-26T01:34:08.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_4 2024-04-26T01:34:09.144 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:08.867+0000 7f9ee4e45200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:34:09.145 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.037+0000 7f9ee4e45200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:34:09.196 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:34:09.396 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.211+0000 7f9ee4e45200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:34:09.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-mon[32745]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:09.650 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.494+0000 7f9ee4e45200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:34:09.650 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.583+0000 7f9ee4e45200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:34:09.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:09 smithi183 ceph-mon[32752]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:09.672 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:09.596+0000 7f95f367a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:34:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:09 smithi052 ceph-mon[27851]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:10.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.787+0000 7f9ee4e45200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:34:10.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:09.978+0000 7f9ee4e45200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:34:10.172 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:09.783+0000 7f95f367a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:34:10.172 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:09.885+0000 7f95f367a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:34:10.633 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:10.201+0000 7f95f367a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:34:10.633 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:10.287+0000 7f95f367a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:10 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.899 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:10.614+0000 7f9ee4e45200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:10.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:10 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:10.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:10 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:11.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 -- 172.21.15.52:0/1579535406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9188076a70 msgr2=0x7f9188074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 --2- 172.21.15.52:0/1579535406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9188076a70 0x7f9188074ed0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f917800b0a0 tx=0x7f917800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 -- 172.21.15.52:0/1579535406 shutdown_connections 2024-04-26T01:34:11.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 --2- 172.21.15.52:0/1579535406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 0x7f9188111480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 --2- 172.21.15.52:0/1579535406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9188076a70 0x7f9188074ed0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 --2- 172.21.15.52:0/1579535406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f91880754a0 0x7f9188075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.302+0000 7f918e400700 1 -- 172.21.15.52:0/1579535406 >> 172.21.15.52:0/1579535406 conn(0x7f91880fe010 msgr2=0x7f9188100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:11.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.303+0000 7f918e400700 1 -- 172.21.15.52:0/1579535406 shutdown_connections 2024-04-26T01:34:11.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.303+0000 7f918e400700 1 -- 172.21.15.52:0/1579535406 wait complete. 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.304+0000 7f918e400700 1 Processor -- start 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.304+0000 7f918e400700 1 -- start start 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 0x7f918819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 0x7f91881a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9188114b70 con 0x7f918810f090 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f91881149f0 con 0x7f9188076a70 2024-04-26T01:34:11.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f918e400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9188114cf0 con 0x7f91880754a0 2024-04-26T01:34:11.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f917ea00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 0x7f918819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45920/0 (socket says 172.21.15.52:45920) 2024-04-26T01:34:11.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 -- 172.21.15.52:0/1842287523 learned_addr learned my addr 172.21.15.52:0/1842287523 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:34:11.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187e00700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 0x7f91881a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 -- 172.21.15.52:0/1842287523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 msgr2=0x7f918819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 0x7f918819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 -- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 msgr2=0x7f91881a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 0x7f91881a6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.305+0000 7f9187400700 1 -- 172.21.15.52:0/1842287523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91881a6eb0 con 0x7f91880754a0 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f917ea00700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 0x7f918819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f9187400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9178004ce0 tx=0x7f9178004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:11.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9178021070 con 0x7f91880754a0 2024-04-26T01:34:11.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9178007b80 con 0x7f91880754a0 2024-04-26T01:34:11.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91881a7140 con 0x7f91880754a0 2024-04-26T01:34:11.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780052a0 con 0x7f91880754a0 2024-04-26T01:34:11.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.306+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91881a76d0 con 0x7f91880754a0 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.307+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9178024070 con 0x7f91880754a0 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.307+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9188069730 con 0x7f91880754a0 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.308+0000 7f9184c00700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 0x7f916c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.308+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f917800f020 con 0x7f91880754a0 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.308+0000 7f917ea00700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 0x7f916c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:11.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.309+0000 7f917ea00700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 0x7f916c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91880767a0 tx=0x7f9170009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:11.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.313+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f917802b020 con 0x7f91880754a0 2024-04-26T01:34:11.345 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:11.043+0000 7f9ee4e45200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:34:11.345 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:11.140+0000 7f9ee4e45200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:34:11.345 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:11.349+0000 7f9ee4e45200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:34:11.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:11.577+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9188062e80 con 0x7f916c039890 2024-04-26T01:34:11.857 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:11.529+0000 7f95f367a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:34:11.857 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:11.693+0000 7f95f367a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:34:11.857 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:11.862+0000 7f95f367a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:34:12.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:11.869+0000 7f9ee4e45200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:34:12.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:11.958+0000 7f9ee4e45200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:12.141+0000 7f95f367a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:12.231+0000 7f95f367a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:12.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:12.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.523 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:12.194+0000 7f9ee4e45200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:34:12.523 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:12.324+0000 7f9ee4e45200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:34:12.523 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:12.416+0000 7f9ee4e45200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:34:12.523 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:12.527+0000 7f9ee4e45200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:12.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T01:34:12.578 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T01:34:12.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:12.922 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:12.433+0000 7f95f367a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:34:12.922 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:12.619+0000 7f95f367a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:34:13.149 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:12.827+0000 7f9ee4e45200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:34:13.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:13 smithi052 ceph-mon[27851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:13.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:13 smithi052 ceph-mon[27851]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:13.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:13 smithi052 ceph-mon[27851]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:34:13.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:13 smithi052 ceph-mon[27851]: Reconfiguring daemon mgr.x on smithi052 2024-04-26T01:34:13.610 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-mon[32752]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:13.610 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-mon[32752]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:13.610 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-mon[32752]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:34:13.610 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-mon[32752]: Reconfiguring daemon mgr.x on smithi052 2024-04-26T01:34:13.610 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:13.206+0000 7f95f367a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:34:13.619 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-mon[32745]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:13.619 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-mon[32745]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:13.619 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-mon[32745]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T01:34:13.619 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-mon[32745]: Reconfiguring daemon mgr.x on smithi052 2024-04-26T01:34:13.620 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:13.415+0000 7f9ee4e45200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:34:13.620 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:34:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:34:13.624+0000 7f9ee4e45200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:34:13.906 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:13.614+0000 7f95f367a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:34:13.906 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:13.708+0000 7f95f367a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:34:13.906 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:13.907+0000 7f95f367a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:34:14.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:14.376+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f9178005440 con 0x7f91880754a0 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: Standby manager daemon z started 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:14.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:14.424+0000 7f95f367a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:14.513+0000 7f95f367a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: Standby manager daemon z started 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:14.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: Standby manager daemon z started 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/3050816589' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:14.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:14.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:14.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:15.074 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:14.747+0000 7f95f367a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:34:15.074 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:14.875+0000 7f95f367a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:34:15.074 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:14.967+0000 7f95f367a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:34:15.074 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:15.078+0000 7f95f367a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:34:15.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:15 smithi052 ceph-mon[27851]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:15.442 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:15 smithi052 ceph-mon[27851]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T01:34:15.442 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:15 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:34:15.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:15 smithi191 ceph-mon[32745]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:15.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:15 smithi191 ceph-mon[32745]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T01:34:15.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:15 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:34:15.671 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-mon[32752]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:15.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-mon[32752]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T01:34:15.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:34:15.672 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:15.378+0000 7f95f367a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:34:16.371 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:15.965+0000 7f95f367a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:34:16.371 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:34:16.175+0000 7f95f367a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:34:16.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:16.393+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mon.1 v2:172.21.15.191:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f9178011390 con 0x7f91880754a0 2024-04-26T01:34:16.642 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:16 smithi052 ceph-mon[27851]: Standby manager daemon y started 2024-04-26T01:34:16.642 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:16 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:34:16.642 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:16 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:16.642 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:16 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:34:16.643 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:16 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:16.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:16 smithi191 ceph-mon[32745]: Standby manager daemon y started 2024-04-26T01:34:16.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:16 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:34:16.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:16 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:16.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:16 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:34:16.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:16 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:16.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-mon[32752]: Standby manager daemon y started 2024-04-26T01:34:16.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:34:16.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:34:16.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:34:16.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:16 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/3913148134' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:34:17.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:17.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: mgrmap e15: x(active, since 90s), standbys: y, z 2024-04-26T01:34:17.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:34:17.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]: dispatch 2024-04-26T01:34:17.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]': finished 2024-04-26T01:34:17.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:34:17.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: mgrmap e15: x(active, since 90s), standbys: y, z 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]: dispatch 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]': finished 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:34:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: mgrmap e15: x(active, since 90s), standbys: y, z 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]: dispatch 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2291139246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db649844-3866-493f-aa45-bbdcc973f556"}]': finished 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T01:34:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:18.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:18 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3928089953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:18.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:18 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3928089953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:18.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:18 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3928089953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:19.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:19 smithi191 ceph-mon[32745]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:19.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:19 smithi183 ceph-mon[32752]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:19.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:19 smithi052 ceph-mon[27851]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:20.708 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:20 smithi052 ceph-mon[27851]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:20.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:20 smithi191 ceph-mon[32745]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:20.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:20 smithi183 ceph-mon[32752]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:22.894 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:22 smithi052 ceph-mon[27851]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:22.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:22 smithi191 ceph-mon[32745]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:22.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:22 smithi183 ceph-mon[32752]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:24.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:24 smithi191 ceph-mon[32745]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:24.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:24 smithi183 ceph-mon[32752]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:24.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:24 smithi052 ceph-mon[27851]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:26.892 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:26 smithi052 ceph-mon[27851]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:26.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:34:26.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:26.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:26 smithi052 ceph-mon[27851]: Deploying daemon osd.0 on smithi052 2024-04-26T01:34:26.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:26 smithi191 ceph-mon[32745]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:26.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:34:26.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:26.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:26 smithi191 ceph-mon[32745]: Deploying daemon osd.0 on smithi052 2024-04-26T01:34:26.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:26 smithi183 ceph-mon[32752]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:26.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T01:34:26.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:26.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:26 smithi183 ceph-mon[32752]: Deploying daemon osd.0 on smithi052 2024-04-26T01:34:28.702 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:28 smithi052 ceph-mon[27851]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:28.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:28 smithi191 ceph-mon[32745]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:28.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:28 smithi183 ceph-mon[32752]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:30.839 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:30 smithi052 ceph-mon[27851]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:30.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:30 smithi191 ceph-mon[32745]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:30.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:30 smithi183 ceph-mon[32752]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:32.849 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:32 smithi052 ceph-mon[27851]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:32.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:32 smithi191 ceph-mon[32745]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:32.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:32 smithi183 ceph-mon[32752]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:35.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:35 smithi052 ceph-mon[27851]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:35.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:35.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:35.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:35.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:35 smithi191 ceph-mon[32745]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:35.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:35.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:35.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:35 smithi183 ceph-mon[32752]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.313 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:37.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:37.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:37.314 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:37 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:37.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:37.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:37.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:37 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:37 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:38.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.976+0000 7f9184c00700 1 -- 172.21.15.52:0/1842287523 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9188062e80 con 0x7f916c039890 2024-04-26T01:34:38.973 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 msgr2=0x7f916c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 0x7f916c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91880767a0 tx=0x7f9170009260 comp rx=0 tx=0).stop 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 msgr2=0x7f918819f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9178004ce0 tx=0x7f9178004d10 comp rx=0 tx=0).stop 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 shutdown_connections 2024-04-26T01:34:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f916c039890 0x7f916c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f918810f090 0x7f91881a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91880754a0 0x7f918819f250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 --2- 172.21.15.52:0/1842287523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9188076a70 0x7f918819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.980+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 >> 172.21.15.52:0/1842287523 conn(0x7f91880fe010 msgr2=0x7f9188102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.981+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 shutdown_connections 2024-04-26T01:34:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:34:38.981+0000 7f918e400700 1 -- 172.21.15.52:0/1842287523 wait complete. 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:39.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:39.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:39.715 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T01:34:39.721 INFO:tasks.cephadm:Deploying osd.1 on smithi183 with /dev/vg_nvme/lv_4... 2024-04-26T01:34:39.721 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:34:39.964 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:34:40.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-mon[27851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:34:40.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-mon[27851]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:40.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-mon[27851]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:34:40.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:40.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-mon[27851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:40.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0[42537]: 2024-04-26T01:34:40.108+0000 7fd629000700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T01:34:40.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:34:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0[42537]: 2024-04-26T01:34:40.118+0000 7fd623e00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:34:40.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:40 smithi191 ceph-mon[32745]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:34:40.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:40 smithi191 ceph-mon[32745]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:40.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:40 smithi191 ceph-mon[32745]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:34:40.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:40 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:40.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:40 smithi191 ceph-mon[32745]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:40 smithi183 ceph-mon[32752]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T01:34:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:40 smithi183 ceph-mon[32752]: from='osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:40 smithi183 ceph-mon[32752]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T01:34:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:40 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:40 smithi183 ceph-mon[32752]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:34:41.267 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:41 smithi183 ceph-mon[32752]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:41.267 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:41 smithi183 ceph-mon[32752]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:34:41.267 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:41 smithi183 ceph-mon[32752]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:34:41.267 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:41 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.268 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:41 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:41 smithi191 ceph-mon[32745]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:41 smithi191 ceph-mon[32745]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:34:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:41 smithi191 ceph-mon[32745]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:34:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:41 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:41 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:41 smithi052 ceph-mon[27851]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:41 smithi052 ceph-mon[27851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:34:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:41 smithi052 ceph-mon[27851]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T01:34:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:41 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:41 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:41.862 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:34:41.862 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:34:41.862 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-04-26T01:34:41.863 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-04-26T01:34:41.863 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112622 s, 931 MB/s 2024-04-26T01:34:41.863 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-04-26T01:34:42.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:42 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:34:42.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:42 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:34:42.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:42 smithi191 ceph-mon[32745]: osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] boot 2024-04-26T01:34:42.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:42 smithi191 ceph-mon[32745]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:34:42.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:42.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:42 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:34:42.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:42 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:34:42.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:42 smithi183 ceph-mon[32752]: osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] boot 2024-04-26T01:34:42.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:42 smithi183 ceph-mon[32752]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:34:42.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:42.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:42 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:34:42.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:42 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:34:42.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:42 smithi052 ceph-mon[27851]: osd.0 [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] boot 2024-04-26T01:34:42.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:42 smithi052 ceph-mon[27851]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T01:34:42.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:34:42.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-04-26T01:34:42.801 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:34:43.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:43 smithi183 ceph-mon[32752]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:43.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:43 smithi183 ceph-mon[32752]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:34:43.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:43 smithi191 ceph-mon[32745]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:43.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:43 smithi191 ceph-mon[32745]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:34:43.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:43 smithi052 ceph-mon[27851]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T01:34:43.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:43 smithi052 ceph-mon[27851]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T01:34:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 -- 172.21.15.183:0/3459827164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470103540 msgr2=0x7f6470109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 --2- 172.21.15.183:0/3459827164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470103540 0x7f6470109df0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f646800b0d0 tx=0x7f646800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 -- 172.21.15.183:0/3459827164 shutdown_connections 2024-04-26T01:34:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 --2- 172.21.15.183:0/3459827164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470103540 0x7f6470109df0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 --2- 172.21.15.183:0/3459827164 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f64701019a0 0x7f6470101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 --2- 172.21.15.183:0/3459827164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6470102ba0 0x7f6470103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 -- 172.21.15.183:0/3459827164 >> 172.21.15.183:0/3459827164 conn(0x7f64700fcf40 msgr2=0x7f64700ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 -- 172.21.15.183:0/3459827164 shutdown_connections 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.258+0000 7f64765f6700 1 -- 172.21.15.183:0/3459827164 wait complete. 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 Processor -- start 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 -- start start 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64701019a0 0x7f647006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6470103540 0x7f64701a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f647010d520 con 0x7f6470103540 2024-04-26T01:34:44.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f647010d3a0 con 0x7f6470102ba0 2024-04-26T01:34:44.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f64765f6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f647010d6a0 con 0x7f64701019a0 2024-04-26T01:34:44.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f646f7fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:44.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f646f7fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:43758/0 (socket says 172.21.15.183:43758) 2024-04-26T01:34:44.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f646f7fe700 1 -- 172.21.15.183:0/3198802353 learned_addr learned my addr 172.21.15.183:0/3198802353 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.259+0000 7f646ffff700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64701019a0 0x7f647006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 -- 172.21.15.183:0/3198802353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64701019a0 msgr2=0x7f647006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64701019a0 0x7f647006b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 -- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6470103540 msgr2=0x7f64701a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6470103540 0x7f64701a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 -- 172.21.15.183:0/3198802353 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701a0900 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646f7fe700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f646000d980 tx=0x7f646000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460017070 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6460005020 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64701a0bf0 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460009b50 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.260+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701a11b0 con 0x7f6470102ba0 2024-04-26T01:34:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.261+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6470069730 con 0x7f6470102ba0 2024-04-26T01:34:44.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.262+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f64600051d0 con 0x7f6470102ba0 2024-04-26T01:34:44.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.262+0000 7f646d7fa700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 0x7f64580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:34:44.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.263+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f64600ceb80 con 0x7f6470102ba0 2024-04-26T01:34:44.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.263+0000 7f646ffff700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 0x7f64580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:34:44.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.264+0000 7f646ffff700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 0x7f64580a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6470102a00 tx=0x7f6464009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:34:44.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.266+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6460020020 con 0x7f6470102ba0 2024-04-26T01:34:44.469 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:34:44.474+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6470062e80 con 0x7f64580a3c30 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi052 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 17094M 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:45.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:45.404 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:45.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi052 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 17094M 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi052 2024-04-26T01:34:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 17094M 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:34:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:46.610 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:46 smithi052 ceph-mon[27851]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:46.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:46 smithi191 ceph-mon[32745]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:46.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:46 smithi183 ceph-mon[32752]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:34:47.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:47.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/3834880252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]': finished 2024-04-26T01:34:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:34:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/3834880252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]': finished 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:34:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:34:47.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:47.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/3834880252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]: dispatch 2024-04-26T01:34:47.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285c6829-a343-4ccb-9883-8a1e24fb28ba"}]': finished 2024-04-26T01:34:47.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T01:34:47.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:34:48.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:48 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/3608824519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:48.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:48 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/3608824519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:48.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:48 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/3608824519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:34:49.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:49 smithi191 ceph-mon[32745]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:49.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:49 smithi183 ceph-mon[32752]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:49.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:49 smithi052 ceph-mon[27851]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:51.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:51 smithi191 ceph-mon[32745]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:51.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:51 smithi183 ceph-mon[32752]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:51.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:51 smithi052 ceph-mon[27851]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:53.483 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:53 smithi183 ceph-mon[32752]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:53.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:53 smithi191 ceph-mon[32745]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:53.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:53 smithi052 ceph-mon[27851]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:55.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:55 smithi183 ceph-mon[32752]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:55.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:34:55.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:55.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:55 smithi191 ceph-mon[32745]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:55.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:34:55.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:55 smithi052 ceph-mon[27851]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T01:34:55.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:34:56.630 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:56 smithi183 ceph-mon[32752]: Deploying daemon osd.1 on smithi183 2024-04-26T01:34:56.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:56 smithi191 ceph-mon[32745]: Deploying daemon osd.1 on smithi183 2024-04-26T01:34:56.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:56 smithi052 ceph-mon[27851]: Deploying daemon osd.1 on smithi183 2024-04-26T01:34:57.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:57 smithi183 ceph-mon[32752]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:57.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:57 smithi191 ceph-mon[32745]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:57.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:57 smithi052 ceph-mon[27851]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:59.559 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:34:59 smithi183 ceph-mon[32752]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:59.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:34:59 smithi191 ceph-mon[32745]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:34:59.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:34:59 smithi052 ceph-mon[27851]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:00 smithi191 ceph-mon[32745]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:00 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:00 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:00 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:01.062 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:00 smithi183 ceph-mon[32752]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:01.062 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:00 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:01.062 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:00 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:01.062 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:00 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:00 smithi052 ceph-mon[27851]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:00 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:00 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:01.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:00 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:02.963 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:02.963 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:02.964 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:02.964 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:02.964 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:02.964 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.311+0000 7f646d7fa700 1 -- 172.21.15.183:0/3198802353 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6470062e80 con 0x7f64580a3c30 2024-04-26T01:35:03.308 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 1 on host 'smithi183' 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 msgr2=0x7f64580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 0x7f64580a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6470102a00 tx=0x7f6464009190 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 msgr2=0x7f647019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f646000d980 tx=0x7f646000dc90 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 shutdown_connections 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f64580a3c30 0x7f64580a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6470103540 0x7f64701a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64701019a0 0x7f647006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 --2- 172.21.15.183:0/3198802353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6470102ba0 0x7f647019d9e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 >> 172.21.15.183:0/3198802353 conn(0x7f64700fcf40 msgr2=0x7f6470105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:03.310 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 shutdown_connections 2024-04-26T01:35:03.311 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:35:03.314+0000 7f64765f6700 1 -- 172.21.15.183:0/3198802353 wait complete. 2024-04-26T01:35:03.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:03.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:03.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:03 smithi183 ceph-mon[32752]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:03.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:03 smithi183 ceph-mon[32752]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:03.967 DEBUG:teuthology.orchestra.run.smithi183:osd.1> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T01:35:03.970 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/vg_nvme/lv_4... 2024-04-26T01:35:03.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T01:35:04.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:03 smithi191 ceph-mon[32745]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:03 smithi191 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:03 smithi052 ceph-mon[27851]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:03 smithi052 ceph-mon[27851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T01:35:04.230 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:05.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:05 smithi183 ceph-mon[32752]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:05.522 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:05 smithi191 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: from='osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:05 smithi052 ceph-mon[27851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114507 s, 916 MB/s 2024-04-26T01:35:06.097 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: from='osd.1 ' entity='osd.1' 2024-04-26T01:35:06.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.613 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:35:06.613 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:35:06.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: from='osd.1 ' entity='osd.1' 2024-04-26T01:35:06.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.614 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1[37336]: 2024-04-26T01:35:06.245+0000 7f5e32638700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T01:35:06.614 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1[37336]: 2024-04-26T01:35:06.339+0000 7f5e2a423700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:35:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:35:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T01:35:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: from='osd.1 ' entity='osd.1' 2024-04-26T01:35:06.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:06.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-04-26T01:35:06.984 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:07.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:07.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:35:07.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:35:07.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:08.456 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:08 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi183 2024-04-26T01:35:08.456 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:08 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 17088M 2024-04-26T01:35:08.456 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:08 smithi183 ceph-mon[32752]: osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] boot 2024-04-26T01:35:08.456 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:08 smithi183 ceph-mon[32752]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:35:08.456 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 -- 172.21.15.191:0/2280840386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 msgr2=0x7fa46c10aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 --2- 172.21.15.191:0/2280840386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c10aef0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa46400b0a0 tx=0x7fa46400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 -- 172.21.15.191:0/2280840386 shutdown_connections 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 --2- 172.21.15.191:0/2280840386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c10aef0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 --2- 172.21.15.191:0/2280840386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa46c102aa0 0x7fa46c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 --2- 172.21.15.191:0/2280840386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c103ca0 0x7fa46c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 -- 172.21.15.191:0/2280840386 >> 172.21.15.191:0/2280840386 conn(0x7fa46c0fe040 msgr2=0x7fa46c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 -- 172.21.15.191:0/2280840386 shutdown_connections 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.561+0000 7fa472744700 1 -- 172.21.15.191:0/2280840386 wait complete. 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 Processor -- start 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 -- start start 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c102aa0 0x7fa46c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa46c103ca0 0x7fa46c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:08.559 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10e600 con 0x7fa46c103ca0 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10e480 con 0x7fa46c102aa0 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.562+0000 7fa472744700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10e780 con 0x7fa46c104640 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:60324/0 (socket says 172.21.15.191:60324) 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 -- 172.21.15.191:0/2588452291 learned_addr learned my addr 172.21.15.191:0/2588452291 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa46bfff700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c102aa0 0x7fa46c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 -- 172.21.15.191:0/2588452291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c102aa0 msgr2=0x7fa46c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c102aa0 0x7fa46c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 -- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa46c103ca0 msgr2=0x7fa46c1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa46c103ca0 0x7fa46c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 -- 172.21.15.191:0/2588452291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1a08b0 con 0x7fa46c104640 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa470ce1700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4640062a0 tx=0x7fa46400ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:08.560 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464021020 con 0x7fa46c104640 2024-04-26T01:35:08.561 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c1a0b40 con 0x7fa46c104640 2024-04-26T01:35:08.561 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46401d920 con 0x7fa46c104640 2024-04-26T01:35:08.561 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464005540 con 0x7fa46c104640 2024-04-26T01:35:08.561 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.563+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1a1070 con 0x7fa46c104640 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.564+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46c069730 con 0x7fa46c104640 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.565+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa464024020 con 0x7fa46c104640 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.566+0000 7fa4697fa700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 0x7fa4540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.566+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fa46400f020 con 0x7fa46c104640 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.567+0000 7fa46bfff700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 0x7fa4540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.568+0000 7fa46bfff700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 0x7fa4540a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa46c19a140 tx=0x7fa45c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:08.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.571+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa46409e4d0 con 0x7fa46c104640 2024-04-26T01:35:08.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:08 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi183 2024-04-26T01:35:08.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:08 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 17088M 2024-04-26T01:35:08.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:08 smithi191 ceph-mon[32745]: osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] boot 2024-04-26T01:35:08.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:08 smithi191 ceph-mon[32745]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:35:08.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:08.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:08 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi183 2024-04-26T01:35:08.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:08 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 17088M 2024-04-26T01:35:08.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:08 smithi052 ceph-mon[27851]: osd.1 [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] boot 2024-04-26T01:35:08.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:08 smithi052 ceph-mon[27851]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T01:35:08.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:35:08.774 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:08.779+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa46c062e80 con 0x7fa4540a3bb0 2024-04-26T01:35:09.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:09 smithi191 ceph-mon[32745]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:09.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:09 smithi191 ceph-mon[32745]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:35:09.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:09 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:09.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:09 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:09.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:09 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:09.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:09 smithi183 ceph-mon[32752]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:09.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:09 smithi183 ceph-mon[32752]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:35:09.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:09 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:09.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:09 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:09.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:09 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:09 smithi052 ceph-mon[27851]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T01:35:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:09 smithi052 ceph-mon[27851]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T01:35:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:09 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:09 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:09.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:09 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:10.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:10 smithi191 ceph-mon[32745]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:10.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:10 smithi183 ceph-mon[32752]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:10.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:10 smithi052 ceph-mon[27851]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:11.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:11 smithi191 ceph-mon[32745]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:11.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:11 smithi183 ceph-mon[32752]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:11.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:11 smithi052 ceph-mon[27851]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2736451621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]': finished 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:12.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:12 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2561118248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:12.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2736451621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]': finished 2024-04-26T01:35:12.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:35:12.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:12.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:12 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2561118248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:12.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2736451621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]: dispatch 2024-04-26T01:35:12.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc51fafd-4a20-45b8-9b1f-d22980ab4b86"}]': finished 2024-04-26T01:35:12.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T01:35:12.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:12.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:12 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2561118248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:13.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:13 smithi191 ceph-mon[32745]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:13.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:13 smithi183 ceph-mon[32752]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:13.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:13 smithi052 ceph-mon[27851]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:15.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:15 smithi191 ceph-mon[32745]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:15.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:15 smithi183 ceph-mon[32752]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:15.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:15 smithi052 ceph-mon[27851]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:17.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:17 smithi191 ceph-mon[32745]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:17.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:17 smithi183 ceph-mon[32752]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:17.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:17 smithi052 ceph-mon[27851]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:19.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:19 smithi052 ceph-mon[27851]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:19.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:35:19.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:19.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:19 smithi191 ceph-mon[32745]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:19.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:35:19.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:19.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:19 smithi183 ceph-mon[32752]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:19.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T01:35:19.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:20.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:20 smithi191 ceph-mon[32745]: Deploying daemon osd.2 on smithi191 2024-04-26T01:35:20.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:20 smithi183 ceph-mon[32752]: Deploying daemon osd.2 on smithi191 2024-04-26T01:35:20.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:20 smithi052 ceph-mon[27851]: Deploying daemon osd.2 on smithi191 2024-04-26T01:35:21.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:21 smithi183 ceph-mon[32752]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:21.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:21 smithi052 ceph-mon[27851]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:21.759 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:21 smithi191 ceph-mon[32745]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:23.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:23 smithi191 ceph-mon[32745]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:23.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:23 smithi183 ceph-mon[32752]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:23.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:23 smithi052 ceph-mon[27851]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:25.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:25 smithi183 ceph-mon[32752]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:25.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:25.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:25 smithi052 ceph-mon[27851]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:25.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:25.899 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:25 smithi191 ceph-mon[32745]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:25.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:25.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:27.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:27.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:27.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:27.777 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:28.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.115+0000 7fa4697fa700 1 -- 172.21.15.191:0/2588452291 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa46c062e80 con 0x7fa4540a3bb0 2024-04-26T01:35:28.111 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2024-04-26T01:35:28.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 msgr2=0x7fa4540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:28.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 0x7fa4540a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa46c19a140 tx=0x7fa45c009190 comp rx=0 tx=0).stop 2024-04-26T01:35:28.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 msgr2=0x7fa46c1a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:28.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa4640062a0 tx=0x7fa46400ced0 comp rx=0 tx=0).stop 2024-04-26T01:35:28.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 shutdown_connections 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4540a3bb0 0x7fa4540a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa46c104640 0x7fa46c1a01e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa46c102aa0 0x7fa46c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 --2- 172.21.15.191:0/2588452291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa46c103ca0 0x7fa46c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 >> 172.21.15.191:0/2588452291 conn(0x7fa46c0fe040 msgr2=0x7fa46c1070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 shutdown_connections 2024-04-26T01:35:28.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:35:28.117+0000 7fa472744700 1 -- 172.21.15.191:0/2588452291 wait complete. 2024-04-26T01:35:28.782 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T01:35:28.784 INFO:tasks.cephadm:Deploying osd.3 on smithi052 with /dev/vg_nvme/lv_3... 2024-04-26T01:35:28.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:35:29.270 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:29.325 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:29 smithi052 ceph-mon[27851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:29 smithi191 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:29.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:29.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:29.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:29.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:29.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:29 smithi183 ceph-mon[32752]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T01:35:30.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:30 smithi191 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:35:30.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:30 smithi191 ceph-mon[32745]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:30.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:30 smithi191 ceph-mon[32745]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:35:30.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:30.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:30 smithi191 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:30.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:30 smithi052 ceph-mon[27851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:35:30.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:30 smithi052 ceph-mon[27851]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:30.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:30 smithi052 ceph-mon[27851]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:35:30.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:30.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:30 smithi052 ceph-mon[27851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:30.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:30 smithi183 ceph-mon[32752]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T01:35:30.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:30 smithi183 ceph-mon[32752]: from='osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:30.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:30 smithi183 ceph-mon[32752]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T01:35:30.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:30.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:30 smithi183 ceph-mon[32752]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: from='osd.2 ' entity='osd.2' 2024-04-26T01:35:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.401 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2[37876]: 2024-04-26T01:35:31.013+0000 7f11fef03700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T01:35:31.401 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2[37876]: 2024-04-26T01:35:31.129+0000 7f11f6cee700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: from='osd.2 ' entity='osd.2' 2024-04-26T01:35:31.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:31 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:35:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T01:35:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: from='osd.2 ' entity='osd.2' 2024-04-26T01:35:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:31 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:31.963 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:35:31.964 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:35:31.964 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-26T01:35:31.964 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-26T01:35:31.964 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0586531 s, 179 MB/s 2024-04-26T01:35:31.964 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-26T01:35:32.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:32.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:35:32.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:35:32.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:35:32.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:32.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:35:32.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:32.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:32.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_3 2024-04-26T01:35:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:35:33.369 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:33.370 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi191 2024-04-26T01:35:33.370 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-26T01:35:33.370 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] boot 2024-04-26T01:35:33.370 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:35:33.370 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi191 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] boot 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:35:33.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:33.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T01:35:33.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi191 2024-04-26T01:35:33.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 17088M 2024-04-26T01:35:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: osd.2 [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] boot 2024-04-26T01:35:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T01:35:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:35:34.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:34 smithi191 ceph-mon[32745]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:35:34.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:34 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:34.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:34 smithi183 ceph-mon[32752]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:35:34.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:34 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:34.457 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:34 smithi052 ceph-mon[27851]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T01:35:34.457 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:34 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:35.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 -- 172.21.15.52:0/833107015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 msgr2=0x7f803810d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 --2- 172.21.15.52:0/833107015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f803810d140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8030009fc0 tx=0x7f803000b330 comp rx=0 tx=0).stop 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 -- 172.21.15.52:0/833107015 shutdown_connections 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 --2- 172.21.15.52:0/833107015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f803810d140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 --2- 172.21.15.52:0/833107015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038104cf0 0x7f8038105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 --2- 172.21.15.52:0/833107015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038105ef0 0x7f8038106350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 -- 172.21.15.52:0/833107015 >> 172.21.15.52:0/833107015 conn(0x7f8038100290 msgr2=0x7f80381026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 -- 172.21.15.52:0/833107015 shutdown_connections 2024-04-26T01:35:35.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.303+0000 7f803e600700 1 -- 172.21.15.52:0/833107015 wait complete. 2024-04-26T01:35:35.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.304+0000 7f803e600700 1 Processor -- start 2024-04-26T01:35:35.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 -- start start 2024-04-26T01:35:35.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:35.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038105ef0 0x7f803819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f80381a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8038110790 con 0x7f8038105ef0 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8038110610 con 0x7f8038106890 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f803e600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8038110910 con 0x7f8038104cf0 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f8037400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f8037400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56476/0 (socket says 172.21.15.52:56476) 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f8037400700 1 -- 172.21.15.52:0/2988239187 learned_addr learned my addr 172.21.15.52:0/2988239187 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.305+0000 7f802fe00700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038105ef0 0x7f803819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037e00700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f80381a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 -- 172.21.15.52:0/2988239187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 msgr2=0x7f80381a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f80381a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 -- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038105ef0 msgr2=0x7f803819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:35:35.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038105ef0 0x7f803819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:35:35.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 -- 172.21.15.52:0/2988239187 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80381a29f0 con 0x7f8038104cf0 2024-04-26T01:35:35.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8037400700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f802800e970 tx=0x7f802800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:35.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802800cbd0 con 0x7f8038104cf0 2024-04-26T01:35:35.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80280044d0 con 0x7f8038104cf0 2024-04-26T01:35:35.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80381a2ce0 con 0x7f8038104cf0 2024-04-26T01:35:35.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028003de0 con 0x7f8038104cf0 2024-04-26T01:35:35.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.306+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80381a31c0 con 0x7f8038104cf0 2024-04-26T01:35:35.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.308+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f803802df80 con 0x7f8038104cf0 2024-04-26T01:35:35.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.309+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f8028003f80 con 0x7f8038104cf0 2024-04-26T01:35:35.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.311+0000 7f8034c00700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 0x7f801c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:35:35.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.311+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f80280cf1a0 con 0x7f8038104cf0 2024-04-26T01:35:35.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.311+0000 7f802fe00700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 0x7f801c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:35:35.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.312+0000 7f802fe00700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 0x7f801c0a60f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8020005fd0 tx=0x7f8020009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:35:35.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.315+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f802809cf90 con 0x7f8038104cf0 2024-04-26T01:35:35.358 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:35.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.359 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46764]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 ceph-mon[27851]: pgmap v93: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:35.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:35.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 ceph-mon[27851]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:35:35.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:35.399 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 ceph-mon[32745]: pgmap v93: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:35.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:35.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 ceph-mon[32745]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:35:35.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 ceph-mon[32752]: pgmap v93: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 ceph-mon[32752]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T01:35:35.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T01:35:35.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:35.573+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8038062e80 con 0x7f801c0a3c30 2024-04-26T01:35:35.626 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46767]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:35.627 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46767]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.627 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46767]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.627 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46767]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.732 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41021]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:35.732 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.732 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41021]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.733 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41021]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.733 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41024]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:35.733 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.733 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41024]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.733 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41024]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41961]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41961]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41961]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41961]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41964]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41964]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41964]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.900 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41955]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41955]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41955]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41955]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41958]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41958]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41958]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.901 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 01:35:35 smithi191 sudo[41958]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46779]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:35.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46779]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46779]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46803]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:35 smithi052 sudo[46803]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:36.156 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41027]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T01:35:36.156 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41027]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41027]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41027]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41030]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41030]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41030]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:35 smithi183 sudo[41030]: pam_unix(sudo:session): session closed for user root 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:36.157 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:36.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:36.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:36 smithi183 ceph-mon[32752]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:36.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:35:36.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:36 smithi052 ceph-mon[27851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:35:36.619+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8028098f00 con 0x7f8038104cf0 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:35:36.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T01:35:36.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:35:36.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:35:36.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:35:36.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:36 smithi191 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T01:35:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:37 smithi183 ceph-mon[32752]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:37 smithi183 ceph-mon[32752]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:37 smithi183 ceph-mon[32752]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:35:37.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:37 smithi183 ceph-mon[32752]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:35:37.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:37 smithi052 ceph-mon[27851]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:37.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:37 smithi052 ceph-mon[27851]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:37.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:37 smithi052 ceph-mon[27851]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:35:37.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:37 smithi052 ceph-mon[27851]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:35:37.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:37 smithi191 ceph-mon[32745]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:37 smithi191 ceph-mon[32745]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:35:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:37 smithi191 ceph-mon[32745]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T01:35:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:37 smithi191 ceph-mon[32745]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-26T01:35:39.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:39 smithi183 ceph-mon[32752]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:39.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:39 smithi052 ceph-mon[27851]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:39.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:39 smithi191 ceph-mon[32745]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:40 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4140828953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:40 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:40 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]': finished 2024-04-26T01:35:40.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:40 smithi183 ceph-mon[32752]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:35:40.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:40 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:35:40.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:40 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4140828953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:40 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:40 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]': finished 2024-04-26T01:35:40.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:40 smithi052 ceph-mon[27851]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:35:40.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:40 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:35:40.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:40 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4140828953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:40 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]: dispatch 2024-04-26T01:35:40.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:40 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d"}]': finished 2024-04-26T01:35:40.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:40 smithi191 ceph-mon[32745]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T01:35:40.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:40 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:35:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:41 smithi191 ceph-mon[32745]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:41.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:41 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/169878053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:41.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:41 smithi183 ceph-mon[32752]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:41.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:41 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/169878053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:41 smithi052 ceph-mon[27851]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:41.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:41 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/169878053' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:35:43.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:43 smithi052 ceph-mon[27851]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:43.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:43 smithi191 ceph-mon[32745]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:43.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:43 smithi183 ceph-mon[32752]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:45.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:45 smithi052 ceph-mon[27851]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:45.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:45 smithi191 ceph-mon[32745]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:45.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:45 smithi183 ceph-mon[32752]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:47.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:47 smithi052 ceph-mon[27851]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:47 smithi191 ceph-mon[32745]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:47 smithi183 ceph-mon[32752]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:49.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:49 smithi052 ceph-mon[27851]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:49.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:35:49.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:49.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:49 smithi191 ceph-mon[32745]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:49.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:35:49.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:49.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:49 smithi183 ceph-mon[32752]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:49.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T01:35:49.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:35:50.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:50 smithi052 ceph-mon[27851]: Deploying daemon osd.3 on smithi052 2024-04-26T01:35:50.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:50 smithi191 ceph-mon[32745]: Deploying daemon osd.3 on smithi052 2024-04-26T01:35:50.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:50 smithi183 ceph-mon[32752]: Deploying daemon osd.3 on smithi052 2024-04-26T01:35:51.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:51 smithi052 ceph-mon[27851]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:51.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:51 smithi191 ceph-mon[32745]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:51.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:51 smithi183 ceph-mon[32752]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:53.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:53 smithi052 ceph-mon[27851]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:53.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:53 smithi191 ceph-mon[32745]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:53 smithi183 ceph-mon[32752]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:55.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:55 smithi052 ceph-mon[27851]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:55.649 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:55 smithi191 ceph-mon[32745]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:55.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:55 smithi183 ceph-mon[32752]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:57.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:57 smithi052 ceph-mon[27851]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:57.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:57.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:57.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:57 smithi191 ceph-mon[32745]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:57 smithi183 ceph-mon[32752]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:35:57.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:35:59.345 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:35:59 smithi052 ceph-mon[27851]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:59.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:35:59 smithi191 ceph-mon[32745]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:35:59.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:35:59 smithi183 ceph-mon[32752]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:01 smithi052 ceph-mon[27851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:01.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.370+0000 7f8034c00700 1 -- 172.21.15.52:0/2988239187 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8038062e80 con 0x7f801c0a3c30 2024-04-26T01:36:01.367 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 3 on host 'smithi052' 2024-04-26T01:36:01.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 msgr2=0x7f801c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:01.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 0x7f801c0a60f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8020005fd0 tx=0x7f8020009310 comp rx=0 tx=0).stop 2024-04-26T01:36:01.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 msgr2=0x7f803819ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:01.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f802800e970 tx=0x7f802800ed30 comp rx=0 tx=0).stop 2024-04-26T01:36:01.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 shutdown_connections 2024-04-26T01:36:01.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f801c0a3c30 0x7f801c0a60f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:01.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8038106890 0x7f80381a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:01.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8038104cf0 0x7f803819ada0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:01.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 --2- 172.21.15.52:0/2988239187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8038105ef0 0x7f803819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:01.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 >> 172.21.15.52:0/2988239187 conn(0x7f8038100290 msgr2=0x7f80381091f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:01.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 shutdown_connections 2024-04-26T01:36:01.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:01.375+0000 7f803e600700 1 -- 172.21.15.52:0/2988239187 wait complete. 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:01.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:01 smithi183 ceph-mon[32752]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:01.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:01 smithi191 ceph-mon[32745]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T01:36:02.152 DEBUG:teuthology.orchestra.run.smithi052:osd.3> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T01:36:02.157 INFO:tasks.cephadm:Deploying osd.4 on smithi183 with /dev/vg_nvme/lv_3... 2024-04-26T01:36:02.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:36:02.407 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:02.529 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:36:02.529 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.530 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:36:02.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:02.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:03.279 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:03.279 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:36:03.280 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:36:03.280 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.280 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.280 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: osdmap e26: 4 total, 3 up, 4 in 2024-04-26T01:36:03.280 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.535 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: osdmap e26: 4 total, 3 up, 4 in 2024-04-26T01:36:03.536 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: osdmap e26: 4 total, 3 up, 4 in 2024-04-26T01:36:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:03.658 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T01:36:03.282+0000 7f9740200700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T01:36:03.659 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 01:36:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T01:36:03.300+0000 7f9736000700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:36:04.230 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:36:04.230 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:36:04.230 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-04-26T01:36:04.230 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-04-26T01:36:04.230 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112489 s, 932 MB/s 2024-04-26T01:36:04.231 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: from='osd.3 ' entity='osd.3' 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] boot 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:36:04.531 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: from='osd.3 ' entity='osd.3' 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] boot 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:36:04.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:36:04.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:36:04.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: from='osd.3 ' entity='osd.3' 2024-04-26T01:36:04.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: osd.3 [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] boot 2024-04-26T01:36:04.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T01:36:04.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:36:04.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-04-26T01:36:05.153 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:36:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:05 smithi052 ceph-mon[27851]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:05.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:05 smithi052 ceph-mon[27851]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:36:05.589 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:05 smithi183 ceph-mon[32752]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:05.589 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:05 smithi183 ceph-mon[32752]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:36:05.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:05 smithi191 ceph-mon[32745]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-26T01:36:05.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:05 smithi191 ceph-mon[32745]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.645+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/1166324583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 msgr2=0x7fb6bc104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.645+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/1166324583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc104100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80099c0 tx=0x7fb6a8009cd0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/1166324583 shutdown_connections 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/1166324583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 0x7fb6bc10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/1166324583 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 0x7fb6bc102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/1166324583 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc104100 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/1166324583 >> 172.21.15.183:0/1166324583 conn(0x7fb6bc0fe040 msgr2=0x7fb6bc100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/1166324583 shutdown_connections 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/1166324583 wait complete. 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 Processor -- start 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- start start 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 0x7fb6bc198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 0x7fb6bc1a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc10e5b0 con 0x7fb6bc104640 2024-04-26T01:36:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc10e430 con 0x7fb6bc103ca0 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.646+0000 7fb6c2ea9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc10e730 con 0x7fb6bc102aa0 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:33136/0 (socket says 172.21.15.183:33136) 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 -- 172.21.15.183:0/21569827 learned_addr learned my addr 172.21.15.183:0/21569827 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6c0c45700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 0x7fb6bc198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 -- 172.21.15.183:0/21569827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 msgr2=0x7fb6bc198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 0x7fb6bc198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.646 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 -- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 msgr2=0x7fb6bc1a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6c1446700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 0x7fb6bc1a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 0x7fb6bc1a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 -- 172.21.15.183:0/21569827 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc1a0900 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6bbfff700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8003790 tx=0x7fb6a8005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8027070 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6a8004160 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc1a0b90 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8022d90 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.647+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc1a10c0 con 0x7fb6bc103ca0 2024-04-26T01:36:06.647 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.648+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6bc069730 con 0x7fb6bc103ca0 2024-04-26T01:36:06.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.650+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb6a802a070 con 0x7fb6bc103ca0 2024-04-26T01:36:06.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.650+0000 7fb6b9ffb700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 0x7fb6ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:06.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.650+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3698+0+0 (secure 0 0 0) 0x7fb6a80d01a0 con 0x7fb6bc103ca0 2024-04-26T01:36:06.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.651+0000 7fb6c0c45700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 0x7fb6ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:06.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.651+0000 7fb6c0c45700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 0x7fb6ac0a6180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc199b60 tx=0x7fb6b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:06.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.654+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6a8013070 con 0x7fb6bc103ca0 2024-04-26T01:36:06.858 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:06.862+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb6bc062e80 con 0x7fb6ac0a3cc0 2024-04-26T01:36:06.858 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:06 smithi183 ceph-mon[32752]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:06.858 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:06 smithi183 ceph-mon[32752]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:36:07.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:06 smithi191 ceph-mon[32745]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:07.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:06 smithi191 ceph-mon[32745]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:36:07.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:06 smithi052 ceph-mon[27851]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:07.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:06 smithi052 ceph-mon[27851]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T01:36:07.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:07 smithi052 ceph-mon[27851]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:36:07.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:07 smithi052 ceph-mon[27851]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:07.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:07.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:07.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:08.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:07 smithi191 ceph-mon[32745]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:36:08.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:07 smithi191 ceph-mon[32745]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:08.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:08.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:08.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:08.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:07 smithi183 ceph-mon[32752]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T01:36:08.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:07 smithi183 ceph-mon[32752]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: osdmap e31: 4 total, 4 up, 4 in 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi052 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 8547M 2024-04-26T01:36:09.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:09.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:09.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: osdmap e31: 4 total, 4 up, 4 in 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi052 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 8547M 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: osdmap e31: 4 total, 4 up, 4 in 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi052 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 8547M 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:10.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:09 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/428745221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:09 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:09 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]': finished 2024-04-26T01:36:10.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:09 smithi191 ceph-mon[32745]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:36:10.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:09 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:10.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:09 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/428745221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:09 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:09 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]': finished 2024-04-26T01:36:10.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:09 smithi183 ceph-mon[32752]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:36:10.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:09 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:09 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/428745221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:09 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]: dispatch 2024-04-26T01:36:10.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:09 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "014075df-27cd-4f11-8542-ecc8ac24a820"}]': finished 2024-04-26T01:36:10.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:09 smithi052 ceph-mon[27851]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T01:36:10.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:09 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:11.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:10 smithi191 ceph-mon[32745]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T01:36:11.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:10 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/883613677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:11.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:10 smithi183 ceph-mon[32752]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T01:36:11.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:10 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/883613677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:10 smithi052 ceph-mon[27851]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T01:36:11.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:10 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/883613677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:12 smithi191 ceph-mon[32745]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T01:36:13.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:12 smithi183 ceph-mon[32752]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T01:36:13.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:12 smithi052 ceph-mon[27851]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T01:36:15.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:14 smithi183 ceph-mon[32752]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:36:15.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:14 smithi191 ceph-mon[32745]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:36:15.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:14 smithi052 ceph-mon[27851]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T01:36:17.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:16 smithi183 ceph-mon[32752]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:36:17.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:16 smithi191 ceph-mon[32745]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:36:17.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:16 smithi052 ceph-mon[27851]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:36:18.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:36:18.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:18.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:17 smithi183 ceph-mon[32752]: Deploying daemon osd.4 on smithi183 2024-04-26T01:36:18.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:36:18.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:18.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:17 smithi191 ceph-mon[32745]: Deploying daemon osd.4 on smithi183 2024-04-26T01:36:18.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T01:36:18.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:18.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:17 smithi052 ceph-mon[27851]: Deploying daemon osd.4 on smithi183 2024-04-26T01:36:18.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:18 smithi183 ceph-mon[32752]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:36:19.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:18 smithi191 ceph-mon[32745]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:36:19.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:18 smithi052 ceph-mon[27851]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T01:36:20.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:20 smithi183 ceph-mon[32752]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:21.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:20 smithi191 ceph-mon[32745]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:21.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:20 smithi052 ceph-mon[27851]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:23.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:22 smithi191 ceph-mon[32745]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:23.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:23.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:22 smithi183 ceph-mon[32752]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:23.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:23.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:22 smithi052 ceph-mon[27851]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:23.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:23.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:24.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:24 smithi183 ceph-mon[32752]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:25.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:24 smithi191 ceph-mon[32745]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:25.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:24 smithi052 ceph-mon[27851]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:25.935 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.939+0000 7fb6b9ffb700 1 -- 172.21.15.183:0/21569827 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb6bc062e80 con 0x7fb6ac0a3cc0 2024-04-26T01:36:25.935 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 4 on host 'smithi183' 2024-04-26T01:36:25.937 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 msgr2=0x7fb6ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:25.937 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 0x7fb6ac0a6180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc199b60 tx=0x7fb6b000b410 comp rx=0 tx=0).stop 2024-04-26T01:36:25.937 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 msgr2=0x7fb6bc1991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:25.937 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8003790 tx=0x7fb6a8005b00 comp rx=0 tx=0).stop 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 shutdown_connections 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb6ac0a3cc0 0x7fb6ac0a6180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6bc104640 0x7fb6bc1a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6bc102aa0 0x7fb6bc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 --2- 172.21.15.183:0/21569827 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6bc103ca0 0x7fb6bc1991c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 >> 172.21.15.183:0/21569827 conn(0x7fb6bc0fe040 msgr2=0x7fb6bc107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 shutdown_connections 2024-04-26T01:36:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:36:25.941+0000 7fb6c2ea9700 1 -- 172.21.15.183:0/21569827 wait complete. 2024-04-26T01:36:26.555 DEBUG:teuthology.orchestra.run.smithi183:osd.4> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T01:36:26.557 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-04-26T01:36:26.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T01:36:26.803 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:27.104 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:26 smithi183 ceph-mon[32752]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:27.191 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:27.191 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.192 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.192 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.192 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.192 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:27.192 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:26 smithi191 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:27.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:26 smithi052 ceph-mon[27851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T01:36:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:28 smithi191 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:36:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:28 smithi191 ceph-mon[32745]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:28 smithi191 ceph-mon[32745]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:36:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:28.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:28 smithi191 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:28 smithi183 ceph-mon[32752]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:36:28.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:28 smithi183 ceph-mon[32752]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:28 smithi183 ceph-mon[32752]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:36:28.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:28.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:28 smithi183 ceph-mon[32752]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:28 smithi052 ceph-mon[27851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T01:36:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:28 smithi052 ceph-mon[27851]: from='osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:28 smithi052 ceph-mon[27851]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T01:36:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:28.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:28 smithi052 ceph-mon[27851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113183 s, 926 MB/s 2024-04-26T01:36:28.982 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-26T01:36:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:36:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: osdmap e34: 5 total, 4 up, 5 in 2024-04-26T01:36:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-mon[32752]: from='osd.4 ' entity='osd.4' 2024-04-26T01:36:29.254 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T01:36:29.099+0000 7fcce3829700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T01:36:29.254 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 01:36:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T01:36:29.257+0000 7fccdb614700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:36:29.456 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:29.456 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:36:29.457 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: osdmap e34: 5 total, 4 up, 5 in 2024-04-26T01:36:29.457 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.457 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.457 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:29 smithi191 ceph-mon[32745]: from='osd.4 ' entity='osd.4' 2024-04-26T01:36:29.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:29.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:36:29.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: osdmap e34: 5 total, 4 up, 5 in 2024-04-26T01:36:29.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:29.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:29 smithi052 ceph-mon[27851]: from='osd.4 ' entity='osd.4' 2024-04-26T01:36:29.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-04-26T01:36:29.933 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:36:30.305 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:30.306 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.307 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:30.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:30.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi183 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 8544M 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] boot 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:36:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:31 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:31.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi183 2024-04-26T01:36:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 8544M 2024-04-26T01:36:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] boot 2024-04-26T01:36:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:36:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:31 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1997116563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac102bc0 msgr2=0x7fb2ac103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1997116563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac102bc0 0x7fb2ac103020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb298009990 tx=0x7fb298009ca0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1997116563 shutdown_connections 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1997116563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac103560 0x7fb2ac109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1997116563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.480 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1997116563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac102bc0 0x7fb2ac103020 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1997116563 >> 172.21.15.191:0/1997116563 conn(0x7fb2ac0fcfa0 msgr2=0x7fb2ac0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1997116563 shutdown_connections 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.482+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1997116563 wait complete. 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 Processor -- start 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 -- start start 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac102bc0 0x7fb2ac19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac103560 0x7fb2ac1a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10d4f0 con 0x7fb2ac102bc0 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10d370 con 0x7fb2ac103560 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b2ba7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10d670 con 0x7fb2ac1019c0 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:31.481 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59456/0 (socket says 172.21.15.191:59456) 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 -- 172.21.15.191:0/1674685940 learned_addr learned my addr 172.21.15.191:0/1674685940 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 -- 172.21.15.191:0/1674685940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac103560 msgr2=0x7fb2ac1a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac103560 0x7fb2ac1a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 -- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac102bc0 msgr2=0x7fb2ac19d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac102bc0 0x7fb2ac19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.483+0000 7fb2b0943700 1 -- 172.21.15.191:0/1674685940 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac1a08f0 con 0x7fb2ac1019c0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2b0943700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b500 tx=0x7fb2a000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0015020 con 0x7fb2ac1019c0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a0011ce0 con 0x7fb2ac1019c0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac1a0be0 con 0x7fb2ac1019c0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0010d60 con 0x7fb2ac1019c0 2024-04-26T01:36:31.482 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.484+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac1a1170 con 0x7fb2ac1019c0 2024-04-26T01:36:31.483 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.485+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ac069730 con 0x7fb2ac1019c0 2024-04-26T01:36:31.483 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.486+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb2a0018020 con 0x7fb2ac1019c0 2024-04-26T01:36:31.483 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.486+0000 7fb2a9ffb700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 0x7fb29c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:31.483 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.487+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4101+0+0 (secure 0 0 0) 0x7fb2a00ceb40 con 0x7fb2ac1019c0 2024-04-26T01:36:31.484 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.487+0000 7fb2abfff700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 0x7fb29c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:31.484 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.488+0000 7fb2abfff700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 0x7fb29c0a6180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b5c0 tx=0x7fb298003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:31.491 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.491+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2a001f020 con 0x7fb2ac1019c0 2024-04-26T01:36:31.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 357 GiB / 358 GiB avail 2024-04-26T01:36:31.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi183 2024-04-26T01:36:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 8544M 2024-04-26T01:36:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: osd.4 [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] boot 2024-04-26T01:36:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T01:36:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:31 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:36:31.694 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:31.698+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb2ac062e80 con 0x7fb29c0a3cc0 2024-04-26T01:36:32.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:32 smithi191 ceph-mon[32745]: osdmap e36: 5 total, 5 up, 5 in 2024-04-26T01:36:32.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:32.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:32.543 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:32.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:32 smithi183 ceph-mon[32752]: osdmap e36: 5 total, 5 up, 5 in 2024-04-26T01:36:32.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:32.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:32.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:32.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:32 smithi052 ceph-mon[27851]: osdmap e36: 5 total, 5 up, 5 in 2024-04-26T01:36:32.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:36:32.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:36:32.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:33.568 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:33 smithi191 ceph-mon[32745]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:33.569 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:33 smithi191 ceph-mon[32745]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:33.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:33 smithi183 ceph-mon[32752]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:33 smithi183 ceph-mon[32752]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:33.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:33 smithi052 ceph-mon[27851]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:33.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:33 smithi052 ceph-mon[27851]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:36:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/1856496015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]': finished 2024-04-26T01:36:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:36:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:35 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2862812511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:35.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/1856496015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]': finished 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:35 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2862812511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:35.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/1856496015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]: dispatch 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59ea2f70-c81c-4121-890e-1216b8b142f2"}]': finished 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:35 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2862812511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:36:37.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:37 smithi183 ceph-mon[32752]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:37 smithi191 ceph-mon[32745]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:37.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:37 smithi052 ceph-mon[27851]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:39.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:39 smithi191 ceph-mon[32745]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:39.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:39 smithi183 ceph-mon[32752]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:39.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:39 smithi052 ceph-mon[27851]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:41.299 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:41 smithi191 ceph-mon[32745]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:41.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:41 smithi183 ceph-mon[32752]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:41.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:41 smithi052 ceph-mon[27851]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:42.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:36:42.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:42.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:36:42.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:42.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T01:36:42.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:43.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:43 smithi191 ceph-mon[32745]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:43.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:43 smithi191 ceph-mon[32745]: Deploying daemon osd.5 on smithi191 2024-04-26T01:36:43.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:43 smithi183 ceph-mon[32752]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:43 smithi183 ceph-mon[32752]: Deploying daemon osd.5 on smithi191 2024-04-26T01:36:43.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:43 smithi052 ceph-mon[27851]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:43.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:43 smithi052 ceph-mon[27851]: Deploying daemon osd.5 on smithi191 2024-04-26T01:36:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:45 smithi191 ceph-mon[32745]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:45.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:45 smithi183 ceph-mon[32752]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:45 smithi052 ceph-mon[27851]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:47.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:47 smithi191 ceph-mon[32745]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:47.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:47 smithi183 ceph-mon[32752]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:47.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:47 smithi052 ceph-mon[27851]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:49.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:48 smithi191 ceph-mon[32745]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:49.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:49.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:48 smithi183 ceph-mon[32752]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:49.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:49.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:48 smithi052 ceph-mon[27851]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:49.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:49.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:51.129 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:50 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:50 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:51.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:50 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:51.483 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.487+0000 7fb2a9ffb700 1 -- 172.21.15.191:0/1674685940 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb2ac062e80 con 0x7fb29c0a3cc0 2024-04-26T01:36:51.484 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-04-26T01:36:51.485 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 msgr2=0x7fb29c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 0x7fb29c0a6180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b5c0 tx=0x7fb298003860 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 msgr2=0x7fb2ac06bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b500 tx=0x7fb2a000b8c0 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 shutdown_connections 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb29c0a3cc0 0x7fb29c0a6180 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb2ac103560 0x7fb2ac1a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2ac1019c0 0x7fb2ac06bd30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 --2- 172.21.15.191:0/1674685940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac102bc0 0x7fb2ac19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 >> 172.21.15.191:0/1674685940 conn(0x7fb2ac0fcfa0 msgr2=0x7fb2ac105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:51.486 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 shutdown_connections 2024-04-26T01:36:51.487 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:36:51.489+0000 7fb2b2ba7700 1 -- 172.21.15.191:0/1674685940 wait complete. 2024-04-26T01:36:52.123 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T01:36:52.126 INFO:tasks.cephadm:Deploying osd.6 on smithi052 with /dev/vg_nvme/lv_2... 2024-04-26T01:36:52.126 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:36:52.590 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:52.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:52 smithi052 ceph-mon[27851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:52.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:52 smithi191 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:52.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:36:52.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:52.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:52.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:52.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:52 smithi183 ceph-mon[32752]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T01:36:53.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:53 smithi191 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:36:53.682 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:53 smithi191 ceph-mon[32745]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:53.682 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:53 smithi191 ceph-mon[32745]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:36:53.682 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:53.682 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:53 smithi191 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:53.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:53 smithi052 ceph-mon[27851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:36:53.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:53 smithi052 ceph-mon[27851]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:53.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:53 smithi052 ceph-mon[27851]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:36:53.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:53.850 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:53 smithi052 ceph-mon[27851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:53.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:53 smithi183 ceph-mon[32752]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T01:36:53.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:53 smithi183 ceph-mon[32752]: from='osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:53.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:53 smithi183 ceph-mon[32752]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T01:36:53.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:53.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:53 smithi183 ceph-mon[32752]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:36:54.650 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T01:36:54.381+0000 7f38f7612700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T01:36:54.650 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T01:36:54.564+0000 7f38ef3fd700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: from='osd.5 ' entity='osd.5' 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: osdmap e40: 6 total, 5 up, 6 in 2024-04-26T01:36:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:54.672 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:36:54.672 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:36:54.672 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:54.672 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:54.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: from='osd.5 ' entity='osd.5' 2024-04-26T01:36:54.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: osdmap e40: 6 total, 5 up, 6 in 2024-04-26T01:36:54.673 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:54.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: osdmap e39: 6 total, 5 up, 6 in 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: from='osd.5 ' entity='osd.5' 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: osdmap e40: 6 total, 5 up, 6 in 2024-04-26T01:36:54.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:54 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:55.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:36:55.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:36:55.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi191 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] boot 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:36:55.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:55.561 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:36:55.562 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:36:55.562 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-26T01:36:55.562 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-26T01:36:55.562 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0352208 s, 298 MB/s 2024-04-26T01:36:55.562 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-26T01:36:55.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:36:55.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi191 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] boot 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:36:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:55.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:36:55.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi191 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 8544M 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: osd.5 [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] boot 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T01:36:55.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:36:56.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_2 2024-04-26T01:36:56.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:56 smithi052 ceph-mon[27851]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:56.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:56 smithi052 ceph-mon[27851]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:36:56.773 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:36:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:56 smithi191 ceph-mon[32745]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:56 smithi191 ceph-mon[32745]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:36:56.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:56 smithi183 ceph-mon[32752]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 447 GiB / 447 GiB avail 2024-04-26T01:36:56.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:56 smithi183 ceph-mon[32752]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T01:36:58.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:58 smithi052 ceph-mon[27851]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:36:58.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:58 smithi052 ceph-mon[27851]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:36:58.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:58 smithi052 ceph-mon[27851]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:36:58.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:58 smithi191 ceph-mon[32745]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:36:58.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:58 smithi191 ceph-mon[32745]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:36:58.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:58 smithi191 ceph-mon[32745]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:36:58.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:58 smithi183 ceph-mon[32752]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T01:36:58.922 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:58 smithi183 ceph-mon[32752]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:36:58.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:58 smithi183 ceph-mon[32752]: osdmap e44: 6 total, 6 up, 6 in 2024-04-26T01:36:58.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 -- 172.21.15.52:0/3891676447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 msgr2=0x7f6944108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 --2- 172.21.15.52:0/3891676447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f6944108250 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f692c0098f0 tx=0x7f692c009c00 comp rx=0 tx=0).stop 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 -- 172.21.15.52:0/3891676447 shutdown_connections 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 --2- 172.21.15.52:0/3891676447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 0x7f694410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 --2- 172.21.15.52:0/3891676447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 0x7f6944105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 --2- 172.21.15.52:0/3891676447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f6944108250 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 -- 172.21.15.52:0/3891676447 >> 172.21.15.52:0/3891676447 conn(0x7f69440fcf60 msgr2=0x7f69440ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 -- 172.21.15.52:0/3891676447 shutdown_connections 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.947+0000 7f694ac00700 1 -- 172.21.15.52:0/3891676447 wait complete. 2024-04-26T01:36:58.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 Processor -- start 2024-04-26T01:36:58.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 -- start start 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 0x7f694419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 0x7f69441a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f694410e580 con 0x7f6944108790 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f694410e400 con 0x7f6944105e60 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.949+0000 7f694ac00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f694410e700 con 0x7f6944103530 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6948c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 0x7f69441a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49574/0 (socket says 172.21.15.52:49574) 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 -- 172.21.15.52:0/3273993439 learned_addr learned my addr 172.21.15.52:0/3273993439 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:36:58.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943e00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 0x7f694419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:58.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 -- 172.21.15.52:0/3273993439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 msgr2=0x7f694419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:58.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 0x7f694419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:58.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 -- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 msgr2=0x7f69441a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:36:58.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 0x7f69441a2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.950+0000 7f6943400700 1 -- 172.21.15.52:0/3273993439 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69441a2b10 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f6943400700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f692c00c010 tx=0x7f692c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c005460 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f692c005600 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69441a2da0 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c003b60 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.951+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69441a31f0 con 0x7f6944105e60 2024-04-26T01:36:58.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.952+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6944069730 con 0x7f6944105e60 2024-04-26T01:36:58.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.954+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f692c003d00 con 0x7f6944105e60 2024-04-26T01:36:58.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.956+0000 7f6940c00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 0x7f69280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:36:58.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.956+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 4533+0+0 (secure 0 0 0) 0x7f692c0d05e0 con 0x7f6944105e60 2024-04-26T01:36:58.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.956+0000 7f6943e00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 0x7f69280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:36:58.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.957+0000 7f6943e00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 0x7f69280a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6934009c60 tx=0x7f6934009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:36:58.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:58.958+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f692c013030 con 0x7f6944105e60 2024-04-26T01:36:59.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:36:59.203+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6944101bb0 con 0x7f69280a3cc0 2024-04-26T01:37:00.072 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:59 smithi191 ceph-mon[32745]: osdmap e45: 6 total, 6 up, 6 in 2024-04-26T01:37:00.072 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:59 smithi191 ceph-mon[32745]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:00.072 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:00.072 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:00.072 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:36:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:00.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:59 smithi183 ceph-mon[32752]: osdmap e45: 6 total, 6 up, 6 in 2024-04-26T01:37:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:59 smithi183 ceph-mon[32752]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:36:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:00.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:59 smithi052 ceph-mon[27851]: osdmap e45: 6 total, 6 up, 6 in 2024-04-26T01:37:00.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:59 smithi052 ceph-mon[27851]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:00.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:00.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:00.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:36:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:00.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:00 smithi052 ceph-mon[27851]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:37:01.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:00 smithi191 ceph-mon[32745]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:37:01.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:00 smithi183 ceph-mon[32752]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:37:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:02 smithi191 ceph-mon[32745]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-26T01:37:03.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:02 smithi183 ceph-mon[32752]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-26T01:37:03.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:02 smithi052 ceph-mon[27851]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-26T01:37:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:03 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/453199387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:03 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:03 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]': finished 2024-04-26T01:37:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:03 smithi191 ceph-mon[32745]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:37:04.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:04.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:03 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/453199387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:03 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:03 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]': finished 2024-04-26T01:37:04.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:03 smithi183 ceph-mon[32752]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:37:04.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:04.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:03 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/453199387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:03 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]: dispatch 2024-04-26T01:37:04.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:03 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b37abf58-d1b9-4290-bcc1-c1163dfc256f"}]': finished 2024-04-26T01:37:04.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:03 smithi052 ceph-mon[27851]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T01:37:04.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:04 smithi191 ceph-mon[32745]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:04 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3762843223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:05.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:04 smithi183 ceph-mon[32752]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:05.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:04 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3762843223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:05.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:04 smithi052 ceph-mon[27851]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:37:05.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:04 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3762843223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:07.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:06 smithi191 ceph-mon[32745]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:37:07.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:06 smithi183 ceph-mon[32752]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:37:07.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:06 smithi052 ceph-mon[27851]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T01:37:09.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:08 smithi191 ceph-mon[32745]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:09.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:08 smithi183 ceph-mon[32752]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:09.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:08 smithi052 ceph-mon[27851]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:11.085 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:10 smithi191 ceph-mon[32745]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:11.094 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:10 smithi052 ceph-mon[27851]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:11.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:10 smithi183 ceph-mon[32752]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:12 smithi191 ceph-mon[32745]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:37:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:13.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:12 smithi183 ceph-mon[32752]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:13.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:37:13.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:13.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:12 smithi052 ceph-mon[27851]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:13.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T01:37:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:13.880 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:13 smithi052 ceph-mon[27851]: Deploying daemon osd.6 on smithi052 2024-04-26T01:37:14.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:13 smithi191 ceph-mon[32745]: Deploying daemon osd.6 on smithi052 2024-04-26T01:37:14.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:13 smithi183 ceph-mon[32752]: Deploying daemon osd.6 on smithi052 2024-04-26T01:37:15.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:14 smithi191 ceph-mon[32745]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:15.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:14 smithi183 ceph-mon[32752]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:15.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:14 smithi052 ceph-mon[27851]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:17.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:16 smithi191 ceph-mon[32745]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:17.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:16 smithi052 ceph-mon[27851]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:17.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:16 smithi183 ceph-mon[32752]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:19.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:18 smithi191 ceph-mon[32745]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:19.171 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:18 smithi052 ceph-mon[27851]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:19.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:18 smithi183 ceph-mon[32752]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:21.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:20 smithi191 ceph-mon[32745]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:21.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:20 smithi183 ceph-mon[32752]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:21.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:20 smithi052 ceph-mon[27851]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:22.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:22.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:22.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:22.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:23.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:23 smithi191 ceph-mon[32745]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:23.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:23 smithi183 ceph-mon[32752]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:23.445 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:23 smithi052 ceph-mon[27851]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:25.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:25 smithi052 ceph-mon[27851]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:25.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:25 smithi191 ceph-mon[32745]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:25.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:25 smithi183 ceph-mon[32752]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:25.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.853+0000 7f6940c00700 1 -- 172.21.15.52:0/3273993439 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6944101bb0 con 0x7f69280a3cc0 2024-04-26T01:37:25.851 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 6 on host 'smithi052' 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 msgr2=0x7f69280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 0x7f69280a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6934009c60 tx=0x7f6934009260 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 msgr2=0x7f694419b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f692c00c010 tx=0x7f692c005270 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 shutdown_connections 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f69280a3cc0 0x7f69280a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6944108790 0x7f69441a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6944103530 0x7f694419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 --2- 172.21.15.52:0/3273993439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6944105e60 0x7f694419b3f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:25.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.856+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 >> 172.21.15.52:0/3273993439 conn(0x7f69440fcf60 msgr2=0x7f69441014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:25.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.857+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 shutdown_connections 2024-04-26T01:37:25.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:37:25.857+0000 7f694ac00700 1 -- 172.21.15.52:0/3273993439 wait complete. 2024-04-26T01:37:26.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:26 smithi052 ceph-mon[27851]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:37:26.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:26 smithi191 ceph-mon[32745]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:37:26.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:26 smithi183 ceph-mon[32752]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T01:37:26.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:26.797 DEBUG:teuthology.orchestra.run.smithi052:osd.6> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T01:37:26.804 INFO:tasks.cephadm:Deploying osd.7 on smithi183 with /dev/vg_nvme/lv_2... 2024-04-26T01:37:26.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:37:27.048 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:37:27.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:27 smithi183 ceph-mon[32752]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:27.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:27 smithi183 ceph-mon[32752]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:37:27.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:27 smithi183 ceph-mon[32752]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:37:27.170 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:27 smithi183 ceph-mon[32752]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:37:27.170 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:27.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-mon[27851]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:27.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-mon[27851]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:37:27.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-mon[27851]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:37:27.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-mon[27851]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:37:27.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:27.380 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T01:37:27.058+0000 7fe815000700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T01:37:27.380 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 01:37:27 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T01:37:27.079+0000 7fe80fe00700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:37:27.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:27 smithi191 ceph-mon[32745]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:27.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:27 smithi191 ceph-mon[32745]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T01:37:27.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:27 smithi191 ceph-mon[32745]: osdmap e47: 7 total, 6 up, 7 in 2024-04-26T01:37:27.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:27 smithi191 ceph-mon[32745]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:37:27.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.305 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:28 smithi183 ceph-mon[32752]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:37:28.305 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:28 smithi183 ceph-mon[32752]: osdmap e48: 7 total, 6 up, 7 in 2024-04-26T01:37:28.305 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.305 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:28 smithi191 ceph-mon[32745]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:37:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:28 smithi191 ceph-mon[32745]: osdmap e48: 7 total, 6 up, 7 in 2024-04-26T01:37:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:28 smithi052 ceph-mon[27851]: from='osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:28 smithi052 ceph-mon[27851]: osdmap e48: 7 total, 6 up, 7 in 2024-04-26T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:28.965 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:37:28.966 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:37:28.966 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-04-26T01:37:28.966 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-04-26T01:37:28.966 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111985 s, 936 MB/s 2024-04-26T01:37:28.966 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] boot 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:37:29.195 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] boot 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:37:29.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:37:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:37:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-04-26T01:37:29.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: osd.6 [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] boot 2024-04-26T01:37:29.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: osdmap e49: 7 total, 7 up, 7 in 2024-04-26T01:37:29.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:37:29.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-04-26T01:37:29.840 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:37:30.229 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:30 smithi183 ceph-mon[32752]: osdmap e50: 7 total, 7 up, 7 in 2024-04-26T01:37:30.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:30 smithi191 ceph-mon[32745]: osdmap e50: 7 total, 7 up, 7 in 2024-04-26T01:37:30.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:30 smithi052 ceph-mon[27851]: osdmap e50: 7 total, 7 up, 7 in 2024-04-26T01:37:31.363 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:31 smithi183 ceph-mon[32752]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 -- 172.21.15.183:0/3829275950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc103550 msgr2=0x7f92cc109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 --2- 172.21.15.183:0/3829275950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc103550 0x7f92cc109e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92c400b0d0 tx=0x7f92c400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 -- 172.21.15.183:0/3829275950 shutdown_connections 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 --2- 172.21.15.183:0/3829275950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc103550 0x7f92cc109e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 --2- 172.21.15.183:0/3829275950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92cc1019b0 0x7f92cc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 --2- 172.21.15.183:0/3829275950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 -- 172.21.15.183:0/3829275950 >> 172.21.15.183:0/3829275950 conn(0x7f92cc0fcf70 msgr2=0x7f92cc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 -- 172.21.15.183:0/3829275950 shutdown_connections 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.393+0000 7f92d0da1700 1 -- 172.21.15.183:0/3829275950 wait complete. 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 Processor -- start 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 -- start start 2024-04-26T01:37:31.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc19fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92cc103550 0x7f92cc1a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc10d590 con 0x7f92cc103550 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc10d410 con 0x7f92cc1019b0 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92d0da1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc10d710 con 0x7f92cc102bb0 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92ca59c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92ca59c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:54080/0 (socket says 172.21.15.183:54080) 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.394+0000 7f92ca59c700 1 -- 172.21.15.183:0/1421169036 learned_addr learned my addr 172.21.15.183:0/1421169036 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 -- 172.21.15.183:0/1421169036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 msgr2=0x7f92cc19fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92c9d9b700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc19fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc19fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 -- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92cc103550 msgr2=0x7f92cc1a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:31.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92cc103550 0x7f92cc1a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 -- 172.21.15.183:0/1421169036 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92cc1a2c20 con 0x7f92cc1019b0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92c9d9b700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc19fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92ca59c700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00b500 tx=0x7f92bc00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc015020 con 0x7f92cc1019b0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92bc011ce0 con 0x7f92cc1019b0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92cc1a2f10 con 0x7f92cc1019b0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc010d60 con 0x7f92cc1019b0 2024-04-26T01:37:31.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.395+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92cc1a3440 con 0x7f92cc1019b0 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.396+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92cc069730 con 0x7f92cc1019b0 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.397+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f92bc018020 con 0x7f92cc1019b0 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.398+0000 7f92bb7fe700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 0x7f92b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.398+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4965+0+0 (secure 0 0 0) 0x7f92bc0cf4c0 con 0x7f92cc1019b0 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.399+0000 7f92c9d9b700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 0x7f92b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:31.395 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.400+0000 7f92c9d9b700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 0x7f92b40a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f92cc06c270 tx=0x7f92c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:31.398 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.402+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92bc01f020 con 0x7f92cc1019b0 2024-04-26T01:37:31.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:31 smithi191 ceph-mon[32745]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:31 smithi052 ceph-mon[27851]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:31.602 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:31.606+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f92cc062e80 con 0x7f92b40a3c40 2024-04-26T01:37:32.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:32.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:32.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:32 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:32.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:32 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:32.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:32.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:32.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:32 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi052 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 5698M 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi052 2024-04-26T01:37:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 5698M 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi052 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 5698M 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/2950384718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]': finished 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:37:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:35.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:35 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/2725302977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/2950384718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]': finished 2024-04-26T01:37:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:37:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:35 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/2725302977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/2950384718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]: dispatch 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "569c973e-98a6-4957-bb2e-3eb411ea369c"}]': finished 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:35 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/2725302977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:37:37.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:37 smithi052 ceph-mon[27851]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:37 smithi191 ceph-mon[32745]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:37.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:37 smithi183 ceph-mon[32752]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:39.509 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:39 smithi183 ceph-mon[32752]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:39.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:39 smithi191 ceph-mon[32745]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:39.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:39 smithi052 ceph-mon[27851]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:41.507 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:41 smithi183 ceph-mon[32752]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:41.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:41 smithi191 ceph-mon[32745]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:41.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:41 smithi052 ceph-mon[27851]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:42.524 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:37:42.525 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:42.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:37:42.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:42.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T01:37:42.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:43.540 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:43 smithi183 ceph-mon[32752]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:43.540 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:43 smithi183 ceph-mon[32752]: Deploying daemon osd.7 on smithi183 2024-04-26T01:37:43.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:43 smithi191 ceph-mon[32745]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:43.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:43 smithi191 ceph-mon[32745]: Deploying daemon osd.7 on smithi183 2024-04-26T01:37:43.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:43 smithi052 ceph-mon[27851]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:43.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:43 smithi052 ceph-mon[27851]: Deploying daemon osd.7 on smithi183 2024-04-26T01:37:45.495 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:45 smithi183 ceph-mon[32752]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:45 smithi191 ceph-mon[32745]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:45.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:45 smithi052 ceph-mon[27851]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:47.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:47 smithi052 ceph-mon[27851]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:47.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:47 smithi183 ceph-mon[32752]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:47 smithi191 ceph-mon[32745]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:48.550 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.550 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.551 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:48 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:48.581 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.581 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.581 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:48 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:48.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:48.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:48 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:49.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:49 smithi191 ceph-mon[32745]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:49.672 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:49 smithi183 ceph-mon[32752]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:49.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:49 smithi052 ceph-mon[27851]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:51.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.201+0000 7f92bb7fe700 1 -- 172.21.15.183:0/1421169036 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f92cc062e80 con 0x7f92b40a3c40 2024-04-26T01:37:51.198 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 7 on host 'smithi183' 2024-04-26T01:37:51.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 msgr2=0x7f92b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:51.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 0x7f92b40a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f92cc06c270 tx=0x7f92c0008040 comp rx=0 tx=0).stop 2024-04-26T01:37:51.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 msgr2=0x7f92cc06bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00b500 tx=0x7f92bc00b8c0 comp rx=0 tx=0).stop 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 shutdown_connections 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f92b40a3c40 0x7f92b40a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92cc103550 0x7f92cc1a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92cc1019b0 0x7f92cc06bcf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 --2- 172.21.15.183:0/1421169036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92cc102bb0 0x7f92cc19fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 >> 172.21.15.183:0/1421169036 conn(0x7f92cc0fcf70 msgr2=0x7f92cc105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 shutdown_connections 2024-04-26T01:37:51.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:37:51.204+0000 7f92d0da1700 1 -- 172.21.15.183:0/1421169036 wait complete. 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:51.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:51.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T01:37:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:37:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:51.828 DEBUG:teuthology.orchestra.run.smithi183:osd.7> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T01:37:51.831 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_2... 2024-04-26T01:37:51.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T01:37:52.076 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:37:52.480 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:52 smithi191 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:37:52.480 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:52 smithi191 ceph-mon[32745]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:52.480 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:52 smithi191 ceph-mon[32745]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:37:52.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:52.481 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:52 smithi191 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:52.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-mon[32752]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:37:52.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-mon[32752]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:52.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-mon[32752]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:37:52.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:52.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-mon[32752]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:52.673 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T01:37:52.345+0000 7ff3c962c700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T01:37:52.674 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 01:37:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T01:37:52.355+0000 7ff3c541f700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:37:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:52 smithi052 ceph-mon[27851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T01:37:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:52 smithi052 ceph-mon[27851]: from='osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:52 smithi052 ceph-mon[27851]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T01:37:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:52.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:52 smithi052 ceph-mon[27851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:37:53.397 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:53.397 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:37:53.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: osdmap e53: 8 total, 7 up, 8 in 2024-04-26T01:37:53.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] boot 2024-04-26T01:37:53.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:37:53.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:37:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: osdmap e53: 8 total, 7 up, 8 in 2024-04-26T01:37:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] boot 2024-04-26T01:37:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:37:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: osdmap e53: 8 total, 7 up, 8 in 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: osd.7 [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] boot 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T01:37:53.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113769 s, 922 MB/s 2024-04-26T01:37:53.953 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-26T01:37:54.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-04-26T01:37:54.781 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:37:55.140 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:54 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:37:55.140 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:54 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:37:55.140 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:54 smithi191 ceph-mon[32745]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:55.141 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:54 smithi191 ceph-mon[32745]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:37:55.141 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:54 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:37:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:54 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:37:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:54 smithi183 ceph-mon[32752]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:54 smithi183 ceph-mon[32752]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:37:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:54 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:54 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:37:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:54 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:37:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:54 smithi052 ceph-mon[27851]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:54 smithi052 ceph-mon[27851]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T01:37:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:55.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi183 2024-04-26T01:37:55.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 5696M 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:55.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi183 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 5696M 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:56.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:56.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi183 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 5696M 2024-04-26T01:37:56.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:56.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:37:56.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:37:56.414 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 -- 172.21.15.191:0/872402387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c104640 msgr2=0x7fdf6c10aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:56.414 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 --2- 172.21.15.191:0/872402387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c104640 0x7fdf6c10aef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400b0a0 tx=0x7fdf6400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.414 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 -- 172.21.15.191:0/872402387 shutdown_connections 2024-04-26T01:37:56.414 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 --2- 172.21.15.191:0/872402387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c104640 0x7fdf6c10aef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 --2- 172.21.15.191:0/872402387 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 0x7fdf6c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 --2- 172.21.15.191:0/872402387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.415+0000 7fdf73b39700 1 -- 172.21.15.191:0/872402387 >> 172.21.15.191:0/872402387 conn(0x7fdf6c0fe040 msgr2=0x7fdf6c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- 172.21.15.191:0/872402387 shutdown_connections 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- 172.21.15.191:0/872402387 wait complete. 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 Processor -- start 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- start start 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 0x7fdf6c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf6c104640 0x7fdf6c1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10e5b0 con 0x7fdf6c104640 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10e430 con 0x7fdf6c102aa0 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.416+0000 7fdf73b39700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c10e730 con 0x7fdf6c103ca0 2024-04-26T01:37:56.415 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:53612/0 (socket says 172.21.15.191:53612) 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 -- 172.21.15.191:0/3105336733 learned_addr learned my addr 172.21.15.191:0/3105336733 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf718d5700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 0x7fdf6c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 -- 172.21.15.191:0/3105336733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 msgr2=0x7fdf6c198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 0x7fdf6c198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 -- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf6c104640 msgr2=0x7fdf6c1a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf6c104640 0x7fdf6c1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 -- 172.21.15.191:0/3105336733 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c1a0890 con 0x7fdf6c103ca0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf710d4700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b530 tx=0x7fdf6800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68015020 con 0x7fdf6c103ca0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf68011ce0 con 0x7fdf6c103ca0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c1a0b80 con 0x7fdf6c103ca0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68010e40 con 0x7fdf6c103ca0 2024-04-26T01:37:56.416 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.417+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c1a1140 con 0x7fdf6c103ca0 2024-04-26T01:37:56.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.418+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c069730 con 0x7fdf6c103ca0 2024-04-26T01:37:56.417 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.419+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf68018020 con 0x7fdf6c103ca0 2024-04-26T01:37:56.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.420+0000 7fdf62ffd700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 0x7fdf580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:37:56.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.420+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5397+0+0 (secure 0 0 0) 0x7fdf680cf0f0 con 0x7fdf6c103ca0 2024-04-26T01:37:56.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.420+0000 7fdf718d5700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 0x7fdf580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:37:56.418 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.421+0000 7fdf718d5700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 0x7fdf580a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c19a120 tx=0x7fdf5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:37:56.421 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.424+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf6801f020 con 0x7fdf6c103ca0 2024-04-26T01:37:56.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:37:56.630+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdf6c062e80 con 0x7fdf580a3c40 2024-04-26T01:37:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:56 smithi191 ceph-mon[32745]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:56 smithi191 ceph-mon[32745]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:37:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:56 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:56 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:56.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:56 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:56 smithi183 ceph-mon[32752]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:56 smithi183 ceph-mon[32752]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:37:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:56 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:56 smithi052 ceph-mon[27851]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:56 smithi052 ceph-mon[27851]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T01:37:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:56 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:37:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:56 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:37:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:56 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:37:58.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:57 smithi191 ceph-mon[32745]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:58.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:57 smithi191 ceph-mon[32745]: osdmap e58: 8 total, 8 up, 8 in 2024-04-26T01:37:58.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:57 smithi183 ceph-mon[32752]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:58.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:57 smithi183 ceph-mon[32752]: osdmap e58: 8 total, 8 up, 8 in 2024-04-26T01:37:58.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:57 smithi052 ceph-mon[27851]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:37:58.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:57 smithi052 ceph-mon[27851]: osdmap e58: 8 total, 8 up, 8 in 2024-04-26T01:37:59.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:58 smithi191 ceph-mon[32745]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:59.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:58 smithi183 ceph-mon[32752]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:59.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:58 smithi052 ceph-mon[27851]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:37:59.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:59 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2295501146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:37:59.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:59 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:37:59.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:59 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]': finished 2024-04-26T01:37:59.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:59 smithi052 ceph-mon[27851]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:37:59.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:37:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:00.051 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:59 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2295501146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:38:00.052 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:59 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:38:00.052 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:59 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]': finished 2024-04-26T01:38:00.052 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:59 smithi191 ceph-mon[32745]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:38:00.052 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:37:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:59 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2295501146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:38:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:59 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]: dispatch 2024-04-26T01:38:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:59 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85ed3a78-d945-4ea1-a9a6-1391924890fa"}]': finished 2024-04-26T01:38:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:59 smithi183 ceph-mon[32752]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T01:38:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:37:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:01.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:00 smithi191 ceph-mon[32745]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:01.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:00 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/3942806819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:01.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:00 smithi183 ceph-mon[32752]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:01.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:00 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/3942806819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:00 smithi052 ceph-mon[27851]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:00 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/3942806819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:03.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:02 smithi191 ceph-mon[32745]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:03.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:02 smithi183 ceph-mon[32752]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:03.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:02 smithi052 ceph-mon[27851]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:05.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:04 smithi191 ceph-mon[32745]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:05.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:04 smithi183 ceph-mon[32752]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:05.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:04 smithi052 ceph-mon[27851]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:07.057 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:06 smithi191 ceph-mon[32745]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:07.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:06 smithi183 ceph-mon[32752]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:07.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:06 smithi052 ceph-mon[27851]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:07.986 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:38:07.986 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:07 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:07.986 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:07 smithi191 ceph-mon[32745]: Deploying daemon osd.8 on smithi191 2024-04-26T01:38:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:38:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:07 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:08.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:07 smithi183 ceph-mon[32752]: Deploying daemon osd.8 on smithi191 2024-04-26T01:38:08.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T01:38:08.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:07 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:08.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:07 smithi052 ceph-mon[27851]: Deploying daemon osd.8 on smithi191 2024-04-26T01:38:09.051 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:08 smithi191 ceph-mon[32745]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:08 smithi183 ceph-mon[32752]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:09.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:08 smithi052 ceph-mon[27851]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:11.094 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:10 smithi191 ceph-mon[32745]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:11.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:10 smithi183 ceph-mon[32752]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:11.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:10 smithi052 ceph-mon[27851]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:12 smithi191 ceph-mon[32745]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:13.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:12 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:13.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:12 smithi183 ceph-mon[32752]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:13.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:12 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:12 smithi052 ceph-mon[27851]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:12 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:15.107 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:14 smithi191 ceph-mon[32745]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:15.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:14 smithi183 ceph-mon[32752]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:15.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:14 smithi052 ceph-mon[27851]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:16.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.876+0000 7fdf62ffd700 1 -- 172.21.15.191:0/3105336733 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdf6c062e80 con 0x7fdf580a3c40 2024-04-26T01:38:16.872 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 msgr2=0x7fdf580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 0x7fdf580a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c19a120 tx=0x7fdf5c00b410 comp rx=0 tx=0).stop 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 msgr2=0x7fdf6c199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800b530 tx=0x7fdf6800b840 comp rx=0 tx=0).stop 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 shutdown_connections 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf580a3c40 0x7fdf580a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:16.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf6c104640 0x7fdf6c1a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf6c102aa0 0x7fdf6c198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 --2- 172.21.15.191:0/3105336733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf6c103ca0 0x7fdf6c199180 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.878+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 >> 172.21.15.191:0/3105336733 conn(0x7fdf6c0fe040 msgr2=0x7fdf6c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.879+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 shutdown_connections 2024-04-26T01:38:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:38:16.879+0000 7fdf73b39700 1 -- 172.21.15.191:0/3105336733 wait complete. 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:17.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:17.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:17.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T01:38:17.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:17.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:17.569 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T01:38:17.572 INFO:tasks.cephadm:Deploying osd.9 on smithi052 with /dev/vg_nvme/lv_1... 2024-04-26T01:38:17.572 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:38:18.041 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:18.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:18 smithi052 ceph-mon[27851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:38:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:18 smithi052 ceph-mon[27851]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:18 smithi052 ceph-mon[27851]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:38:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:18 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:18 smithi052 ceph-mon[27851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:18.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:38:18.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-mon[32745]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:18.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-mon[32745]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:38:18.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:18.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:18.401 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T01:38:18.106+0000 7f7523e28700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T01:38:18.401 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 01:38:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T01:38:18.118+0000 7f751fc1b700 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:38:18.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:18 smithi183 ceph-mon[32752]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T01:38:18.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:18 smithi183 ceph-mon[32752]: from='osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:18.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:18 smithi183 ceph-mon[32752]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T01:38:18.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:18 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:18.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:18 smithi183 ceph-mon[32752]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:38:19.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:19 smithi052 ceph-mon[27851]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:19.206 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:19 smithi052 ceph-mon[27851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:38:19.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:19 smithi052 ceph-mon[27851]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:38:19.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:19.207 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:19.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:19 smithi191 ceph-mon[32745]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:19 smithi191 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:38:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:19 smithi191 ceph-mon[32745]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:38:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:19.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:19 smithi183 ceph-mon[32752]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:19.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:19 smithi183 ceph-mon[32752]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:38:19.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:19 smithi183 ceph-mon[32752]: osdmap e61: 9 total, 8 up, 9 in 2024-04-26T01:38:19.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:19.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:20.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:20 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:38:20.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:20 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:38:20.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:20 smithi052 ceph-mon[27851]: osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] boot 2024-04-26T01:38:20.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:20 smithi052 ceph-mon[27851]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:38:20.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:20 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:20.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:20 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:38:20.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:20 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:38:20.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:20 smithi191 ceph-mon[32745]: osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] boot 2024-04-26T01:38:20.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:20 smithi191 ceph-mon[32745]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:38:20.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:20 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:20.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:20 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:38:20.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:20 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:38:20.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:20 smithi183 ceph-mon[32752]: osd.8 [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] boot 2024-04-26T01:38:20.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:20 smithi183 ceph-mon[32752]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T01:38:20.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:20 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:38:20.858 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:38:20.858 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:38:20.858 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-04-26T01:38:20.859 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-04-26T01:38:20.859 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0351125 s, 299 MB/s 2024-04-26T01:38:20.859 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-04-26T01:38:21.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:21.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:21 smithi191 ceph-mon[32745]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:21 smithi183 ceph-mon[32752]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:21 smithi052 ceph-mon[27851]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T01:38:21.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_1 2024-04-26T01:38:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:38:22.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:22 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi191 2024-04-26T01:38:22.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:22 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-26T01:38:22.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:22 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi191 2024-04-26T01:38:22.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:22 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-26T01:38:22.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:22 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi191 2024-04-26T01:38:22.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:22 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 5696M 2024-04-26T01:38:23.400 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:23 smithi191 ceph-mon[32745]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:23.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:23 smithi191 ceph-mon[32745]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:38:23.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:23 smithi183 ceph-mon[32752]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:23.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:23 smithi183 ceph-mon[32752]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:38:23.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:23 smithi052 ceph-mon[27851]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:23.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:23 smithi052 ceph-mon[27851]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 -- 172.21.15.52:0/1414884298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc102ba0 msgr2=0x7f9ddc103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 --2- 172.21.15.52:0/1414884298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80099c0 tx=0x7f9dc8009cd0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 -- 172.21.15.52:0/1414884298 shutdown_connections 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 --2- 172.21.15.52:0/1414884298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc103540 0x7f9ddc109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 --2- 172.21.15.52:0/1414884298 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 --2- 172.21.15.52:0/1414884298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 -- 172.21.15.52:0/1414884298 >> 172.21.15.52:0/1414884298 conn(0x7f9ddc0fcf40 msgr2=0x7f9ddc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 -- 172.21.15.52:0/1414884298 shutdown_connections 2024-04-26T01:38:24.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.396+0000 7f9de4600700 1 -- 172.21.15.52:0/1414884298 wait complete. 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.397+0000 7f9de4600700 1 Processor -- start 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.397+0000 7f9de4600700 1 -- start start 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.397+0000 7f9de4600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.397+0000 7f9de4600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de4600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc103540 0x7f9ddc1044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de4600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc10d490 con 0x7f9ddc103540 2024-04-26T01:38:24.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de4600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc10d310 con 0x7f9ddc1019a0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de4600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc10d610 con 0x7f9ddc102ba0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34448/0 (socket says 172.21.15.52:34448) 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 -- 172.21.15.52:0/3235641452 learned_addr learned my addr 172.21.15.52:0/3235641452 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1200700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de2600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc103540 0x7f9ddc1044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 -- 172.21.15.52:0/3235641452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 msgr2=0x7f9ddc103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 -- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc103540 msgr2=0x7f9ddc1044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc103540 0x7f9ddc1044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1c00700 1 -- 172.21.15.52:0/3235641452 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc104d90 con 0x7f9ddc1019a0 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.398+0000 7f9de1200700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:38:24.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9de1c00700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c140 tx=0x7f9dd000c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:24.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd000ccd0 con 0x7f9ddc1019a0 2024-04-26T01:38:24.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd0018070 con 0x7f9ddc1019a0 2024-04-26T01:38:24.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ddc1a2dd0 con 0x7f9ddc1019a0 2024-04-26T01:38:24.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0013660 con 0x7f9ddc1019a0 2024-04-26T01:38:24.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.399+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ddc1a31c0 con 0x7f9ddc1019a0 2024-04-26T01:38:24.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.400+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ddc069730 con 0x7f9ddc1019a0 2024-04-26T01:38:24.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.403+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9dd0012070 con 0x7f9ddc1019a0 2024-04-26T01:38:24.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.404+0000 7f9dda000700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:24.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.404+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5829+0+0 (secure 0 0 0) 0x7f9dd00cf520 con 0x7f9ddc1019a0 2024-04-26T01:38:24.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.404+0000 7f9de1200700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:24.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.405+0000 7f9de1200700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003820 tx=0x7f9dc8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:24.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.407+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dd0022020 con 0x7f9ddc1019a0 2024-04-26T01:38:24.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:24 smithi183 ceph-mon[32752]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:38:24.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:24 smithi052 ceph-mon[27851]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:38:24.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:24.646+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9ddc06ae30 con 0x7f9dc40a3cc0 2024-04-26T01:38:24.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:24 smithi191 ceph-mon[32745]: osdmap e66: 9 total, 9 up, 9 in 2024-04-26T01:38:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:25 smithi183 ceph-mon[32752]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:25.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:25.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:25 smithi052 ceph-mon[27851]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:25.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:25.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:25.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:25.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:25 smithi191 ceph-mon[32745]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T01:38:25.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:26.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:26 smithi183 ceph-mon[32752]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:26.457 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:26 smithi052 ceph-mon[27851]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:26.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:26 smithi191 ceph-mon[32745]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:27.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:27 smithi183 ceph-mon[32752]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T01:38:27.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:27 smithi052 ceph-mon[27851]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T01:38:27.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:27 smithi191 ceph-mon[32745]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-26T01:38:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:29 smithi052 ceph-mon[27851]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-26T01:38:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:29 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]: dispatch 2024-04-26T01:38:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:29 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]': finished 2024-04-26T01:38:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:29 smithi052 ceph-mon[27851]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:38:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:29 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:29.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:29 smithi191 ceph-mon[32745]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-26T01:38:29.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:29 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]: dispatch 2024-04-26T01:38:29.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:29 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]': finished 2024-04-26T01:38:29.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:29 smithi191 ceph-mon[32745]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:38:29.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:29 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:29 smithi183 ceph-mon[32752]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-26T01:38:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:29 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]: dispatch 2024-04-26T01:38:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:29 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2707724444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd9bae00-f288-4771-a5c7-da5fb4c4f7d8"}]': finished 2024-04-26T01:38:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:29 smithi183 ceph-mon[32752]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T01:38:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:29 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:30.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:30 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1219023423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:30.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:30 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1219023423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:30 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1219023423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:38:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:31 smithi052 ceph-mon[27851]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:38:31.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:31 smithi191 ceph-mon[32745]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:38:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:31 smithi183 ceph-mon[32752]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T01:38:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:33 smithi052 ceph-mon[27851]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:38:33.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:33 smithi191 ceph-mon[32745]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:38:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:33 smithi183 ceph-mon[32752]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T01:38:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:35 smithi052 ceph-mon[27851]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:35.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:35 smithi191 ceph-mon[32745]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:35 smithi183 ceph-mon[32752]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:37.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:37 smithi052 ceph-mon[27851]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:37.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:37 smithi191 ceph-mon[32745]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:37.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:37 smithi183 ceph-mon[32752]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:39.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:39 smithi052 ceph-mon[27851]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:39.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:38:39.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:39.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:39 smithi052 ceph-mon[27851]: Deploying daemon osd.9 on smithi052 2024-04-26T01:38:39.475 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:39 smithi183 ceph-mon[32752]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:39.475 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:38:39.475 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:39.475 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:39 smithi183 ceph-mon[32752]: Deploying daemon osd.9 on smithi052 2024-04-26T01:38:39.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:39 smithi191 ceph-mon[32745]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T01:38:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:39 smithi191 ceph-mon[32745]: Deploying daemon osd.9 on smithi052 2024-04-26T01:38:41.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:41 smithi052 ceph-mon[27851]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:41 smithi191 ceph-mon[32745]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:41.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:41 smithi183 ceph-mon[32752]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:43.337 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:43 smithi052 ceph-mon[27851]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:43.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:43 smithi191 ceph-mon[32745]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:43 smithi183 ceph-mon[32752]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:45.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:45 smithi052 ceph-mon[27851]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:45.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:45 smithi191 ceph-mon[32745]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:45.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:45 smithi183 ceph-mon[32752]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:47.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:47 smithi052 ceph-mon[27851]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:47.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:47.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:47.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:47.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:47 smithi191 ceph-mon[32745]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:47.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:47.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:47.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:47.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:47 smithi183 ceph-mon[32752]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:47.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:47.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:47.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:49.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:49 smithi052 ceph-mon[27851]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:49.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:49 smithi191 ceph-mon[32745]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:49.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:49 smithi183 ceph-mon[32752]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:51.588 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:51 smithi052 ceph-mon[27851]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:51.588 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:51 smithi052 ceph-mon[27851]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:38:51.588 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.588 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:51 smithi191 ceph-mon[32745]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:51 smithi191 ceph-mon[32745]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:38:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:51 smithi183 ceph-mon[32752]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:51.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:51 smithi183 ceph-mon[32752]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T01:38:51.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:51.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.694+0000 7f9dda000700 1 -- 172.21.15.52:0/3235641452 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9ddc06ae30 con 0x7f9dc40a3cc0 2024-04-26T01:38:51.692 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 9 on host 'smithi052' 2024-04-26T01:38:51.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.697+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 msgr2=0x7f9dc40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:51.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.697+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003820 tx=0x7f9dc8005fb0 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 msgr2=0x7f9ddc1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c140 tx=0x7f9dd000c450 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 shutdown_connections 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ddc103540 0x7f9ddc1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ddc1019a0 0x7f9ddc1058e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 --2- 172.21.15.52:0/3235641452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ddc102ba0 0x7f9ddc103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 >> 172.21.15.52:0/3235641452 conn(0x7f9ddc0fcf40 msgr2=0x7f9ddc06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 shutdown_connections 2024-04-26T01:38:51.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:38:51.698+0000 7f9de4600700 1 -- 172.21.15.52:0/3235641452 wait complete. 2024-04-26T01:38:52.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:38:52.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.503 DEBUG:teuthology.orchestra.run.smithi052:osd.9> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T01:38:52.510 INFO:tasks.cephadm:Deploying osd.10 on smithi183 with /dev/vg_nvme/lv_1... 2024-04-26T01:38:52.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:38:52.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: osdmap e69: 10 total, 9 up, 10 in 2024-04-26T01:38:52.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:52.756 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:38:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:53 smithi183 ceph-mon[32752]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:53.506 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:53 smithi052 ceph-mon[27851]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:53.507 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:53.507 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:53.507 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 01:38:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T01:38:53.293+0000 7fc4cca00700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T01:38:53.507 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 01:38:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T01:38:53.332+0000 7fc4c1e00700 -1 osd.9 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:38:53.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:53 smithi191 ceph-mon[32745]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:54.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:54 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:38:54.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:54 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:38:54.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:54 smithi052 ceph-mon[27851]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' 2024-04-26T01:38:54.459 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:54 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:38:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:54 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:38:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:54 smithi191 ceph-mon[32745]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' 2024-04-26T01:38:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:54 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:38:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:54 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:38:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:54 smithi183 ceph-mon[32752]: from='osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213]' entity='osd.9' 2024-04-26T01:38:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:54 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:54.865 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:38:54.865 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:38:54.865 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-04-26T01:38:54.866 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-04-26T01:38:54.866 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115968 s, 904 MB/s 2024-04-26T01:38:54.866 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-04-26T01:38:55.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:55 smithi052 ceph-mon[27851]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:55 smithi052 ceph-mon[27851]: osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] boot 2024-04-26T01:38:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:55 smithi052 ceph-mon[27851]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:38:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:55 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:55.583 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:55 smithi183 ceph-mon[32752]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:55.583 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:55 smithi183 ceph-mon[32752]: osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] boot 2024-04-26T01:38:55.583 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:55 smithi183 ceph-mon[32752]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:38:55.583 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:55 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:55.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-04-26T01:38:55.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:55 smithi191 ceph-mon[32745]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail 2024-04-26T01:38:55.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:55 smithi191 ceph-mon[32745]: osd.9 [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] boot 2024-04-26T01:38:55.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:55 smithi191 ceph-mon[32745]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T01:38:55.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:55 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:38:55.899 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T01:38:56.612 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:56 smithi183 ceph-mon[32752]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:38:56.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:56 smithi191 ceph-mon[32745]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:38:56.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:56 smithi052 ceph-mon[27851]: osdmap e71: 10 total, 10 up, 10 in 2024-04-26T01:38:57.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 -- 172.21.15.183:0/330755915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 msgr2=0x7fea70109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:57.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 --2- 172.21.15.183:0/330755915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea70109df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b0d0 tx=0x7fea6800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 -- 172.21.15.183:0/330755915 shutdown_connections 2024-04-26T01:38:57.446 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 --2- 172.21.15.183:0/330755915 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea70109df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 --2- 172.21.15.183:0/330755915 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea701019a0 0x7fea70101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 --2- 172.21.15.183:0/330755915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 0x7fea70103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 -- 172.21.15.183:0/330755915 >> 172.21.15.183:0/330755915 conn(0x7fea700fcf40 msgr2=0x7fea700ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 -- 172.21.15.183:0/330755915 shutdown_connections 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.447+0000 7fea77d09700 1 -- 172.21.15.183:0/330755915 wait complete. 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 Processor -- start 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 -- start start 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea701019a0 0x7fea70198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 0x7fea70199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:57.447 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010d530 con 0x7fea701019a0 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010d3b0 con 0x7fea70103540 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea77d09700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010d6b0 con 0x7fea70102ba0 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42120/0 (socket says 172.21.15.183:42120) 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 -- 172.21.15.183:0/307119185 learned_addr learned my addr 172.21.15.183:0/307119185 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea752a4700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 0x7fea70199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 -- 172.21.15.183:0/307119185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 msgr2=0x7fea70199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 0x7fea70199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 -- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea701019a0 msgr2=0x7fea70198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea701019a0 0x7fea70198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.448+0000 7fea762a6700 1 -- 172.21.15.183:0/307119185 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea701a0990 con 0x7fea70103540 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea762a6700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea6800d010 tx=0x7fea68007b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:57.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68021070 con 0x7fea70103540 2024-04-26T01:38:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea68007e50 con 0x7fea70103540 2024-04-26T01:38:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea701a0c20 con 0x7fea70103540 2024-04-26T01:38:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6801bd30 con 0x7fea70103540 2024-04-26T01:38:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.449+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea701a10d0 con 0x7fea70103540 2024-04-26T01:38:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.450+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea70069760 con 0x7fea70103540 2024-04-26T01:38:57.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.451+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fea68024070 con 0x7fea70103540 2024-04-26T01:38:57.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.451+0000 7fea66ffd700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 0x7fea5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:38:57.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.452+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6261+0+0 (secure 0 0 0) 0x7fea6800f020 con 0x7fea70103540 2024-04-26T01:38:57.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.452+0000 7fea75aa5700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 0x7fea5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:38:57.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.453+0000 7fea75aa5700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 0x7fea5c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea7019a220 tx=0x7fea6c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:38:57.451 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.455+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea70069760 con 0x7fea70103540 2024-04-26T01:38:57.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:57 smithi191 ceph-mon[32745]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:57.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:38:57.660+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fea7006ae30 con 0x7fea5c0a3c40 2024-04-26T01:38:57.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:57 smithi183 ceph-mon[32752]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:57.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:57 smithi052 ceph-mon[27851]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:58.627 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:58 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:58 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:38:58.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:58.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:58 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi052 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 4273M 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:59.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:38:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:59.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi052 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 4273M 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:38:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:38:59.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:38:59.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi052 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 4273M 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:38:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:38:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/2141027128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]': finished 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:01.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:01 smithi191 ceph-mon[32745]: from='client.? 172.21.15.183:0/762722340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:01.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:01.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/2141027128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]': finished 2024-04-26T01:39:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:39:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:01 smithi183 ceph-mon[32752]: from='client.? 172.21.15.183:0/762722340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:01.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:01.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/2141027128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]: dispatch 2024-04-26T01:39:01.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3f07a6-f55c-443d-9e9d-16036d02a9ab"}]': finished 2024-04-26T01:39:01.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T01:39:01.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:01.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:01 smithi052 ceph-mon[27851]: from='client.? 172.21.15.183:0/762722340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:03.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:03 smithi191 ceph-mon[32745]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:03.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:03 smithi183 ceph-mon[32752]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:03 smithi052 ceph-mon[27851]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:05.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:05 smithi191 ceph-mon[32745]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:05.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:05 smithi183 ceph-mon[32752]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:05.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:05 smithi052 ceph-mon[27851]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:07 smithi191 ceph-mon[32745]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:07 smithi183 ceph-mon[32752]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:07 smithi052 ceph-mon[27851]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:08.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:39:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:08.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:39:08.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:08.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T01:39:08.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:09.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:09 smithi191 ceph-mon[32745]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:09.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:09 smithi191 ceph-mon[32745]: Deploying daemon osd.10 on smithi183 2024-04-26T01:39:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:09 smithi183 ceph-mon[32752]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:09 smithi183 ceph-mon[32752]: Deploying daemon osd.10 on smithi183 2024-04-26T01:39:09.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:09 smithi052 ceph-mon[27851]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:09.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:09 smithi052 ceph-mon[27851]: Deploying daemon osd.10 on smithi183 2024-04-26T01:39:11.574 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:11 smithi183 ceph-mon[32752]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:11.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:11 smithi191 ceph-mon[32745]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:11.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:11 smithi052 ceph-mon[27851]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:13.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:13 smithi191 ceph-mon[32745]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:13.664 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:13 smithi183 ceph-mon[32752]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:13.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:13 smithi052 ceph-mon[27851]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:15.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:14 smithi191 ceph-mon[32745]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:15.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:15.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:14 smithi183 ceph-mon[32752]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:15.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:15.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:14 smithi052 ceph-mon[27851]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:15.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:15.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:17.013 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:16 smithi052 ceph-mon[27851]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:17.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:16 smithi191 ceph-mon[32745]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:17.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:16 smithi183 ceph-mon[32752]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:17.594 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 10 on host 'smithi183' 2024-04-26T01:39:17.595 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.597+0000 7fea66ffd700 1 -- 172.21.15.183:0/307119185 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fea7006ae30 con 0x7fea5c0a3c40 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 msgr2=0x7fea5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 0x7fea5c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea7019a220 tx=0x7fea6c006d20 comp rx=0 tx=0).stop 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 msgr2=0x7fea701a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea6800d010 tx=0x7fea68007b40 comp rx=0 tx=0).stop 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 shutdown_connections 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fea5c0a3c40 0x7fea5c0a6100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:17.597 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea70103540 0x7fea701a02c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:17.598 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fea701019a0 0x7fea70198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:17.598 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 --2- 172.21.15.183:0/307119185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea70102ba0 0x7fea70199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:17.598 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 >> 172.21.15.183:0/307119185 conn(0x7fea700fcf40 msgr2=0x7fea7006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:17.598 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 shutdown_connections 2024-04-26T01:39:17.598 INFO:teuthology.orchestra.run.smithi183.stderr:2024-04-26T01:39:17.600+0000 7fea77d09700 1 -- 172.21.15.183:0/307119185 wait complete. 2024-04-26T01:39:17.908 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:17.908 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:17.908 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:17.908 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:17.908 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:17.909 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:17.909 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:17.909 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:17.909 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:17.909 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:17 smithi183 ceph-mon[32752]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.049 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.049 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:18.049 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:18.049 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:17 smithi052 ceph-mon[27851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:18.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:18.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:17 smithi191 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T01:39:18.209 DEBUG:teuthology.orchestra.run.smithi183:osd.10> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T01:39:18.211 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2024-04-26T01:39:18.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T01:39:18.453 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:18 smithi191 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:19.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:19.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:39:19.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:19.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:39:19.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:19.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:18 smithi183 ceph-mon[32752]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:19.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:19.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T01:39:19.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: from='osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:19.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T01:39:19.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:19.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:18 smithi052 ceph-mon[27851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113396 s, 925 MB/s 2024-04-26T01:39:20.526 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:20.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:20 smithi191 ceph-mon[32745]: from='osd.10 ' entity='osd.10' 2024-04-26T01:39:20.783 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:39:20.783 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:39:20.783 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.783 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:20.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-mon[32752]: from='osd.10 ' entity='osd.10' 2024-04-26T01:39:20.784 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T01:39:20.499+0000 7f964fb4c700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T01:39:20.784 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 01:39:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T01:39:20.690+0000 7f9647937700 -1 osd.10 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:39:20.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-04-26T01:39:20.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: osdmap e74: 11 total, 10 up, 11 in 2024-04-26T01:39:20.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:20 smithi052 ceph-mon[27851]: from='osd.10 ' entity='osd.10' 2024-04-26T01:39:21.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-04-26T01:39:21.380 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:39:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:39:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:39:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: osdmap e75: 11 total, 10 up, 11 in 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi183 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:21 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: osdmap e75: 11 total, 10 up, 11 in 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi183 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.755 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:21 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: osdmap e75: 11 total, 10 up, 11 in 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi183 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:21 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:22.868 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:22 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 4272M 2024-04-26T01:39:22.869 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:22 smithi191 ceph-mon[32745]: osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] boot 2024-04-26T01:39:22.869 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:22 smithi191 ceph-mon[32745]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:39:22.869 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:22.869 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:22 smithi191 ceph-mon[32745]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:22.888 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.891+0000 7fb72ff29700 1 -- 172.21.15.191:0/3734765214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 msgr2=0x7fb728102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:22.888 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.891+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3734765214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728102ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb724009990 tx=0x7fb724009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- 172.21.15.191:0/3734765214 shutdown_connections 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3734765214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb72810af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3734765214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728102ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3734765214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb728103cd0 0x7fb728104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- 172.21.15.191:0/3734765214 >> 172.21.15.191:0/3734765214 conn(0x7fb7280fe090 msgr2=0x7fb7281004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- 172.21.15.191:0/3734765214 shutdown_connections 2024-04-26T01:39:22.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- 172.21.15.191:0/3734765214 wait complete. 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 Processor -- start 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- start start 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb728103cd0 0x7fb7281990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb7281a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810e5c0 con 0x7fb728103cd0 2024-04-26T01:39:22.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.892+0000 7fb72ff29700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810e440 con 0x7fb728104670 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72ff29700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810e740 con 0x7fb728102ad0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:37226/0 (socket says 172.21.15.191:37226) 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 -- 172.21.15.191:0/3394003975 learned_addr learned my addr 172.21.15.191:0/3394003975 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72e4c6700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb7281a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 -- 172.21.15.191:0/3394003975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 msgr2=0x7fb7281a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb7281a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 -- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb728103cd0 msgr2=0x7fb7281990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb728103cd0 0x7fb7281990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 -- 172.21.15.191:0/3394003975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7281a0810 con 0x7fb728102ad0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72e4c6700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb7281a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72dcc5700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb724003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb724027070 con 0x7fb728102ad0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7281a0aa0 con 0x7fb728102ad0 2024-04-26T01:39:22.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72400bd90 con 0x7fb728102ad0 2024-04-26T01:39:22.893 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72400f460 con 0x7fb728102ad0 2024-04-26T01:39:22.893 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.893+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7281a0ea0 con 0x7fb728102ad0 2024-04-26T01:39:22.893 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.894+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb72802df80 con 0x7fb728102ad0 2024-04-26T01:39:22.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.895+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb72402a070 con 0x7fb728102ad0 2024-04-26T01:39:22.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.896+0000 7fb71effd700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 0x7fb7140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:22.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.896+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6723+0+0 (secure 0 0 0) 0x7fb7240d0e90 con 0x7fb728102ad0 2024-04-26T01:39:22.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.896+0000 7fb72d4c4700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 0x7fb7140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:22.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.897+0000 7fb72d4c4700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 0x7fb7140a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7180097b0 tx=0x7fb718006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:22.897 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:22.900+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb724013070 con 0x7fb728102ad0 2024-04-26T01:39:22.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:22 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 4272M 2024-04-26T01:39:22.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:22 smithi183 ceph-mon[32752]: osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] boot 2024-04-26T01:39:22.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:22 smithi183 ceph-mon[32752]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:39:22.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:22.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:22 smithi183 ceph-mon[32752]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:22 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 4272M 2024-04-26T01:39:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:22 smithi052 ceph-mon[27851]: osd.10 [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] boot 2024-04-26T01:39:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:22 smithi052 ceph-mon[27851]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T01:39:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:39:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:22 smithi052 ceph-mon[27851]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T01:39:23.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:23.107+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb728062e80 con 0x7fb7140a3c40 2024-04-26T01:39:23.890 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:23 smithi191 ceph-mon[32745]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:39:23.891 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:23 smithi191 ceph-mon[32745]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:23.891 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:23 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:23.891 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:23 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:23.891 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:23 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:23.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:23 smithi183 ceph-mon[32752]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:39:23.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:23 smithi183 ceph-mon[32752]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:23.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:23 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:23.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:23 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:23.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:23 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:23 smithi052 ceph-mon[27851]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T01:39:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:23 smithi052 ceph-mon[27851]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:39:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:23 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T01:39:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:23 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T01:39:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:23 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:24.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:24 smithi191 ceph-mon[32745]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:24.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:24 smithi191 ceph-mon[32745]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:39:24.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:24 smithi183 ceph-mon[32752]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:24.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:24 smithi183 ceph-mon[32752]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:39:24.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:24 smithi052 ceph-mon[27851]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:24.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:24 smithi052 ceph-mon[27851]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T01:39:25.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:25 smithi183 ceph-mon[32752]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:39:25.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:25 smithi052 ceph-mon[27851]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:39:26.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:25 smithi191 ceph-mon[32745]: osdmap e79: 11 total, 11 up, 11 in 2024-04-26T01:39:26.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:26.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: osdmap e80: 11 total, 11 up, 11 in 2024-04-26T01:39:26.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/1003338034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:26.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:26.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]': finished 2024-04-26T01:39:26.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:39:26.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:26.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:26 smithi052 ceph-mon[27851]: from='client.? 172.21.15.191:0/2015805941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: osdmap e80: 11 total, 11 up, 11 in 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/1003338034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]': finished 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:27.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:26 smithi191 ceph-mon[32745]: from='client.? 172.21.15.191:0/2015805941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: osdmap e80: 11 total, 11 up, 11 in 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/1003338034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]: dispatch 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c795e48-0fbd-4ff6-b778-f57b2a07a4fd"}]': finished 2024-04-26T01:39:27.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T01:39:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:26 smithi183 ceph-mon[32752]: from='client.? 172.21.15.191:0/2015805941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T01:39:29.123 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:28 smithi052 ceph-mon[27851]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:29.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:28 smithi191 ceph-mon[32745]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:29.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:28 smithi183 ceph-mon[32752]: pgmap v272: 1 pgs: 1 remapped+peering; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:31.112 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:30 smithi191 ceph-mon[32745]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:31.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:30 smithi183 ceph-mon[32752]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:31.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:30 smithi052 ceph-mon[27851]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.098 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:32 smithi191 ceph-mon[32745]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:32 smithi183 ceph-mon[32752]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-26T01:39:33.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:32 smithi052 ceph-mon[27851]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-26T01:39:34.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:39:34.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:34.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:33 smithi191 ceph-mon[32745]: Deploying daemon osd.11 on smithi191 2024-04-26T01:39:34.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:39:34.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:34.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:33 smithi183 ceph-mon[32752]: Deploying daemon osd.11 on smithi191 2024-04-26T01:39:34.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T01:39:34.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:34.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:33 smithi052 ceph-mon[27851]: Deploying daemon osd.11 on smithi191 2024-04-26T01:39:35.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:34 smithi191 ceph-mon[32745]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:35.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:34 smithi183 ceph-mon[32752]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:35.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:34 smithi052 ceph-mon[27851]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T01:39:37.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:36 smithi191 ceph-mon[32745]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:39:37.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:36 smithi183 ceph-mon[32752]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:39:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:36 smithi052 ceph-mon[27851]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T01:39:38.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:38 smithi191 ceph-mon[32745]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T01:39:39.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:38 smithi183 ceph-mon[32752]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T01:39:39.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:38 smithi052 ceph-mon[27851]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T01:39:40.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:40.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:40.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:40.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:41.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:40 smithi191 ceph-mon[32745]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T01:39:41.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:40 smithi183 ceph-mon[32752]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T01:39:41.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:40 smithi052 ceph-mon[27851]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T01:39:42.800 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.803+0000 7fb71effd700 1 -- 172.21.15.191:0/3394003975 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb728062e80 con 0x7fb7140a3c40 2024-04-26T01:39:42.801 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2024-04-26T01:39:42.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 msgr2=0x7fb7140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:42.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 0x7fb7140a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7180097b0 tx=0x7fb718006d20 comp rx=0 tx=0).stop 2024-04-26T01:39:42.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 msgr2=0x7fb728198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:42.802 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb724003c00 comp rx=0 tx=0).stop 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 shutdown_connections 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb7140a3c40 0x7fb7140a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb728104670 0x7fb7281a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb728102ad0 0x7fb728198bb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 --2- 172.21.15.191:0/3394003975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb728103cd0 0x7fb7281990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.805+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 >> 172.21.15.191:0/3394003975 conn(0x7fb7280fe090 msgr2=0x7fb728106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.806+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 shutdown_connections 2024-04-26T01:39:42.803 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:39:42.806+0000 7fb72ff29700 1 -- 172.21.15.191:0/3394003975 wait complete. 2024-04-26T01:39:43.104 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:43.104 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.105 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:42 smithi191 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:42 smithi052 ceph-mon[27851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:43.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:42 smithi183 ceph-mon[32752]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T01:39:43.426 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T01:39:43.429 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T01:39:43.429 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T01:39:43.903 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:44.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:44 smithi191 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:45.089 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:45.089 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:39:45.089 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:45.089 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:39:45.090 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:45.090 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:44 smithi052 ceph-mon[27851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:45.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:45.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T01:39:45.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: from='osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:45.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: osdmap e82: 12 total, 11 up, 12 in 2024-04-26T01:39:45.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:45.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:44 smithi183 ceph-mon[32752]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-04-26T01:39:45.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:45 smithi052 ceph-mon[27851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:39:45.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:45 smithi052 ceph-mon[27851]: osdmap e83: 12 total, 11 up, 12 in 2024-04-26T01:39:45.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:45.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:45 smithi052 ceph-mon[27851]: from='osd.11 ' entity='osd.11' 2024-04-26T01:39:45.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:45 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:46.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 -- 172.21.15.52:0/3410940701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 msgr2=0x7f8454075920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 --2- 172.21.15.52:0/3410940701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f8454075920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f843c0098f0 tx=0x7f843c009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 -- 172.21.15.52:0/3410940701 shutdown_connections 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 --2- 172.21.15.52:0/3410940701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f845410f090 0x7f8454111480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 --2- 172.21.15.52:0/3410940701 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8454076a70 0x7f8454074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 --2- 172.21.15.52:0/3410940701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f8454075920 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 -- 172.21.15.52:0/3410940701 >> 172.21.15.52:0/3410940701 conn(0x7f84540fe010 msgr2=0x7f8454100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 -- 172.21.15.52:0/3410940701 shutdown_connections 2024-04-26T01:39:46.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.131+0000 7f845b200700 1 -- 172.21.15.52:0/3410940701 wait complete. 2024-04-26T01:39:46.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 Processor -- start 2024-04-26T01:39:46.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 -- start start 2024-04-26T01:39:46.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f845419f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:46.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 0x7f845419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:46.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 0x7f84541a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8454114bc0 con 0x7f845410f090 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8454114a40 con 0x7f84540754a0 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.133+0000 7f845b200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8454114d40 con 0x7f8454076a70 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f8459200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 0x7f84541a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f8459200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 0x7f84541a6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40636/0 (socket says 172.21.15.52:40636) 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f8459200700 1 -- 172.21.15.52:0/4265675641 learned_addr learned my addr 172.21.15.52:0/4265675641 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 0x7f845419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f8453e00700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f845419f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 -- 172.21.15.52:0/4265675641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 msgr2=0x7f845419f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f845419f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 -- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 msgr2=0x7f84541a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 0x7f84541a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.134+0000 7f844b400700 1 -- 172.21.15.52:0/4265675641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84541a6f00 con 0x7f8454076a70 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f844b400700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 0x7f845419f7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f843c000c00 tx=0x7f843c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c025070 con 0x7f8454076a70 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f843c005670 con 0x7f8454076a70 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84541a7190 con 0x7f8454076a70 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c003ae0 con 0x7f8454076a70 2024-04-26T01:39:46.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.135+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84541a75e0 con 0x7f8454076a70 2024-04-26T01:39:46.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.136+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8454069730 con 0x7f8454076a70 2024-04-26T01:39:46.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.138+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f843c028070 con 0x7f8454076a70 2024-04-26T01:39:46.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.140+0000 7f8451600700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 0x7f84380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:46.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.140+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 6847+0+0 (secure 0 0 0) 0x7f843c0d0850 con 0x7f8454076a70 2024-04-26T01:39:46.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.140+0000 7f8453e00700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 0x7f84380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:46.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.141+0000 7f8453e00700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 0x7f84380a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8444009e60 tx=0x7f844400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:46.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.142+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f843c013030 con 0x7f8454076a70 2024-04-26T01:39:46.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:39:46.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-mon[32745]: osdmap e83: 12 total, 11 up, 12 in 2024-04-26T01:39:46.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:46.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-mon[32745]: from='osd.11 ' entity='osd.11' 2024-04-26T01:39:46.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:46.151 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T01:39:45.685+0000 7f6365bcc700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T01:39:46.151 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 01:39:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T01:39:45.839+0000 7f635d9b7700 -1 osd.11 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:39:46.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:45 smithi183 ceph-mon[32752]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-04-26T01:39:46.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:45 smithi183 ceph-mon[32752]: osdmap e83: 12 total, 11 up, 12 in 2024-04-26T01:39:46.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:46.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:45 smithi183 ceph-mon[32752]: from='osd.11 ' entity='osd.11' 2024-04-26T01:39:46.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:45 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:46.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.372+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8454063650 con 0x7f8454076a70 2024-04-26T01:39:46.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.373+0000 7f8451600700 1 -- 172.21.15.52:0/4265675641 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f843c09d7e0 con 0x7f8454076a70 2024-04-26T01:39:46.370 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:39:46.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 msgr2=0x7f84380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 0x7f84380a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8444009e60 tx=0x7f844400b040 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 msgr2=0x7f845419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 0x7f845419f7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f843c000c00 tx=0x7f843c005270 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 shutdown_connections 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f84380a3c40 0x7f84380a6100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f845410f090 0x7f84541a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84540754a0 0x7f845419f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 --2- 172.21.15.52:0/4265675641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8454076a70 0x7f845419f7e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.376+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 >> 172.21.15.52:0/4265675641 conn(0x7f84540fe010 msgr2=0x7f8454102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.377+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 shutdown_connections 2024-04-26T01:39:46.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:46.377+0000 7f845b200700 1 -- 172.21.15.52:0/4265675641 wait complete. 2024-04-26T01:39:47.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: purged_snaps scrub starts 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: purged_snaps scrub ok 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4265675641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: purged_snaps scrub starts 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: purged_snaps scrub ok 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4265675641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:47.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: purged_snaps scrub starts 2024-04-26T01:39:47.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: purged_snaps scrub ok 2024-04-26T01:39:47.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:47.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4265675641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:39:47.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:47.203 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":83,"num_osds":12,"num_up_osds":11,"osd_up_since":1714095561,"num_in_osds":12,"osd_in_since":1714095565,"num_remapped_pgs":0} 2024-04-26T01:39:48.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:47 smithi052 ceph-mon[27851]: Detected new or changed devices on smithi191 2024-04-26T01:39:48.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:47 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-26T01:39:48.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:47 smithi052 ceph-mon[27851]: osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] boot 2024-04-26T01:39:48.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:47 smithi052 ceph-mon[27851]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:39:48.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:48.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:47 smithi191 ceph-mon[32745]: Detected new or changed devices on smithi191 2024-04-26T01:39:48.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:47 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-26T01:39:48.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:47 smithi191 ceph-mon[32745]: osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] boot 2024-04-26T01:39:48.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:47 smithi191 ceph-mon[32745]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:39:48.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:48.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:47 smithi183 ceph-mon[32752]: Detected new or changed devices on smithi191 2024-04-26T01:39:48.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:47 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 4272M 2024-04-26T01:39:48.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:47 smithi183 ceph-mon[32752]: osd.11 [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] boot 2024-04-26T01:39:48.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:47 smithi183 ceph-mon[32752]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T01:39:48.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:39:48.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T01:39:48.647 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:49.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:48 smithi052 ceph-mon[27851]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:49.112 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:48 smithi052 ceph-mon[27851]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:39:49.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:48 smithi191 ceph-mon[32745]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:49.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:48 smithi191 ceph-mon[32745]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:39:49.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:48 smithi183 ceph-mon[32752]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-26T01:39:49.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:48 smithi183 ceph-mon[32752]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 -- 172.21.15.52:0/1055700014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac0773f0 msgr2=0x7facac077810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 --2- 172.21.15.52:0/1055700014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac0773f0 0x7facac077810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0098f0 tx=0x7fac9c009c00 comp rx=0 tx=0).stop 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 -- 172.21.15.52:0/1055700014 shutdown_connections 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 --2- 172.21.15.52:0/1055700014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac079880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 --2- 172.21.15.52:0/1055700014 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac0773f0 0x7facac077810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 --2- 172.21.15.52:0/1055700014 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac078640 0x7facac078ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 -- 172.21.15.52:0/1055700014 >> 172.21.15.52:0/1055700014 conn(0x7facac100290 msgr2=0x7facac1026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 -- 172.21.15.52:0/1055700014 shutdown_connections 2024-04-26T01:39:50.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.764+0000 7facb2800700 1 -- 172.21.15.52:0/1055700014 wait complete. 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.765+0000 7facb2800700 1 Processor -- start 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.765+0000 7facb2800700 1 -- start start 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.765+0000 7facb2800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 0x7facac19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facb2800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac078640 0x7facac19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facb2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facb2800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7facac114ba0 con 0x7facac079000 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facb2800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7facac114a20 con 0x7facac078640 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facb2800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7facac114d20 con 0x7facac0773f0 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facabe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 0x7facac19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facabe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40646/0 (socket says 172.21.15.52:40646) 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facabe00700 1 -- 172.21.15.52:0/3669105621 learned_addr learned my addr 172.21.15.52:0/3669105621 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:39:50.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facaaa00700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac078640 0x7facac19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:50.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 -- 172.21.15.52:0/3669105621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac078640 msgr2=0x7facac19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:50.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac078640 0x7facac19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:50.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 -- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 msgr2=0x7facac1a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facab400700 1 -- 172.21.15.52:0/3669105621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac1a6ee0 con 0x7facac0773f0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.766+0000 7facabe00700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7facab400700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 0x7facac19f280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c025070 con 0x7facac0773f0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac9c005670 con 0x7facac0773f0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c003ae0 con 0x7facac0773f0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac1a7170 con 0x7facac0773f0 2024-04-26T01:39:50.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.767+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac1a75c0 con 0x7facac0773f0 2024-04-26T01:39:50.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.768+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facac069730 con 0x7facac0773f0 2024-04-26T01:39:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.770+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fac9c028070 con 0x7facac0773f0 2024-04-26T01:39:50.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.772+0000 7faca3e00700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 0x7fac900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:50.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.772+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fac9c0d06f0 con 0x7facac0773f0 2024-04-26T01:39:50.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.772+0000 7facaaa00700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 0x7fac900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:50.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.773+0000 7facaaa00700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 0x7fac900a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fac98009c60 tx=0x7fac98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:50.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:50.776+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac9c05cff0 con 0x7facac0773f0 2024-04-26T01:39:50.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:50 smithi052 ceph-mon[27851]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:51.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.011+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7facac063680 con 0x7facac0773f0 2024-04-26T01:39:51.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.012+0000 7faca3e00700 1 -- 172.21.15.52:0/3669105621 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v85) ==== 74+0+133 (secure 0 0 0) 0x7fac9c013030 con 0x7facac0773f0 2024-04-26T01:39:51.008 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 msgr2=0x7fac900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 0x7fac900a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fac98009c60 tx=0x7fac98009260 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 msgr2=0x7facac19f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 0x7facac19f280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac9c000c00 tx=0x7fac9c005270 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 shutdown_connections 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fac900a3c40 0x7fac900a6100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facac079000 0x7facac1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facac0773f0 0x7facac19f280 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 --2- 172.21.15.52:0/3669105621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7facac078640 0x7facac19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 >> 172.21.15.52:0/3669105621 conn(0x7facac100290 msgr2=0x7facac101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 shutdown_connections 2024-04-26T01:39:51.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:51.015+0000 7facb2800700 1 -- 172.21.15.52:0/3669105621 wait complete. 2024-04-26T01:39:51.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:50 smithi191 ceph-mon[32745]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:51.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:50 smithi183 ceph-mon[32752]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:51.728 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":85,"num_osds":12,"num_up_osds":12,"osd_up_since":1714095586,"num_in_osds":12,"osd_in_since":1714095565,"num_remapped_pgs":0} 2024-04-26T01:39:51.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:39:51.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:51 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3669105621' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:52.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:51 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3669105621' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:52.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:51 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3669105621' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T01:39:52.188 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:52.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:52 smithi052 ceph-mon[27851]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:53.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:52 smithi191 ceph-mon[32745]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:53.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:52 smithi183 ceph-mon[32752]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:54.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 -- 172.21.15.52:0/13416609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8075590 msgr2=0x7f05c810ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 --2- 172.21.15.52:0/13416609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8075590 0x7f05c810ef00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009990 tx=0x7f05b0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 -- 172.21.15.52:0/13416609 shutdown_connections 2024-04-26T01:39:54.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 --2- 172.21.15.52:0/13416609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 0x7f05c8111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 --2- 172.21.15.52:0/13416609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8074c70 0x7f05c8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 --2- 172.21.15.52:0/13416609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8075590 0x7f05c810ef00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 -- 172.21.15.52:0/13416609 >> 172.21.15.52:0/13416609 conn(0x7f05c8079ba0 msgr2=0x7f05c8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:54.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 -- 172.21.15.52:0/13416609 shutdown_connections 2024-04-26T01:39:54.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.281+0000 7f05cec00700 1 -- 172.21.15.52:0/13416609 wait complete. 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.282+0000 7f05cec00700 1 Processor -- start 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.282+0000 7f05cec00700 1 -- start start 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8075590 0x7f05c819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 0x7f05c819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f05c8114e50 con 0x7f05c8074c70 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05c8114cd0 con 0x7f05c810f440 2024-04-26T01:39:54.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05cec00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f05c8114fd0 con 0x7f05c8075590 2024-04-26T01:39:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05c7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05c7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40664/0 (socket says 172.21.15.52:40664) 2024-04-26T01:39:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05c7e00700 1 -- 172.21.15.52:0/2884801964 learned_addr learned my addr 172.21.15.52:0/2884801964 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:39:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05bf400700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8075590 0x7f05c819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:54.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.283+0000 7f05ccc00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 0x7f05c819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 -- 172.21.15.52:0/2884801964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8075590 msgr2=0x7f05c819b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8075590 0x7f05c819b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 -- 172.21.15.52:0/2884801964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 msgr2=0x7f05c819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 0x7f05c819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 -- 172.21.15.52:0/2884801964 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c81a0300 con 0x7f05c8074c70 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c7e00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e9a0 tx=0x7f05b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b800ca10 con 0x7f05c8074c70 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b800cbb0 con 0x7f05c8074c70 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.284+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c81a05f0 con 0x7f05c8074c70 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.285+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8018680 con 0x7f05c8074c70 2024-04-26T01:39:54.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.285+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c81a7e70 con 0x7f05c8074c70 2024-04-26T01:39:54.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.286+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c8069730 con 0x7f05c8074c70 2024-04-26T01:39:54.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.288+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f05b8010430 con 0x7f05c8074c70 2024-04-26T01:39:54.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.290+0000 7f05c5600700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 0x7f05ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:54.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.290+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f05b80d0450 con 0x7f05c8074c70 2024-04-26T01:39:54.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.290+0000 7f05bf400700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 0x7f05ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:54.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.291+0000 7f05bf400700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 0x7f05ac0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05c8111100 tx=0x7f05b000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:54.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.293+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05b809d230 con 0x7f05c8074c70 2024-04-26T01:39:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.531+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05c802e0a0 con 0x7f05c8074c70 2024-04-26T01:39:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.532+0000 7f05c5600700 1 -- 172.21.15.52:0/2884801964 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+18452 (secure 0 0 0) 0x7f05b809cb30 con 0x7f05c8074c70 2024-04-26T01:39:54.530 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:39:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":85,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:39:47.813981+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 msgr2=0x7f05ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 0x7f05ac0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05c8111100 tx=0x7f05b000b260 comp rx=0 tx=0).stop 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 msgr2=0x7f05c819b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e9a0 tx=0x7f05b800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 shutdown_connections 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f05ac0a3c40 0x7f05ac0a6100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05c810f440 0x7f05c819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f05c8074c70 0x7f05c819b340 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 --2- 172.21.15.52:0/2884801964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f05c8075590 0x7f05c819b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 >> 172.21.15.52:0/2884801964 conn(0x7f05c8079ba0 msgr2=0x7f05c80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 shutdown_connections 2024-04-26T01:39:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:54.536+0000 7f05cec00700 1 -- 172.21.15.52:0/2884801964 wait complete. 2024-04-26T01:39:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:54 smithi191 ceph-mon[32745]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:54 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2884801964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:54 smithi183 ceph-mon[32752]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:54 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2884801964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:54 smithi052 ceph-mon[27851]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:54 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2884801964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:39:55.297 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:35:33.675964+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T01:39:55.297 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T01:39:55.763 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:39:57.020 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:56 smithi052 ceph-mon[27851]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:57.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:56 smithi191 ceph-mon[32745]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:56 smithi183 ceph-mon[32752]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:57.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 -- 172.21.15.52:0/3657180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 msgr2=0x7f5744115de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 --2- 172.21.15.52:0/3657180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f5744115de0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f573c00b0a0 tx=0x7f573c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 -- 172.21.15.52:0/3657180 shutdown_connections 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 --2- 172.21.15.52:0/3657180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f5744115de0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 --2- 172.21.15.52:0/3657180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f574410f030 0x7f574410f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 --2- 172.21.15.52:0/3657180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5744108f80 0x7f5744109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 -- 172.21.15.52:0/3657180 >> 172.21.15.52:0/3657180 conn(0x7f57440787d0 msgr2=0x7f5744107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 -- 172.21.15.52:0/3657180 shutdown_connections 2024-04-26T01:39:57.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.866+0000 7f5748e00700 1 -- 172.21.15.52:0/3657180 wait complete. 2024-04-26T01:39:57.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.867+0000 7f5748e00700 1 Processor -- start 2024-04-26T01:39:57.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 -- start start 2024-04-26T01:39:57.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5744108f80 0x7f574419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f574419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 0x7f57441a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5744119170 con 0x7f5744109940 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5744118ff0 con 0x7f5744108f80 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5748e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f57441192f0 con 0x7f574410f030 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5742a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f574419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5743e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 0x7f57441a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5742a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f574419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57278/0 (socket says 172.21.15.52:57278) 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5742a00700 1 -- 172.21.15.52:0/1823978227 learned_addr learned my addr 172.21.15.52:0/1823978227 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.868+0000 7f5743400700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5744108f80 0x7f574419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 -- 172.21.15.52:0/1823978227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5744108f80 msgr2=0x7f574419f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5744108f80 0x7f574419f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 -- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 msgr2=0x7f574419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:57.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f574419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:57.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 -- 172.21.15.52:0/1823978227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57441a4570 con 0x7f574410f030 2024-04-26T01:39:57.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f5743e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 0x7f57441a3e30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f573c00b070 tx=0x7f573c004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:57.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.869+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c021070 con 0x7f574410f030 2024-04-26T01:39:57.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.870+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c007e30 con 0x7f574410f030 2024-04-26T01:39:57.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.870+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57441a4800 con 0x7f574410f030 2024-04-26T01:39:57.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.870+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c0045f0 con 0x7f574410f030 2024-04-26T01:39:57.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.870+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57441ac0e0 con 0x7f574410f030 2024-04-26T01:39:57.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.871+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5744111380 con 0x7f574410f030 2024-04-26T01:39:57.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.873+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f573c007980 con 0x7f574410f030 2024-04-26T01:39:57.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.874+0000 7f573be00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 0x7f57280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:39:57.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.874+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f573c0d0cf0 con 0x7f574410f030 2024-04-26T01:39:57.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.874+0000 7f5743400700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 0x7f57280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:39:57.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.875+0000 7f5743400700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 0x7f57280a6180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5734009c60 tx=0x7f5734009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:39:57.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:57.877+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f573c09db60 con 0x7f574410f030 2024-04-26T01:39:58.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.106+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f57441a08c0 con 0x7f574410f030 2024-04-26T01:39:58.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.107+0000 7f573be00700 1 -- 172.21.15.52:0/1823978227 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f573c09d460 con 0x7f574410f030 2024-04-26T01:39:58.104 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-04-26T01:39:58.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 msgr2=0x7f57280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:58.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 0x7f57280a6180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5734009c60 tx=0x7f5734009260 comp rx=0 tx=0).stop 2024-04-26T01:39:58.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 msgr2=0x7f57441a3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:39:58.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 0x7f57441a3e30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f573c00b070 tx=0x7f573c004c80 comp rx=0 tx=0).stop 2024-04-26T01:39:58.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 shutdown_connections 2024-04-26T01:39:58.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f57280a3cc0 0x7f57280a6180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f574410f030 0x7f57441a3e30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5744108f80 0x7f574419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 --2- 172.21.15.52:0/1823978227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5744109940 0x7f574419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 >> 172.21.15.52:0/1823978227 conn(0x7f57440787d0 msgr2=0x7f57440799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 shutdown_connections 2024-04-26T01:39:58.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:39:58.110+0000 7f5748e00700 1 -- 172.21.15.52:0/1823978227 wait complete. 2024-04-26T01:39:58.903 INFO:tasks.cephadm:Adding mds.a on smithi052 2024-04-26T01:39:58.903 INFO:tasks.cephadm:Adding mds.d on smithi052 2024-04-26T01:39:58.903 INFO:tasks.cephadm:Adding mds.g on smithi052 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.j on smithi052 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.b on smithi183 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.e on smithi183 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.h on smithi183 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.k on smithi183 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.c on smithi191 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.f on smithi191 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.i on smithi191 2024-04-26T01:39:58.904 INFO:tasks.cephadm:Adding mds.l on smithi191 2024-04-26T01:39:58.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l' 2024-04-26T01:39:59.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:58 smithi191 ceph-mon[32745]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:59.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:39:58 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1823978227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:39:59.154 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T01:39:59.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:58 smithi183 ceph-mon[32752]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:59.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:39:58 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1823978227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:39:59.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:58 smithi052 ceph-mon[27851]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:39:59.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:39:58 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1823978227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T01:40:00.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 -- 172.21.15.191:0/4182875001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 msgr2=0x7f82ac103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/4182875001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac103000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8298009990 tx=0x7f8298009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 -- 172.21.15.191:0/4182875001 shutdown_connections 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/4182875001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac103540 0x7f82ac109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/4182875001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82ac1019a0 0x7f82ac101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/4182875001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac103000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 -- 172.21.15.191:0/4182875001 >> 172.21.15.191:0/4182875001 conn(0x7f82ac0fcf40 msgr2=0x7f82ac0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.875+0000 7f82b2ca4700 1 -- 172.21.15.191:0/4182875001 shutdown_connections 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 -- 172.21.15.191:0/4182875001 wait complete. 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 Processor -- start 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 -- start start 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac1019a0 0x7f82ac06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82ac103540 0x7f82ac19ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10d500 con 0x7f82ac103540 2024-04-26T01:40:00.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10d380 con 0x7f82ac1019a0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b2ca4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10d680 con 0x7f82ac102ba0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82abfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82abfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:42764/0 (socket says 172.21.15.191:42764) 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82abfff700 1 -- 172.21.15.191:0/1690740410 learned_addr learned my addr 172.21.15.191:0/1690740410 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.876+0000 7f82b0a40700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac1019a0 0x7f82ac06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 -- 172.21.15.191:0/1690740410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac1019a0 msgr2=0x7f82ac06bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac1019a0 0x7f82ac06bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 -- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82ac103540 msgr2=0x7f82ac19ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82ac103540 0x7f82ac19ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 -- 172.21.15.191:0/1690740410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ac1a08b0 con 0x7f82ac102ba0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82abfff700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82980036f0 tx=0x7f82980057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298027070 con 0x7f82ac102ba0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8298004180 con 0x7f82ac102ba0 2024-04-26T01:40:00.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ac1a0b40 con 0x7f82ac102ba0 2024-04-26T01:40:00.877 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829800f460 con 0x7f82ac102ba0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.877+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ac1a0ff0 con 0x7f82ac102ba0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.878+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82ac069730 con 0x7f82ac102ba0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.879+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f829802a070 con 0x7f82ac102ba0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.880+0000 7f82a9ffb700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 0x7f829c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.880+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f82980d1050 con 0x7f82ac102ba0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.880+0000 7f82b0a40700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 0x7f829c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:00.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.881+0000 7f82b0a40700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 0x7f829c0a6180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f82ac102a00 tx=0x7f82a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:00.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:00.884+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f829805eff0 con 0x7f82ac102ba0 2024-04-26T01:40:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:00 smithi191 ceph-mon[32745]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:00.900 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:00 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T01:40:01.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.094+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}) -- 0x7f82ac063140 con 0x7f829c0a3cc0 2024-04-26T01:40:01.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.112+0000 7f82a9ffb700 1 -- 172.21.15.191:0/1690740410 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f82ac063140 con 0x7f829c0a3cc0 2024-04-26T01:40:01.109 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mds.all update... 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 msgr2=0x7f829c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 0x7f829c0a6180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f82ac102a00 tx=0x7f82a0009190 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 msgr2=0x7f82ac19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82980036f0 tx=0x7f82980057c0 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 shutdown_connections 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f829c0a3cc0 0x7f829c0a6180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82ac103540 0x7f82ac19ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82ac1019a0 0x7f82ac06bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 --2- 172.21.15.191:0/1690740410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82ac102ba0 0x7f82ac19d9a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 >> 172.21.15.191:0/1690740410 conn(0x7f82ac0fcf40 msgr2=0x7f82ac105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:01.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 shutdown_connections 2024-04-26T01:40:01.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-04-26T01:40:01.114+0000 7f82b2ca4700 1 -- 172.21.15.191:0/1690740410 wait complete. 2024-04-26T01:40:01.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:00 smithi183 ceph-mon[32752]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:01.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:00 smithi183 ceph-mon[32752]: overall HEALTH_OK 2024-04-26T01:40:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:00 smithi052 ceph-mon[27851]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:00 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T01:40:01.806 DEBUG:teuthology.orchestra.run.smithi052:mds.a> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T01:40:01.813 DEBUG:teuthology.orchestra.run.smithi052:mds.d> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T01:40:01.859 DEBUG:teuthology.orchestra.run.smithi052:mds.g> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T01:40:01.906 DEBUG:teuthology.orchestra.run.smithi052:mds.j> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T01:40:01.951 DEBUG:teuthology.orchestra.run.smithi183:mds.b> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T01:40:01.953 DEBUG:teuthology.orchestra.run.smithi183:mds.e> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T01:40:01.996 DEBUG:teuthology.orchestra.run.smithi183:mds.h> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T01:40:02.040 DEBUG:teuthology.orchestra.run.smithi183:mds.k> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T01:40:02.082 DEBUG:teuthology.orchestra.run.smithi191:mds.c> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T01:40:02.085 DEBUG:teuthology.orchestra.run.smithi191:mds.f> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T01:40:02.128 DEBUG:teuthology.orchestra.run.smithi191:mds.i> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T01:40:02.170 DEBUG:teuthology.orchestra.run.smithi191:mds.l> sudo journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T01:40:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:02 smithi052 ceph-mon[27851]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:02 smithi052 ceph-mon[27851]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-26T01:40:02.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:02.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:02 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:02.213 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T01:40:02.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:02.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:02 smithi191 ceph-mon[32745]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:02.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:02 smithi191 ceph-mon[32745]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-26T01:40:02.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:02.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:02 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:02.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:02 smithi183 ceph-mon[32752]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:40:02.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:02 smithi183 ceph-mon[32752]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-26T01:40:02.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:02.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:02 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:02.690 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.323 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:03 smithi191 ceph-mon[32745]: Deploying daemon mds.l on smithi191 2024-04-26T01:40:03.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:03.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:03 smithi183 ceph-mon[32752]: Deploying daemon mds.l on smithi191 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:03 smithi052 ceph-mon[27851]: Deploying daemon mds.l on smithi191 2024-04-26T01:40:04.651 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:04 smithi191 systemd[1]: Starting Ceph mds.l for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:04.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.900+0000 7fe918c00700 1 -- 172.21.15.52:0/1415454196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 msgr2=0x7fe914111a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:04.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.900+0000 7fe918c00700 1 --2- 172.21.15.52:0/1415454196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe914111a60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b0d0 tx=0x7fe90c00b3e0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 -- 172.21.15.52:0/1415454196 shutdown_connections 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 --2- 172.21.15.52:0/1415454196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe914111a60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 --2- 172.21.15.52:0/1415454196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914104db0 0x7fe914105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 --2- 172.21.15.52:0/1415454196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914105760 0x7fe9141097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 -- 172.21.15.52:0/1415454196 >> 172.21.15.52:0/1415454196 conn(0x7fe914100620 msgr2=0x7fe914102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 -- 172.21.15.52:0/1415454196 shutdown_connections 2024-04-26T01:40:04.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.901+0000 7fe918c00700 1 -- 172.21.15.52:0/1415454196 wait complete. 2024-04-26T01:40:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.902+0000 7fe918c00700 1 Processor -- start 2024-04-26T01:40:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.902+0000 7fe918c00700 1 -- start start 2024-04-26T01:40:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914104db0 0x7fe91419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe91419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe914114d70 con 0x7fe914109e90 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe914114bf0 con 0x7fe914104db0 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe918c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe914114ef0 con 0x7fe914105760 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe912a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe912a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34518/0 (socket says 172.21.15.52:34518) 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe912a00700 1 -- 172.21.15.52:0/2716219984 learned_addr learned my addr 172.21.15.52:0/2716219984 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe913e00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe91419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.903+0000 7fe913400700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914104db0 0x7fe91419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 -- 172.21.15.52:0/2716219984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914104db0 msgr2=0x7fe91419b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914104db0 0x7fe91419b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 -- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 msgr2=0x7fe91419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe91419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:04.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 -- 172.21.15.52:0/2716219984 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9141a01d0 con 0x7fe914105760 2024-04-26T01:40:04.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe913e00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe91419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:04.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe912a00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe90000d980 tx=0x7fe90000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:04.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900017070 con 0x7fe914105760 2024-04-26T01:40:04.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe90000fd40 con 0x7fe914105760 2024-04-26T01:40:04.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90000e3b0 con 0x7fe914105760 2024-04-26T01:40:04.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.904+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9141a04c0 con 0x7fe914105760 2024-04-26T01:40:04.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.905+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9141a7d40 con 0x7fe914105760 2024-04-26T01:40:04.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.906+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe914069730 con 0x7fe914105760 2024-04-26T01:40:04.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.908+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe900010610 con 0x7fe914105760 2024-04-26T01:40:04.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.909+0000 7fe90be00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 0x7fe8f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:04.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.910+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fe9000cf9d0 con 0x7fe914105760 2024-04-26T01:40:04.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.910+0000 7fe913400700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 0x7fe8f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:04.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.910+0000 7fe913400700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 0x7fe8f80a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe904009680 tx=0x7fe904009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:04.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:04.913+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe90009c7b0 con 0x7fe914105760 2024-04-26T01:40:04.990 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:04 smithi191 podman[62688]: 2024-04-26 01:40:04.632956667 +0000 UTC m=+0.017720101 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:04.990 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:04 smithi191 podman[62688]: 2024-04-26 01:40:04.775314553 +0000 UTC m=+0.160077974 container create dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:40:04.990 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:04 smithi191 podman[62688]: 2024-04-26 01:40:04.994441682 +0000 UTC m=+0.379205113 container init dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T01:40:05.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.209+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe91402df80 con 0x7fe914105760 2024-04-26T01:40:05.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.210+0000 7fe90be00700 1 -- 172.21.15.52:0/2716219984 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fe90009c0b0 con 0x7fe914105760 2024-04-26T01:40:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:05.207 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 msgr2=0x7fe8f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 0x7fe8f80a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe904009680 tx=0x7fe904009260 comp rx=0 tx=0).stop 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 msgr2=0x7fe91419b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe90000d980 tx=0x7fe90000dc90 comp rx=0 tx=0).stop 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 shutdown_connections 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe8f80a3c40 0x7fe8f80a6100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:05.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe914109e90 0x7fe91419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe914104db0 0x7fe91419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 --2- 172.21.15.52:0/2716219984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe914105760 0x7fe91419b700 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 >> 172.21.15.52:0/2716219984 conn(0x7fe914100620 msgr2=0x7fe9141028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.213+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 shutdown_connections 2024-04-26T01:40:05.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:05.214+0000 7fe918c00700 1 -- 172.21.15.52:0/2716219984 wait complete. 2024-04-26T01:40:05.212 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 2 2024-04-26T01:40:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mon[32745]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6832/4135915621,v1:172.21.15.191:6833/4135915621] up:boot 2024-04-26T01:40:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mon[32745]: fsmap 1 up:standby 2024-04-26T01:40:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:05.402 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:04 smithi191 podman[62688]: 2024-04-26 01:40:04.997208929 +0000 UTC m=+0.381972351 container start dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T01:40:05.402 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-mds[62708]: -- 172.21.15.191:0/815075899 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638f4dc1a40 con 0x5638f4d94c00 2024-04-26T01:40:05.402 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l[62704]: starting mds.l at 2024-04-26T01:40:05.402 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:05 smithi191 bash[62688]: dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 2024-04-26T01:40:05.402 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 01:40:05 smithi191 systemd[1]: Started Ceph mds.l for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:05.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:05 smithi183 ceph-mon[32752]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:05.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:05 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6832/4135915621,v1:172.21.15.191:6833/4135915621] up:boot 2024-04-26T01:40:05.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:05 smithi183 ceph-mon[32752]: fsmap 1 up:standby 2024-04-26T01:40:05.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:05.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:05.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:05 smithi052 ceph-mon[27851]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:05.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:05 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6832/4135915621,v1:172.21.15.191:6833/4135915621] up:boot 2024-04-26T01:40:05.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:05 smithi052 ceph-mon[27851]: fsmap 1 up:standby 2024-04-26T01:40:05.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:05.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: Deploying daemon mds.k on smithi183 2024-04-26T01:40:06.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:06 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2716219984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:06.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:06.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:06.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:06.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: Deploying daemon mds.k on smithi183 2024-04-26T01:40:06.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:06 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2716219984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:06.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: Deploying daemon mds.k on smithi183 2024-04-26T01:40:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:06 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2716219984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:07.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:07 smithi183 ceph-mon[32752]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:07.424 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 systemd[1]: Starting Ceph mds.k for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:07 smithi052 ceph-mon[27851]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:07 smithi191 ceph-mon[32745]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 podman[62476]: 2024-04-26 01:40:07.451106942 +0000 UTC m=+0.018213457 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 podman[62476]: 2024-04-26 01:40:07.557733011 +0000 UTC m=+0.124839554 container create b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 podman[62476]: 2024-04-26 01:40:07.759740865 +0000 UTC m=+0.326847363 container init b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 podman[62476]: 2024-04-26 01:40:07.762303126 +0000 UTC m=+0.329409632 container start b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 ceph-mds[62495]: -- 172.21.15.183:0/2253569745 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7bb93fa40 con 0x55f7bb912c00 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k[62491]: starting mds.k at 2024-04-26T01:40:07.848 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 bash[62476]: b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 2024-04-26T01:40:07.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:08.173 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 01:40:07 smithi183 systemd[1]: Started Ceph mds.k for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:08.334 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:09.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: Deploying daemon mds.j on smithi052 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6832/2084061199,v1:172.21.15.183:6833/2084061199] up:boot 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: fsmap 2 up:standby 2024-04-26T01:40:09.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:08 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: Deploying daemon mds.j on smithi052 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6832/2084061199,v1:172.21.15.183:6833/2084061199] up:boot 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: fsmap 2 up:standby 2024-04-26T01:40:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:08 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: Deploying daemon mds.j on smithi052 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6832/2084061199,v1:172.21.15.183:6833/2084061199] up:boot 2024-04-26T01:40:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: fsmap 2 up:standby 2024-04-26T01:40:09.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:08 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:40:10.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:10 smithi052 ceph-mon[27851]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 -- 172.21.15.52:0/730985351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa0075590 msgr2=0x7f9fa010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/730985351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa0075590 0x7f9fa010ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009990 tx=0x7f9f8c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 -- 172.21.15.52:0/730985351 shutdown_connections 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/730985351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa010f440 0x7f9fa0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/730985351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/730985351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa0075590 0x7f9fa010ef00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 -- 172.21.15.52:0/730985351 >> 172.21.15.52:0/730985351 conn(0x7f9fa0079ba0 msgr2=0x7f9fa0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:11.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 -- 172.21.15.52:0/730985351 shutdown_connections 2024-04-26T01:40:11.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.160+0000 7f9fa7e00700 1 -- 172.21.15.52:0/730985351 wait complete. 2024-04-26T01:40:11.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.161+0000 7f9fa7e00700 1 Processor -- start 2024-04-26T01:40:11.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.161+0000 7f9fa7e00700 1 -- start start 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa0075590 0x7f9fa019b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0114e70 con 0x7f9fa0074c70 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0114cf0 con 0x7f9fa010f440 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa7e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0114ff0 con 0x7f9fa0075590 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa5400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa5e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa5400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43064/0 (socket says 172.21.15.52:43064) 2024-04-26T01:40:11.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa5e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42552/0 (socket says 172.21.15.52:42552) 2024-04-26T01:40:11.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa5400700 1 -- 172.21.15.52:0/2471740395 learned_addr learned my addr 172.21.15.52:0/2471740395 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:11.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.162+0000 7f9fa4a00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa0075590 0x7f9fa019b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:11.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 -- 172.21.15.52:0/2471740395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa0075590 msgr2=0x7f9fa019b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa0075590 0x7f9fa019b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 -- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 msgr2=0x7f9fa019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 -- 172.21.15.52:0/2471740395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa01a0320 con 0x7f9fa010f440 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa5e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e940 tx=0x7f9f9800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98004220 con 0x7f9fa010f440 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f980043c0 con 0x7f9fa010f440 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98010700 con 0x7f9fa010f440 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa01a0610 con 0x7f9fa010f440 2024-04-26T01:40:11.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.163+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa01a7e90 con 0x7f9fa010f440 2024-04-26T01:40:11.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.165+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa0069730 con 0x7f9fa010f440 2024-04-26T01:40:11.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.166+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f98004750 con 0x7f9fa010f440 2024-04-26T01:40:11.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.168+0000 7f9f9e000700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 0x7f9f880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:11.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.168+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f9f980cfcb0 con 0x7f9fa010f440 2024-04-26T01:40:11.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.168+0000 7f9fa5400700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 0x7f9f880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:11.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.169+0000 7f9fa5400700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 0x7f9f880a6180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009c10 tx=0x7f9f9400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:11.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.172+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f9809ca90 con 0x7f9fa010f440 2024-04-26T01:40:11.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:10 smithi183 ceph-mon[32752]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:11.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:10 smithi191 ceph-mon[32745]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:11.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.460+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fa002df80 con 0x7f9fa010f440 2024-04-26T01:40:11.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.461+0000 7f9f9e000700 1 -- 172.21.15.52:0/2471740395 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f9f9801fe10 con 0x7f9fa010f440 2024-04-26T01:40:11.458 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:11.458 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 msgr2=0x7f9f880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 0x7f9f880a6180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009c10 tx=0x7f9f9400b040 comp rx=0 tx=0).stop 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 msgr2=0x7f9fa019fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e940 tx=0x7f9f9800ec50 comp rx=0 tx=0).stop 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 shutdown_connections 2024-04-26T01:40:11.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f9f880a3cc0 0x7f9f880a6180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fa010f440 0x7f9fa019fbe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fa0074c70 0x7f9fa019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 --2- 172.21.15.52:0/2471740395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fa0075590 0x7f9fa019b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 >> 172.21.15.52:0/2471740395 conn(0x7f9fa0079ba0 msgr2=0x7f9fa00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 shutdown_connections 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:11.464+0000 7f9fa7e00700 1 -- 172.21.15.52:0/2471740395 wait complete. 2024-04-26T01:40:11.462 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 3 2024-04-26T01:40:12.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:11 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2471740395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:12.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:11 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2471740395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:12.192 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:11 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2471740395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:12.912 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:12 smithi052 podman[68790]: 2024-04-26 01:40:12.642226818 +0000 UTC m=+0.141328326 container create 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:40:12.912 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:12 smithi052 podman[68790]: 2024-04-26 01:40:12.914923855 +0000 UTC m=+0.414025370 container init 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-26T01:40:13.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:12 smithi183 ceph-mon[32752]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:12 smithi052 ceph-mon[27851]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:13.210 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:12 smithi052 podman[68790]: 2024-04-26 01:40:12.92652976 +0000 UTC m=+0.425631253 container start 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T01:40:13.211 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:12 smithi052 ceph-mds[68809]: -- 172.21.15.52:0/542243782 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b42fe5da40 con 0x55b42fe30800 2024-04-26T01:40:13.211 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:12 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j[68804]: starting mds.j at 2024-04-26T01:40:13.211 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:13 smithi052 bash[68790]: 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 2024-04-26T01:40:13.211 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 01:40:13 smithi052 systemd[1]: Started Ceph mds.j for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:13.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:12 smithi191 ceph-mon[32745]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:14.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:14.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: Deploying daemon mds.d on smithi052 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6834/1730690276,v1:172.21.15.52:6835/1730690276] up:boot 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: fsmap 3 up:standby 2024-04-26T01:40:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:14 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: Deploying daemon mds.d on smithi052 2024-04-26T01:40:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6834/1730690276,v1:172.21.15.52:6835/1730690276] up:boot 2024-04-26T01:40:14.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: fsmap 3 up:standby 2024-04-26T01:40:14.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:14 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: Deploying daemon mds.d on smithi052 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6834/1730690276,v1:172.21.15.52:6835/1730690276] up:boot 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: fsmap 3 up:standby 2024-04-26T01:40:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:14 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:40:15.384 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:15 smithi052 ceph-mon[27851]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:15.650 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:15.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:15 smithi191 ceph-mon[32745]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:15.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:15 smithi183 ceph-mon[32752]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:17.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:17 smithi052 ceph-mon[27851]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:17.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:17 smithi191 ceph-mon[32745]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:17.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:17 smithi183 ceph-mon[32752]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:17.811 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:17 smithi052 podman[69211]: 2024-04-26 01:40:17.813268022 +0000 UTC m=+0.343474375 container create 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T01:40:18.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.190+0000 7fe791800700 1 -- 172.21.15.52:0/3859273576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c108a30 msgr2=0x7fe78c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.190+0000 7fe791800700 1 --2- 172.21.15.52:0/3859273576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c108a30 0x7fe78c108e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00b0a0 tx=0x7fe77c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 -- 172.21.15.52:0/3859273576 shutdown_connections 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 --2- 172.21.15.52:0/3859273576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06b0f0 0x7fe78c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 --2- 172.21.15.52:0/3859273576 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c108a30 0x7fe78c108e10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 --2- 172.21.15.52:0/3859273576 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c06a750 0x7fe78c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 -- 172.21.15.52:0/3859273576 >> 172.21.15.52:0/3859273576 conn(0x7fe78c0788f0 msgr2=0x7fe78c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 -- 172.21.15.52:0/3859273576 shutdown_connections 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.191+0000 7fe791800700 1 -- 172.21.15.52:0/3859273576 wait complete. 2024-04-26T01:40:18.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 Processor -- start 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 -- start start 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 0x7fe78c199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c108a30 0x7fe78c19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c112c30 con 0x7fe78c06a750 2024-04-26T01:40:18.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c112ab0 con 0x7fe78c06b0f0 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.192+0000 7fe791800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c112db0 con 0x7fe78c108a30 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43866/0 (socket says 172.21.15.52:43866) 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe782000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 0x7fe78c199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 -- 172.21.15.52:0/1594663162 learned_addr learned my addr 172.21.15.52:0/1594663162 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78b400700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c108a30 0x7fe78c19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 -- 172.21.15.52:0/1594663162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c108a30 msgr2=0x7fe78c19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c108a30 0x7fe78c19da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 -- 172.21.15.52:0/1594663162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 msgr2=0x7fe78c199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 0x7fe78c199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 -- 172.21.15.52:0/1594663162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c19e1d0 con 0x7fe78c06a750 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe782000700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 0x7fe78c199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.193+0000 7fe78aa00700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00d010 tx=0x7fe77c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:18.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.194+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c021070 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.194+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe77c007e30 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.194+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78c19e460 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.194+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c004640 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.194+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe78c101940 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.195+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe78c069730 con 0x7fe78c06a750 2024-04-26T01:40:18.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.197+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe77c007980 con 0x7fe78c06a750 2024-04-26T01:40:18.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.198+0000 7fe783e00700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 0x7fe7700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:18.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.198+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fe77c00f020 con 0x7fe78c06a750 2024-04-26T01:40:18.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.199+0000 7fe782000700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 0x7fe7700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:18.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.199+0000 7fe782000700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 0x7fe7700a6180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe78c078fd0 tx=0x7fe774009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:18.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.203+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe77c02a020 con 0x7fe78c06a750 2024-04-26T01:40:18.324 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 podman[69211]: 2024-04-26 01:40:18.273224839 +0000 UTC m=+0.803431223 container init 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T01:40:18.324 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 podman[69211]: 2024-04-26 01:40:18.282230319 +0000 UTC m=+0.812436668 container start 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T01:40:18.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.500+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe78c02df80 con 0x7fe78c06a750 2024-04-26T01:40:18.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.501+0000 7fe783e00700 1 -- 172.21.15.52:0/1594663162 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fe77c004000 con 0x7fe78c06a750 2024-04-26T01:40:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:18.499 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T01:40:18.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 msgr2=0x7fe7700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 0x7fe7700a6180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe78c078fd0 tx=0x7fe774009260 comp rx=0 tx=0).stop 2024-04-26T01:40:18.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 msgr2=0x7fe78c1991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:18.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00d010 tx=0x7fe77c004d00 comp rx=0 tx=0).stop 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 shutdown_connections 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe7700a3cc0 0x7fe7700a6180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe78c108a30 0x7fe78c19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe78c06a750 0x7fe78c1991c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 --2- 172.21.15.52:0/1594663162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe78c06b0f0 0x7fe78c199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 >> 172.21.15.52:0/1594663162 conn(0x7fe78c0788f0 msgr2=0x7fe78c0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 shutdown_connections 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:18.504+0000 7fe791800700 1 -- 172.21.15.52:0/1594663162 wait complete. 2024-04-26T01:40:18.502 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 4 2024-04-26T01:40:18.577 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 ceph-mds[69259]: -- 172.21.15.52:0/2048707622 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb4fbe1a40 con 0x55bb4fbb4800 2024-04-26T01:40:18.578 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d[69232]: starting mds.d at 2024-04-26T01:40:18.578 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 bash[69211]: 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 2024-04-26T01:40:18.578 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 01:40:18 smithi052 systemd[1]: Started Ceph mds.d for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:19.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:19.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1594663162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:19.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:19.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:19.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:19 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:19.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:19.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1594663162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:19 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1594663162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:19 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:20.515 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:20 smithi183 ceph-mon[32752]: Deploying daemon mds.e on smithi183 2024-04-26T01:40:20.515 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:20 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6836/2681513710,v1:172.21.15.52:6837/2681513710] up:boot 2024-04-26T01:40:20.515 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:20 smithi183 ceph-mon[32752]: fsmap 4 up:standby 2024-04-26T01:40:20.515 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:20 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:20 smithi191 ceph-mon[32745]: Deploying daemon mds.e on smithi183 2024-04-26T01:40:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:20 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6836/2681513710,v1:172.21.15.52:6837/2681513710] up:boot 2024-04-26T01:40:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:20 smithi191 ceph-mon[32745]: fsmap 4 up:standby 2024-04-26T01:40:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:20 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:20.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:20 smithi052 ceph-mon[27851]: Deploying daemon mds.e on smithi183 2024-04-26T01:40:20.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:20 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6836/2681513710,v1:172.21.15.52:6837/2681513710] up:boot 2024-04-26T01:40:20.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:20 smithi052 ceph-mon[27851]: fsmap 4 up:standby 2024-04-26T01:40:20.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:20 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:40:20.915 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:20 smithi183 systemd[1]: Starting Ceph mds.e for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:21.166 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:20 smithi183 podman[62800]: 2024-04-26 01:40:20.821350915 +0000 UTC m=+0.018139634 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:21.166 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:20 smithi183 podman[62800]: 2024-04-26 01:40:20.934270487 +0000 UTC m=+0.131059230 container create 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-26T01:40:21.166 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 podman[62800]: 2024-04-26 01:40:21.153476671 +0000 UTC m=+0.350265371 container init 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T01:40:21.166 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 podman[62800]: 2024-04-26 01:40:21.156932548 +0000 UTC m=+0.353721245 container start 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:40:21.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:21.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:21 smithi183 ceph-mon[32752]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:21.424 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 ceph-mds[62820]: -- 172.21.15.183:0/887675610 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d50a6e9a40 con 0x55d50a6bcc00 2024-04-26T01:40:21.424 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e[62816]: starting mds.e at 2024-04-26T01:40:21.424 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 bash[62800]: 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e 2024-04-26T01:40:21.424 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 01:40:21 smithi183 systemd[1]: Started Ceph mds.e for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:21.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:21 smithi191 ceph-mon[32745]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:21.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:21 smithi052 ceph-mon[27851]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:21.730 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6834/2708532468,v1:172.21.15.183:6835/2708532468] up:boot 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: fsmap 5 up:standby 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:22.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:22.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:22 smithi191 ceph-mon[32745]: Deploying daemon mds.f on smithi191 2024-04-26T01:40:22.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6834/2708532468,v1:172.21.15.183:6835/2708532468] up:boot 2024-04-26T01:40:22.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: fsmap 5 up:standby 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:22 smithi183 ceph-mon[32752]: Deploying daemon mds.f on smithi191 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6834/2708532468,v1:172.21.15.183:6835/2708532468] up:boot 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: fsmap 5 up:standby 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:40:22.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:22.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:22.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:22.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:22.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:22 smithi052 ceph-mon[27851]: Deploying daemon mds.f on smithi191 2024-04-26T01:40:23.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:23 smithi052 ceph-mon[27851]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:23.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:23 smithi191 ceph-mon[32745]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:23.651 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:23 smithi191 systemd[1]: Starting Ceph mds.f for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:23.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:23 smithi183 ceph-mon[32752]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:23.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 -- 172.21.15.52:0/3587174866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 msgr2=0x7fb0bc108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 --2- 172.21.15.52:0/3587174866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 0x7fb0bc108d50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009990 tx=0x7fb0ac009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 -- 172.21.15.52:0/3587174866 shutdown_connections 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 --2- 172.21.15.52:0/3587174866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc103330 0x7fb0bc10f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 --2- 172.21.15.52:0/3587174866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 0x7fb0bc108d50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.942+0000 7fb0bb200700 1 --2- 172.21.15.52:0/3587174866 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc102970 0x7fb0bc102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.943+0000 7fb0bb200700 1 -- 172.21.15.52:0/3587174866 >> 172.21.15.52:0/3587174866 conn(0x7fb0bc0fe4c0 msgr2=0x7fb0bc1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:23.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.943+0000 7fb0bb200700 1 -- 172.21.15.52:0/3587174866 shutdown_connections 2024-04-26T01:40:23.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.943+0000 7fb0bb200700 1 -- 172.21.15.52:0/3587174866 wait complete. 2024-04-26T01:40:23.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.944+0000 7fb0bb200700 1 Processor -- start 2024-04-26T01:40:23.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.944+0000 7fb0bb200700 1 -- start start 2024-04-26T01:40:23.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 0x7fb0bc19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc112bb0 con 0x7fb0bc103330 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc112a30 con 0x7fb0bc102970 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0bb200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc112d30 con 0x7fb0bc108970 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0b9400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0b9e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0b9400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43898/0 (socket says 172.21.15.52:43898) 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0b9400700 1 -- 172.21.15.52:0/1895464636 learned_addr learned my addr 172.21.15.52:0/1895464636 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.945+0000 7fb0b9e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42614/0 (socket says 172.21.15.52:42614) 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 -- 172.21.15.52:0/1895464636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 msgr2=0x7fb0bc19d950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 0x7fb0bc19d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 -- 172.21.15.52:0/1895464636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 msgr2=0x7fb0bc199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:23.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 -- 172.21.15.52:0/1895464636 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc19e090 con 0x7fb0bc103330 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9e00700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b9400700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800e9a0 tx=0x7fb0a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800cb50 con 0x7fb0bc103330 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc19e380 con 0x7fb0bc103330 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a800ccf0 con 0x7fb0bc103330 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.946+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8018680 con 0x7fb0bc103330 2024-04-26T01:40:23.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.947+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc1a5c00 con 0x7fb0bc103330 2024-04-26T01:40:23.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.948+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc10af00 con 0x7fb0bc103330 2024-04-26T01:40:23.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.950+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb0a8010430 con 0x7fb0bc103330 2024-04-26T01:40:23.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.952+0000 7fb0b2a00700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 0x7fb0a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:23.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.953+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fb0a80d0460 con 0x7fb0bc103330 2024-04-26T01:40:23.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.953+0000 7fb0b9e00700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 0x7fb0a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:23.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.954+0000 7fb0b9e00700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 0x7fb0a00a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b2e0 tx=0x7fb0ac01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:23.956+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0a809d240 con 0x7fb0bc103330 2024-04-26T01:40:24.035 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:23 smithi191 podman[62989]: 2024-04-26 01:40:23.673811336 +0000 UTC m=+0.017782502 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:24.035 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:23 smithi191 podman[62989]: 2024-04-26 01:40:23.778269445 +0000 UTC m=+0.122240600 container create 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:40:24.035 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 podman[62989]: 2024-04-26 01:40:24.039372757 +0000 UTC m=+0.383343906 container init 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T01:40:24.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.238+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0bc19a3e0 con 0x7fb0bc103330 2024-04-26T01:40:24.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.239+0000 7fb0b2a00700 1 -- 172.21.15.52:0/1895464636 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7fb0a809cb40 con 0x7fb0bc103330 2024-04-26T01:40:24.236 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:24.237 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.242+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 msgr2=0x7fb0a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.242+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 0x7fb0a00a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b2e0 tx=0x7fb0ac01a040 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.242+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 msgr2=0x7fb0bc1995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.242+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800e9a0 tx=0x7fb0a800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 shutdown_connections 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb0a00a3c40 0x7fb0a00a6100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc108970 0x7fb0bc19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0bc102970 0x7fb0bc199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 --2- 172.21.15.52:0/1895464636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0bc103330 0x7fb0bc1995c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 >> 172.21.15.52:0/1895464636 conn(0x7fb0bc0fe4c0 msgr2=0x7fb0bc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 shutdown_connections 2024-04-26T01:40:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:24.243+0000 7fb0bb200700 1 -- 172.21.15.52:0/1895464636 wait complete. 2024-04-26T01:40:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 6 2024-04-26T01:40:24.401 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 podman[62989]: 2024-04-26 01:40:24.042015257 +0000 UTC m=+0.385986411 container start 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T01:40:24.401 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 ceph-mds[63008]: -- 172.21.15.191:0/1794197428 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b8c9cfa40 con 0x559b8c9a2c00 2024-04-26T01:40:24.401 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: starting mds.f at 2024-04-26T01:40:24.401 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 bash[62989]: 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb 2024-04-26T01:40:24.401 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 01:40:24 smithi191 systemd[1]: Started Ceph mds.f for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1895464636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: Deploying daemon mds.c on smithi191 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6834/234906181,v1:172.21.15.191:6835/234906181] up:boot 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: fsmap 6 up:standby 2024-04-26T01:40:25.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1895464636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: Deploying daemon mds.c on smithi191 2024-04-26T01:40:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6834/234906181,v1:172.21.15.191:6835/234906181] up:boot 2024-04-26T01:40:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: fsmap 6 up:standby 2024-04-26T01:40:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:25.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1895464636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: Deploying daemon mds.c on smithi191 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6834/234906181,v1:172.21.15.191:6835/234906181] up:boot 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: fsmap 6 up:standby 2024-04-26T01:40:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:40:26.844 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:26 smithi191 systemd[1]: Starting Ceph mds.c for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:27.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:27.098 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:26 smithi191 podman[63301]: 2024-04-26 01:40:26.751379195 +0000 UTC m=+0.018296020 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:27.098 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:26 smithi191 podman[63301]: 2024-04-26 01:40:26.866262527 +0000 UTC m=+0.133179377 container create f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T01:40:27.098 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 podman[63301]: 2024-04-26 01:40:27.102215358 +0000 UTC m=+0.369132164 container init f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:27 smithi191 ceph-mon[32745]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 podman[63301]: 2024-04-26 01:40:27.105028981 +0000 UTC m=+0.371945789 container start f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 ceph-mds[63320]: -- 172.21.15.191:0/291573011 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555acb9fba40 con 0x555acb9cec00 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: starting mds.c at 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 bash[63301]: f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 2024-04-26T01:40:27.401 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 01:40:27 smithi191 systemd[1]: Started Ceph mds.c for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:27.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:27 smithi052 ceph-mon[27851]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:27.573 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:27 smithi183 ceph-mon[32752]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6836/2929119914,v1:172.21.15.191:6837/2929119914] up:boot 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: fsmap 7 up:standby 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:28.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:28.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:28 smithi052 ceph-mon[27851]: Deploying daemon mds.h on smithi183 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6836/2929119914,v1:172.21.15.191:6837/2929119914] up:boot 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: fsmap 7 up:standby 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:28.641 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:28.641 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:28.641 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:28 smithi183 ceph-mon[32752]: Deploying daemon mds.h on smithi183 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6836/2929119914,v1:172.21.15.191:6837/2929119914] up:boot 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: fsmap 7 up:standby 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:28 smithi191 ceph-mon[32745]: Deploying daemon mds.h on smithi183 2024-04-26T01:40:29.442 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:29 smithi052 ceph-mon[27851]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 -- 172.21.15.52:0/2858000202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 msgr2=0x7f7c3810f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 --2- 172.21.15.52:0/2858000202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c3810f940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b0a0 tx=0x7f7c3000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 -- 172.21.15.52:0/2858000202 shutdown_connections 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 --2- 172.21.15.52:0/2858000202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c3810f940 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 --2- 172.21.15.52:0/2858000202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c38108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 --2- 172.21.15.52:0/2858000202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 -- 172.21.15.52:0/2858000202 >> 172.21.15.52:0/2858000202 conn(0x7f7c38079bf0 msgr2=0x7f7c3807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.620+0000 7f7c40400700 1 -- 172.21.15.52:0/2858000202 shutdown_connections 2024-04-26T01:40:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.621+0000 7f7c40400700 1 -- 172.21.15.52:0/2858000202 wait complete. 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 Processor -- start 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 -- start start 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c38075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c380757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:29.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38112c50 con 0x7f7c38103460 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38112ad0 con 0x7f7c38108a70 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.622+0000 7f7c40400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7c38112dd0 con 0x7f7c38102ac0 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35740/0 (socket says 172.21.15.52:35740) 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c380757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3da00700 1 -- 172.21.15.52:0/2478242312 learned_addr learned my addr 172.21.15.52:0/2478242312 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3d000700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c38075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 -- 172.21.15.52:0/2478242312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 msgr2=0x7f7c38074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 -- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 msgr2=0x7f7c38075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c38075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 -- 172.21.15.52:0/2478242312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c381a7630 con 0x7f7c38108a70 2024-04-26T01:40:29.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.623+0000 7f7c3e400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c380757c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b070 tx=0x7f7c30004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:29.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.624+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30021070 con 0x7f7c38108a70 2024-04-26T01:40:29.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.624+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c381a78c0 con 0x7f7c38108a70 2024-04-26T01:40:29.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.624+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c30007e30 con 0x7f7c38108a70 2024-04-26T01:40:29.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.624+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30004530 con 0x7f7c38108a70 2024-04-26T01:40:29.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.624+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c381a7d20 con 0x7f7c38108a70 2024-04-26T01:40:29.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.625+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c38069730 con 0x7f7c38108a70 2024-04-26T01:40:29.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.627+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7c30003c00 con 0x7f7c38108a70 2024-04-26T01:40:29.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.628+0000 7f7c36000700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 0x7f7c200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:29.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.629+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f7c300d0e30 con 0x7f7c38108a70 2024-04-26T01:40:29.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.629+0000 7f7c3da00700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 0x7f7c200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:29.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.630+0000 7f7c3da00700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 0x7f7c200a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009680 tx=0x7f7c2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.632+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c3009dca0 con 0x7f7c38108a70 2024-04-26T01:40:29.647 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:29 smithi183 ceph-mon[32752]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:29.647 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 systemd[1]: Starting Ceph mds.h for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:29.650 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:29 smithi191 ceph-mon[32745]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:29.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.914+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c3802e140 con 0x7f7c38108a70 2024-04-26T01:40:29.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.915+0000 7f7c36000700 1 -- 172.21.15.52:0/2478242312 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f7c3009d5a0 con 0x7f7c38108a70 2024-04-26T01:40:29.912 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:29.912 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.918+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 msgr2=0x7f7c200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.918+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 0x7f7c200a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009680 tx=0x7f7c2c009260 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.918+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 msgr2=0x7f7c380757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.918+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c380757c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b070 tx=0x7f7c30004d00 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.918+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 shutdown_connections 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7c200a3c40 0x7f7c200a6100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c38108a70 0x7f7c380757c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c38102ac0 0x7f7c38074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 --2- 172.21.15.52:0/2478242312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c38103460 0x7f7c38075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 >> 172.21.15.52:0/2478242312 conn(0x7f7c38079bf0 msgr2=0x7f7c38100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 shutdown_connections 2024-04-26T01:40:29.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:29.919+0000 7f7c40400700 1 -- 172.21.15.52:0/2478242312 wait complete. 2024-04-26T01:40:29.917 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 8 2024-04-26T01:40:29.920 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 podman[63113]: 2024-04-26 01:40:29.554031909 +0000 UTC m=+0.017645269 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:29.920 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 podman[63113]: 2024-04-26 01:40:29.654746208 +0000 UTC m=+0.118359601 container create afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T01:40:29.920 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 podman[63113]: 2024-04-26 01:40:29.92386354 +0000 UTC m=+0.387476890 container init afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416) 2024-04-26T01:40:30.173 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 podman[63113]: 2024-04-26 01:40:29.926822355 +0000 UTC m=+0.390435706 container start afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T01:40:30.173 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/2448746823 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df6977da40 con 0x55df69751000 2024-04-26T01:40:30.174 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T01:40:30.174 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:30 smithi183 bash[63113]: afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 2024-04-26T01:40:30.174 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:40:30 smithi183 systemd[1]: Started Ceph mds.h for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2478242312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:30 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2478242312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:30.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:30.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:30 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:30.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2478242312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:30.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:30.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:30.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:30.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:30 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:31 smithi191 ceph-mon[32745]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:31 smithi191 ceph-mon[32745]: Deploying daemon mds.i on smithi191 2024-04-26T01:40:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:31 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/1263803776,v1:172.21.15.183:6837/1263803776] up:boot 2024-04-26T01:40:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:31 smithi191 ceph-mon[32745]: fsmap 8 up:standby 2024-04-26T01:40:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:31 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:31 smithi183 ceph-mon[32752]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:31 smithi183 ceph-mon[32752]: Deploying daemon mds.i on smithi191 2024-04-26T01:40:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:31 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/1263803776,v1:172.21.15.183:6837/1263803776] up:boot 2024-04-26T01:40:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:31 smithi183 ceph-mon[32752]: fsmap 8 up:standby 2024-04-26T01:40:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:31 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:31.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:31 smithi052 ceph-mon[27851]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:31 smithi052 ceph-mon[27851]: Deploying daemon mds.i on smithi191 2024-04-26T01:40:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:31 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/1263803776,v1:172.21.15.183:6837/1263803776] up:boot 2024-04-26T01:40:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:31 smithi052 ceph-mon[27851]: fsmap 8 up:standby 2024-04-26T01:40:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:31 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:40:32.570 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 systemd[1]: Starting Ceph mds.i for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:32.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 podman[63609]: 2024-04-26 01:40:32.476608052 +0000 UTC m=+0.017539861 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 podman[63609]: 2024-04-26 01:40:32.595866036 +0000 UTC m=+0.136797873 container create 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 podman[63609]: 2024-04-26 01:40:32.772980529 +0000 UTC m=+0.313912324 container init 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 podman[63609]: 2024-04-26 01:40:32.775523182 +0000 UTC m=+0.316454985 container start 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 ceph-mds[63628]: -- 172.21.15.191:0/1992597364 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557dc025fa40 con 0x557dc0232c00 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: starting mds.i at 2024-04-26T01:40:32.853 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 bash[63609]: 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 2024-04-26T01:40:33.151 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 01:40:32 smithi191 systemd[1]: Started Ceph mds.i for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:33.296 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:33.594 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:33.594 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.594 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:33.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:33.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:33 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:33 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:33 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:34.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:34 smithi191 ceph-mon[32745]: Deploying daemon mds.g on smithi052 2024-04-26T01:40:34.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:34 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:boot 2024-04-26T01:40:34.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:34 smithi191 ceph-mon[32745]: fsmap 9 up:standby 2024-04-26T01:40:34.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:34 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:34.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:34 smithi183 ceph-mon[32752]: Deploying daemon mds.g on smithi052 2024-04-26T01:40:34.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:34 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:boot 2024-04-26T01:40:34.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:34 smithi183 ceph-mon[32752]: fsmap 9 up:standby 2024-04-26T01:40:34.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:34 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:34.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:34 smithi052 ceph-mon[27851]: Deploying daemon mds.g on smithi052 2024-04-26T01:40:34.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:34 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:boot 2024-04-26T01:40:34.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:34 smithi052 ceph-mon[27851]: fsmap 9 up:standby 2024-04-26T01:40:34.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:34 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:40:35.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:35 smithi191 ceph-mon[32745]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:35 smithi183 ceph-mon[32752]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:35 smithi052 ceph-mon[27851]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:36.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 -- 172.21.15.52:0/4248484089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4103450 msgr2=0x7f37f410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 --2- 172.21.15.52:0/4248484089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4103450 0x7f37f410f8b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009990 tx=0x7f37ec009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 -- 172.21.15.52:0/4248484089 shutdown_connections 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 --2- 172.21.15.52:0/4248484089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4103450 0x7f37f410f8b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 --2- 172.21.15.52:0/4248484089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 0x7f37f4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 --2- 172.21.15.52:0/4248484089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4102a90 0x7f37f4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 -- 172.21.15.52:0/4248484089 >> 172.21.15.52:0/4248484089 conn(0x7f37f4079ba0 msgr2=0x7f37f4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 -- 172.21.15.52:0/4248484089 shutdown_connections 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.205+0000 7f37fa400700 1 -- 172.21.15.52:0/4248484089 wait complete. 2024-04-26T01:40:36.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.206+0000 7f37fa400700 1 Processor -- start 2024-04-26T01:40:36.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 -- start start 2024-04-26T01:40:36.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:36.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4103450 0x7f37f419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:36.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 0x7f37f419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:36.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4112c50 con 0x7f37f4102a90 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4112ad0 con 0x7f37f4103450 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37fa400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4112dd0 con 0x7f37f4108a50 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37f3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37f3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59994/0 (socket says 172.21.15.52:59994) 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.207+0000 7f37f3400700 1 -- 172.21.15.52:0/1235312889 learned_addr learned my addr 172.21.15.52:0/1235312889 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3e00700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 0x7f37f419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37e3e00700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4103450 0x7f37f419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 -- 172.21.15.52:0/1235312889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 msgr2=0x7f37f419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:36.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 0x7f37f419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 -- 172.21.15.52:0/1235312889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4103450 msgr2=0x7f37f419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4103450 0x7f37f419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 -- 172.21.15.52:0/1235312889 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f41a0320 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.208+0000 7f37f3400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37e40060b0 tx=0x7f37e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.209+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e401a070 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.209+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e400dd20 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.209+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4014600 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.209+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f41a0610 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.209+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f41a7e90 con 0x7f37f4102a90 2024-04-26T01:40:36.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.210+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f4069730 con 0x7f37f4102a90 2024-04-26T01:40:36.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.212+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f37e401d070 con 0x7f37f4102a90 2024-04-26T01:40:36.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.215+0000 7f37f0c00700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 0x7f37d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:36.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.215+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f37e40cfb00 con 0x7f37f4102a90 2024-04-26T01:40:36.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.215+0000 7f37e3e00700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 0x7f37d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:36.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.216+0000 7f37e3e00700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 0x7f37d40a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f37f410fcf0 tx=0x7f37dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:36.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.218+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37e4010070 con 0x7f37f4102a90 2024-04-26T01:40:36.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.503+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37f419c670 con 0x7f37f4102a90 2024-04-26T01:40:36.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.504+0000 7f37f0c00700 1 -- 172.21.15.52:0/1235312889 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f37e409c980 con 0x7f37f4102a90 2024-04-26T01:40:36.502 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:36.502 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T01:40:36.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 msgr2=0x7f37d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:36.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 0x7f37d40a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f37f410fcf0 tx=0x7f37dc009260 comp rx=0 tx=0).stop 2024-04-26T01:40:36.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 msgr2=0x7f37f419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:36.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37e40060b0 tx=0x7f37e400d3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 shutdown_connections 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f37d40a3c40 0x7f37d40a6100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37f4108a50 0x7f37f419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f37f4102a90 0x7f37f419b310 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 --2- 172.21.15.52:0/1235312889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37f4103450 0x7f37f419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.507+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 >> 172.21.15.52:0/1235312889 conn(0x7f37f4079ba0 msgr2=0x7f37f41014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.508+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 shutdown_connections 2024-04-26T01:40:36.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:36.508+0000 7f37fa400700 1 -- 172.21.15.52:0/1235312889 wait complete. 2024-04-26T01:40:36.506 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 10 2024-04-26T01:40:37.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:37 smithi052 ceph-mon[27851]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:37.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:37 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1235312889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:37.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:37 smithi191 ceph-mon[32745]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:37.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:37 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1235312889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:37.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:37 smithi183 ceph-mon[32752]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:37.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:37 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1235312889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:38.082 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:37 smithi052 podman[70023]: 2024-04-26 01:40:37.855908499 +0000 UTC m=+0.189797714 container create 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-26T01:40:38.082 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 podman[70023]: 2024-04-26 01:40:38.084824398 +0000 UTC m=+0.418713620 container init 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T01:40:38.460 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 podman[70023]: 2024-04-26 01:40:38.099238549 +0000 UTC m=+0.433127771 container start 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T01:40:38.460 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 ceph-mds[70042]: -- 172.21.15.52:0/4202201380 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bac8757a40 con 0x55bac872a800 2024-04-26T01:40:38.460 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g[70038]: starting mds.g at 2024-04-26T01:40:38.461 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 bash[70023]: 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 2024-04-26T01:40:38.461 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 01:40:38 smithi052 systemd[1]: Started Ceph mds.g for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:39.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6838/3379340628,v1:172.21.15.52:6839/3379340628] up:boot 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: fsmap 10 up:standby 2024-04-26T01:40:39.599 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:39.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:39.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:39.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:39.600 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:39 smithi052 ceph-mon[27851]: Deploying daemon mds.a on smithi052 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6838/3379340628,v1:172.21.15.52:6839/3379340628] up:boot 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: fsmap 10 up:standby 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:39 smithi191 ceph-mon[32745]: Deploying daemon mds.a on smithi052 2024-04-26T01:40:39.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6838/3379340628,v1:172.21.15.52:6839/3379340628] up:boot 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: fsmap 10 up:standby 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:39 smithi183 ceph-mon[32752]: Deploying daemon mds.a on smithi052 2024-04-26T01:40:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:41.432 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:41 smithi052 ceph-mon[27851]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:41.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:41 smithi191 ceph-mon[32745]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:41.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:41 smithi183 ceph-mon[32752]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:41.714 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:41 smithi052 systemd[1]: Starting Ceph mds.a for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T01:40:42.612 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:42 smithi052 podman[70426]: 2024-04-26 01:40:42.254560238 +0000 UTC m=+0.037105482 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T01:40:42.613 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:42 smithi052 podman[70426]: 2024-04-26 01:40:42.373949386 +0000 UTC m=+0.156494655 container create b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T01:40:43.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 -- 172.21.15.52:0/4004713689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4444108a30 msgr2=0x7f4444108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:43.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 --2- 172.21.15.52:0/4004713689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4444108a30 0x7f4444108e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4434009990 tx=0x7f4434009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 -- 172.21.15.52:0/4004713689 shutdown_connections 2024-04-26T01:40:43.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 --2- 172.21.15.52:0/4004713689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f444410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 --2- 172.21.15.52:0/4004713689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4444108a30 0x7f4444108e10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 --2- 172.21.15.52:0/4004713689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f444406a750 0x7f444406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 -- 172.21.15.52:0/4004713689 >> 172.21.15.52:0/4004713689 conn(0x7f44440788f0 msgr2=0x7f4444078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 -- 172.21.15.52:0/4004713689 shutdown_connections 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.028+0000 7f4449200700 1 -- 172.21.15.52:0/4004713689 wait complete. 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.029+0000 7f4449200700 1 Processor -- start 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.029+0000 7f4449200700 1 -- start start 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f444406a750 0x7f4444101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444108a30 0x7f4444102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4444112bb0 con 0x7f444406a750 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4444112a30 con 0x7f444406b0f0 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4449200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4444112d30 con 0x7f4444108a30 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:43.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f444406a750 0x7f4444101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38814/0 (socket says 172.21.15.52:38814) 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 -- 172.21.15.52:0/2691406484 learned_addr learned my addr 172.21.15.52:0/2691406484 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4442800700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444108a30 0x7f4444102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 -- 172.21.15.52:0/2691406484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444108a30 msgr2=0x7f4444102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444108a30 0x7f4444102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 -- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f444406a750 msgr2=0x7f4444101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f444406a750 0x7f4444101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.030+0000 7f4441400700 1 -- 172.21.15.52:0/2691406484 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44441a75f0 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f4441400700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f443000eb30 tx=0x7f443000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443000cbd0 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44300044d0 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44441a78e0 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430003e30 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.031+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44441a7e70 con 0x7f444406b0f0 2024-04-26T01:40:43.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.032+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4444069730 con 0x7f444406b0f0 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.034+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4430004000 con 0x7f444406b0f0 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.035+0000 7f443aa00700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 0x7f44280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.036+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f44300d02d0 con 0x7f444406b0f0 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.036+0000 7f4441e00700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 0x7f44280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.037+0000 7f4441e00700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 0x7f44280a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44340098e0 tx=0x7f4434005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:43.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.039+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44300042f0 con 0x7f444406b0f0 2024-04-26T01:40:43.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:42 smithi052 podman[70426]: 2024-04-26 01:40:42.947510687 +0000 UTC m=+0.730055990 container init b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-26T01:40:43.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:42 smithi052 podman[70426]: 2024-04-26 01:40:42.957844037 +0000 UTC m=+0.740389272 container start b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T01:40:43.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mds[70463]: -- 172.21.15.52:0/18250715 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629c9d77a40 con 0x5629c9d4a800 2024-04-26T01:40:43.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: starting mds.a at 2024-04-26T01:40:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.346+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f444402df80 con 0x7f444406b0f0 2024-04-26T01:40:43.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.350+0000 7f443aa00700 1 -- 172.21.15.52:0/2691406484 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f443009d0b0 con 0x7f444406b0f0 2024-04-26T01:40:43.347 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:43.347 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-26T01:40:43.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.353+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 msgr2=0x7f44280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:43.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.353+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 0x7f44280a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44340098e0 tx=0x7f4434005960 comp rx=0 tx=0).stop 2024-04-26T01:40:43.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.353+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 msgr2=0x7f4444101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:43.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.353+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f443000eb30 tx=0x7f443000ee40 comp rx=0 tx=0).stop 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 shutdown_connections 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f44280a3cc0 0x7f44280a6180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444108a30 0x7f4444102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f444406a750 0x7f4444101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 --2- 172.21.15.52:0/2691406484 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f444406b0f0 0x7f4444101f50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 >> 172.21.15.52:0/2691406484 conn(0x7f44440788f0 msgr2=0x7f44440ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:43.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 shutdown_connections 2024-04-26T01:40:43.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:43.354+0000 7f4449200700 1 -- 172.21.15.52:0/2691406484 wait complete. 2024-04-26T01:40:43.353 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 12 2024-04-26T01:40:43.460 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:43 smithi052 bash[70426]: b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 2024-04-26T01:40:43.460 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 01:40:43 smithi052 systemd[1]: Started Ceph mds.a for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:43 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:43 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:43.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T01:40:43.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:43 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: Deploying daemon mds.b on smithi183 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2691406484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:boot 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: fsmap 11 up:standby 2024-04-26T01:40:44.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:44 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:44.608 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:44.608 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: Deploying daemon mds.b on smithi183 2024-04-26T01:40:44.609 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2691406484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:44.609 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:boot 2024-04-26T01:40:44.609 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: fsmap 11 up:standby 2024-04-26T01:40:44.609 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:44 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: Deploying daemon mds.b on smithi183 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2691406484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:boot 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: fsmap 11 up:standby 2024-04-26T01:40:44.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:44 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:40:45.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:45 smithi191 ceph-mon[32745]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:45.664 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:45 smithi183 ceph-mon[32752]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:45 smithi052 ceph-mon[27851]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:45.922 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:40:45 smithi183 podman[63435]: 2024-04-26 01:40:45.903615495 +0000 UTC m=+0.350751559 container start cef119fcec4f5825996c239f84dc0e7057227f5ef92c22cd57aaa96a542d253e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T01:40:46.173 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:40:45 smithi183 ceph-mds[63453]: -- 172.21.15.183:0/3995815788 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b691fd3a40 con 0x55b691fa7000 2024-04-26T01:40:46.174 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:40:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: starting mds.b at 2024-04-26T01:40:46.174 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:40:46 smithi183 bash[63435]: cef119fcec4f5825996c239f84dc0e7057227f5ef92c22cd57aaa96a542d253e 2024-04-26T01:40:46.174 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:40:46 smithi183 systemd[1]: Started Ceph mds.b for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T01:40:46.281 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:40:46.757 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:boot 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: fsmap 12 up:standby 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:47.323 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:47.323 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:boot 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: fsmap 12 up:standby 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:47.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:47.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:47.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.371 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.372 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:47.372 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:boot 2024-04-26T01:40:47.372 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: fsmap 12 up:standby 2024-04-26T01:40:47.372 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:40:47.372 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:49 smithi052 ceph-mon[27851]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:49 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:49 smithi191 ceph-mon[32745]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:49 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:49 smithi183 ceph-mon[32752]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:49 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:49.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 -- 172.21.15.52:0/4223965378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406b0f0 msgr2=0x7f497410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 --2- 172.21.15.52:0/4223965378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406b0f0 0x7f497410f900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b0a0 tx=0x7f496c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 -- 172.21.15.52:0/4223965378 shutdown_connections 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 --2- 172.21.15.52:0/4223965378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406b0f0 0x7f497410f900 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 --2- 172.21.15.52:0/4223965378 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f4974108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 --2- 172.21.15.52:0/4223965378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406a750 0x7f497406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 -- 172.21.15.52:0/4223965378 >> 172.21.15.52:0/4223965378 conn(0x7f49740788f0 msgr2=0x7f4974078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 -- 172.21.15.52:0/4223965378 shutdown_connections 2024-04-26T01:40:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.632+0000 7f497a200700 1 -- 172.21.15.52:0/4223965378 wait complete. 2024-04-26T01:40:49.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.633+0000 7f497a200700 1 Processor -- start 2024-04-26T01:40:49.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.633+0000 7f497a200700 1 -- start start 2024-04-26T01:40:49.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406b0f0 0x7f497419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f497419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4974112c20 con 0x7f497406a750 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4974112aa0 con 0x7f4974108a30 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f497a200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4974112da0 con 0x7f497406b0f0 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f4973400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f4973400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57776/0 (socket says 172.21.15.52:57776) 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f4972a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406b0f0 0x7f497419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f4973400700 1 -- 172.21.15.52:0/2351748139 learned_addr learned my addr 172.21.15.52:0/2351748139 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.634+0000 7f4973e00700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f497419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 -- 172.21.15.52:0/2351748139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406b0f0 msgr2=0x7f497419b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406b0f0 0x7f497419b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 -- 172.21.15.52:0/2351748139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 msgr2=0x7f497419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f497419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 -- 172.21.15.52:0/2351748139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49741a0310 con 0x7f497406a750 2024-04-26T01:40:49.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973e00700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f497419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:49.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f4973400700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f496400e9a0 tx=0x7f496400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:49.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.635+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496400ca10 con 0x7f497406a750 2024-04-26T01:40:49.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.636+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496400cbb0 con 0x7f497406a750 2024-04-26T01:40:49.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.636+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49741a0600 con 0x7f497406a750 2024-04-26T01:40:49.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.636+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964018680 con 0x7f497406a750 2024-04-26T01:40:49.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.636+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49741a7e80 con 0x7f497406a750 2024-04-26T01:40:49.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.637+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4974069730 con 0x7f497406a750 2024-04-26T01:40:49.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.639+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4964010430 con 0x7f497406a750 2024-04-26T01:40:49.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.640+0000 7f496be00700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 0x7f49580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:49.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.641+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f49640d0090 con 0x7f497406a750 2024-04-26T01:40:49.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.641+0000 7f4972a00700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 0x7f49580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:49.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.642+0000 7f4972a00700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 0x7f49580a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4974078f60 tx=0x7f4960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:49.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.650+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f496409ce70 con 0x7f497406a750 2024-04-26T01:40:49.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.940+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f497402df80 con 0x7f497406a750 2024-04-26T01:40:49.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.943+0000 7f496be00700 1 -- 172.21.15.52:0/2351748139 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7f496409c770 con 0x7f497406a750 2024-04-26T01:40:49.941 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:49.941 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24472,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T01:40:49.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.946+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 msgr2=0x7f49580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.946+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 0x7f49580a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4974078f60 tx=0x7f4960009260 comp rx=0 tx=0).stop 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.946+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 msgr2=0x7f497419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.946+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f496400e9a0 tx=0x7f496400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 shutdown_connections 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f49580a3cc0 0x7f49580a6180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4974108a30 0x7f497419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f497406a750 0x7f497419b350 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 --2- 172.21.15.52:0/2351748139 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497406b0f0 0x7f497419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 >> 172.21.15.52:0/2351748139 conn(0x7f49740788f0 msgr2=0x7f497410df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 shutdown_connections 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:49.947+0000 7f497a200700 1 -- 172.21.15.52:0/2351748139 wait complete. 2024-04-26T01:40:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 13 2024-04-26T01:40:50.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:50 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2351748139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:50.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:50 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2351748139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:50.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:50 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2351748139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:40:50.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-26T01:40:51.340 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:51.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:51 smithi191 ceph-mon[32745]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:51.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:51 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:51.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:51 smithi183 ceph-mon[32752]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:51 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:51.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:51 smithi052 ceph-mon[27851]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:51.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:51 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:52 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.402 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:52 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:52.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:52 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:40:53.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:53 smithi052 ceph-mon[27851]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:53 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:53 smithi183 ceph-mon[32752]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:53 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.462+0000 7fe3daa00700 1 -- 172.21.15.52:0/2665200763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 msgr2=0x7fe3d41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.462+0000 7fe3daa00700 1 --2- 172.21.15.52:0/2665200763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41096b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009990 tx=0x7fe3c4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 -- 172.21.15.52:0/2665200763 shutdown_connections 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 --2- 172.21.15.52:0/2665200763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d410e3b0 0x7fe3d4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 --2- 172.21.15.52:0/2665200763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41096b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 --2- 172.21.15.52:0/2665200763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d4109c80 0x7fe3d410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 -- 172.21.15.52:0/2665200763 >> 172.21.15.52:0/2665200763 conn(0x7fe3d4076210 msgr2=0x7fe3d4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 -- 172.21.15.52:0/2665200763 shutdown_connections 2024-04-26T01:40:53.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.463+0000 7fe3daa00700 1 -- 172.21.15.52:0/2665200763 wait complete. 2024-04-26T01:40:53.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.464+0000 7fe3daa00700 1 Processor -- start 2024-04-26T01:40:53.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.464+0000 7fe3daa00700 1 -- start start 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 0x7fe3d41a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 0x7fe3d419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d4119290 con 0x7fe3d4109c80 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d4119110 con 0x7fe3d41092d0 2024-04-26T01:40:53.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3daa00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d4119410 con 0x7fe3d410e3b0 2024-04-26T01:40:53.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3cbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 0x7fe3d41a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3d8a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 0x7fe3d419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3cbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 0x7fe3d41a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57792/0 (socket says 172.21.15.52:57792) 2024-04-26T01:40:53.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3cbe00700 1 -- 172.21.15.52:0/1995881718 learned_addr learned my addr 172.21.15.52:0/1995881718 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:53.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.465+0000 7fe3d3e00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 -- 172.21.15.52:0/1995881718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 msgr2=0x7fe3d41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 -- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 msgr2=0x7fe3d41a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 0x7fe3d41a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 -- 172.21.15.52:0/1995881718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d419fd10 con 0x7fe3d410e3b0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d8a00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 0x7fe3d419f5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00efc0 tx=0x7fe3cc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0092e0 con 0x7fe3d410e3b0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.466+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d419ffa0 con 0x7fe3d410e3b0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.467+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3cc004bb0 con 0x7fe3d410e3b0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.467+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc004020 con 0x7fe3d410e3b0 2024-04-26T01:40:53.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.467+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d41ac2c0 con 0x7fe3d410e3b0 2024-04-26T01:40:53.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.468+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d4069730 con 0x7fe3d410e3b0 2024-04-26T01:40:53.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.469+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe3cc01b030 con 0x7fe3d410e3b0 2024-04-26T01:40:53.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.471+0000 7fe3d1600700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 0x7fe3b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:53.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.471+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fe3cc0cfa30 con 0x7fe3d410e3b0 2024-04-26T01:40:53.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.471+0000 7fe3d3e00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 0x7fe3b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:53.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.472+0000 7fe3d3e00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 0x7fe3b80a6100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400b790 tx=0x7fe3c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:53.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.475+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3cc022020 con 0x7fe3d410e3b0 2024-04-26T01:40:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:53 smithi191 ceph-mon[32745]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:53 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:53.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.716+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe3d402df80 con 0x7fe3d410e3b0 2024-04-26T01:40:53.714 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:40:53.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.717+0000 7fe3d1600700 1 -- 172.21.15.52:0/1995881718 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) ==== 84+0+65 (secure 0 0 0) 0x7fe3cc010040 con 0x7fe3d410e3b0 2024-04-26T01:40:53.714 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2024-04-26T01:40:53.714 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T01:40:53.715 INFO:teuthology.orchestra.run.smithi052.stdout: "poolnum": 1, 2024-04-26T01:40:53.715 INFO:teuthology.orchestra.run.smithi052.stdout: "poolname": ".mgr" 2024-04-26T01:40:53.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:40:53.715 INFO:teuthology.orchestra.run.smithi052.stdout:] 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 msgr2=0x7fe3b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 0x7fe3b80a6100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400b790 tx=0x7fe3c4005fd0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 msgr2=0x7fe3d419f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 0x7fe3d419f5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00efc0 tx=0x7fe3cc00c3a0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 shutdown_connections 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3b80a3c40 0x7fe3b80a6100 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3d410e3b0 0x7fe3d419f5b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3d41092d0 0x7fe3d41a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 --2- 172.21.15.52:0/1995881718 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3d4109c80 0x7fe3d41a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:53.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 >> 172.21.15.52:0/1995881718 conn(0x7fe3d4076210 msgr2=0x7fe3d4077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:53.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 shutdown_connections 2024-04-26T01:40:53.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:53.720+0000 7fe3daa00700 1 -- 172.21.15.52:0/1995881718 wait complete. 2024-04-26T01:40:54.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:54 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:54.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:54 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1995881718' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:40:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:54 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:54 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1995881718' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:40:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:54 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:54.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:54 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1995881718' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T01:40:54.868 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T01:40:54.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T01:40:55.324 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:55.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:55 smithi052 ceph-mon[27851]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:55.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:55 smithi191 ceph-mon[32745]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:55.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:55 smithi183 ceph-mon[32752]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:57.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:57.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 -- 172.21.15.52:0/364453923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc10f4f0 msgr2=0x7fd9cc1118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 --2- 172.21.15.52:0/364453923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc1118e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40098f0 tx=0x7fd9c4009c00 comp rx=0 tx=0).stop 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 -- 172.21.15.52:0/364453923 shutdown_connections 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 --2- 172.21.15.52:0/364453923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc1118e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 --2- 172.21.15.52:0/364453923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 0x7fd9cc06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 --2- 172.21.15.52:0/364453923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc06b070 0x7fd9cc10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 -- 172.21.15.52:0/364453923 >> 172.21.15.52:0/364453923 conn(0x7fd9cc0788f0 msgr2=0x7fd9cc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 -- 172.21.15.52:0/364453923 shutdown_connections 2024-04-26T01:40:57.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.497+0000 7fd9d2200700 1 -- 172.21.15.52:0/364453923 wait complete. 2024-04-26T01:40:57.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.498+0000 7fd9d2200700 1 Processor -- start 2024-04-26T01:40:57.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 -- start start 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 0x7fd9cc19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc114df0 con 0x7fd9cc06b070 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc114c70 con 0x7fd9cc06a750 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9d2200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc114f70 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9caa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9cb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 0x7fd9cc19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.499+0000 7fd9caa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34614/0 (socket says 172.21.15.52:34614) 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9caa00700 1 -- 172.21.15.52:0/2915435851 learned_addr learned my addr 172.21.15.52:0/2915435851 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:40:57.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 -- 172.21.15.52:0/2915435851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 msgr2=0x7fd9cc19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 0x7fd9cc19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 -- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 msgr2=0x7fd9cc19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 -- 172.21.15.52:0/2915435851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc1a02f0 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9caa00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.500+0000 7fd9cbe00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc19fbb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009230 tx=0x7fd9c400f930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.501+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4005040 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.501+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9c4005690 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.501+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc1a0580 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.501+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4003900 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.501+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9cc1a7e60 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.502+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9cc101f60 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.505+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd9c400b6f0 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.506+0000 7fd9c3e00700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 0x7fd9b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:40:57.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.506+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fd9c40d0df0 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.506+0000 7fd9cb400700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 0x7fd9b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:40:57.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.507+0000 7fd9cb400700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 0x7fd9b00a6100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00f540 tx=0x7fd9bc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:40:57.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.511+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9c4013030 con 0x7fd9cc10f4f0 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:57.563 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:40:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:40:57.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:57.745+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd9cc19c640 con 0x7fd9cc10f4f0 2024-04-26T01:40:58.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.212+0000 7fd9c3e00700 1 -- 172.21.15.52:0/2915435851 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) ==== 138+0+0 (secure 0 0 0) 0x7fd9c409dc60 con 0x7fd9cc10f4f0 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.219+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 msgr2=0x7fd9b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.219+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 0x7fd9b00a6100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00f540 tx=0x7fd9bc00b040 comp rx=0 tx=0).stop 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.219+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 msgr2=0x7fd9cc19fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc19fbb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009230 tx=0x7fd9c400f930 comp rx=0 tx=0).stop 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 shutdown_connections 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd9b00a3c40 0x7fd9b00a6100 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9cc10f4f0 0x7fd9cc19fbb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9cc06a750 0x7fd9cc19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 --2- 172.21.15.52:0/2915435851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9cc06b070 0x7fd9cc19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:40:58.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.220+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 >> 172.21.15.52:0/2915435851 conn(0x7fd9cc0788f0 msgr2=0x7fd9cc1109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:40:58.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.221+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 shutdown_connections 2024-04-26T01:40:58.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:40:58.221+0000 7fd9d2200700 1 -- 172.21.15.52:0/2915435851 wait complete. 2024-04-26T01:40:58.220 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_metadata' created 2024-04-26T01:40:58.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:58 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2915435851' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:58.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:58 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:58 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2915435851' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:58.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:58 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:58.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:58 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2915435851' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:58.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:58 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T01:40:59.114 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T01:40:59.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:59 smithi052 ceph-mon[27851]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:59.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:59 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:40:59.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:40:59 smithi052 ceph-mon[27851]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:40:59.584 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:40:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:59 smithi191 ceph-mon[32745]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:59 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:40:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:40:59 smithi191 ceph-mon[32745]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:40:59.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:59 smithi183 ceph-mon[32752]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:40:59.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:59 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T01:40:59.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:40:59 smithi183 ceph-mon[32752]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T01:41:00.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:00 smithi052 ceph-mon[27851]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:41:00.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:00 smithi052 ceph-mon[27851]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:41:00.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:00 smithi191 ceph-mon[32745]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:41:00.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:00 smithi191 ceph-mon[32745]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:41:00.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:00 smithi183 ceph-mon[32752]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T01:41:00.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:00 smithi183 ceph-mon[32752]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T01:41:01.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:01 smithi052 ceph-mon[27851]: pgmap v324: 33 pgs: 7 creating+peering, 23 unknown, 3 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:01 smithi052 ceph-mon[27851]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:01.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:01 smithi191 ceph-mon[32745]: pgmap v324: 33 pgs: 7 creating+peering, 23 unknown, 3 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:01 smithi191 ceph-mon[32745]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:01.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:01 smithi183 ceph-mon[32752]: pgmap v324: 33 pgs: 7 creating+peering, 23 unknown, 3 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:01 smithi183 ceph-mon[32752]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:01.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 -- 172.21.15.52:0/2428154224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 msgr2=0x7fed00102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 --2- 172.21.15.52:0/2428154224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed00102de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009990 tx=0x7fecf0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 -- 172.21.15.52:0/2428154224 shutdown_connections 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 --2- 172.21.15.52:0/2428154224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed0010f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 --2- 172.21.15.52:0/2428154224 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 0x7fed00108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 --2- 172.21.15.52:0/2428154224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed00102de0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 -- 172.21.15.52:0/2428154224 >> 172.21.15.52:0/2428154224 conn(0x7fed000fe450 msgr2=0x7fed00100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 -- 172.21.15.52:0/2428154224 shutdown_connections 2024-04-26T01:41:01.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.907+0000 7fecff200700 1 -- 172.21.15.52:0/2428154224 wait complete. 2024-04-26T01:41:01.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 Processor -- start 2024-04-26T01:41:01.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 -- start start 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed001990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 0x7fed0019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fed00112ba0 con 0x7fed00108960 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fed00112a20 con 0x7fed00102960 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.909+0000 7fecff200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fed00112d20 con 0x7fed00103320 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecfe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 0x7fed0019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecfde00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed001990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:01.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34158/0 (socket says 172.21.15.52:34158) 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 -- 172.21.15.52:0/1069307091 learned_addr learned my addr 172.21.15.52:0/1069307091 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 -- 172.21.15.52:0/1069307091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 msgr2=0x7fed001990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed001990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 -- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 msgr2=0x7fed0019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 0x7fed0019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:01.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 -- 172.21.15.52:0/1069307091 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0019e0b0 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.910+0000 7fecf5400700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000c00 tx=0x7fecf00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.911+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0027070 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.911+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0019e340 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.911+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf0022470 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.911+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0017650 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.911+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed001a5c20 con 0x7fed00103320 2024-04-26T01:41:01.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.912+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed00069730 con 0x7fed00103320 2024-04-26T01:41:01.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.914+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fecf002a070 con 0x7fed00103320 2024-04-26T01:41:01.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.915+0000 7fecf7400700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 0x7fece40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:01.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.916+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7497+0+0 (secure 0 0 0) 0x7fecf00d0d90 con 0x7fed00103320 2024-04-26T01:41:01.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.916+0000 7fecfde00700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 0x7fece40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:01.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.916+0000 7fecfde00700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 0x7fece40a6100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fece8009be0 tx=0x7fece8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:01.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:01.919+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fecf0013070 con 0x7fed00103320 2024-04-26T01:41:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.154+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fed0002df80 con 0x7fed00103320 2024-04-26T01:41:02.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.251+0000 7fecf7400700 1 -- 172.21.15.52:0/1069307091 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v89) ==== 170+0+0 (secure 0 0 0) 0x7fecf009da90 con 0x7fed00103320 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 msgr2=0x7fece40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 0x7fece40a6100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fece8009be0 tx=0x7fece8009260 comp rx=0 tx=0).stop 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 msgr2=0x7fed00199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000c00 tx=0x7fecf00057c0 comp rx=0 tx=0).stop 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 shutdown_connections 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fece40a3c40 0x7fece40a6100 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed00108960 0x7fed0019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:02.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fed00102960 0x7fed001990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:02.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 --2- 172.21.15.52:0/1069307091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fed00103320 0x7fed00199630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:02.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.260+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 >> 172.21.15.52:0/1069307091 conn(0x7fed000fe450 msgr2=0x7fed000fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:02.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.261+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 shutdown_connections 2024-04-26T01:41:02.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:02.261+0000 7fecff200700 1 -- 172.21.15.52:0/1069307091 wait complete. 2024-04-26T01:41:02.263 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data' created 2024-04-26T01:41:02.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:02 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1069307091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:02.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:02 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:02.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:02 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1069307091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:02.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:02 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:02.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:02 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1069307091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:02.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:02 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T01:41:03.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T01:41:03.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:03 smithi052 ceph-mon[27851]: pgmap v326: 33 pgs: 7 creating+peering, 6 unknown, 20 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:03 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:41:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:03 smithi052 ceph-mon[27851]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:41:03.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:03 smithi191 ceph-mon[32745]: pgmap v326: 33 pgs: 7 creating+peering, 6 unknown, 20 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:03 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:41:03.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:03 smithi191 ceph-mon[32745]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:41:03.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:03 smithi183 ceph-mon[32752]: pgmap v326: 33 pgs: 7 creating+peering, 6 unknown, 20 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:03.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:03 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T01:41:03.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:03 smithi183 ceph-mon[32752]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T01:41:03.692 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:04.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:04 smithi052 ceph-mon[27851]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:41:04.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:04 smithi191 ceph-mon[32745]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:41:04.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:04 smithi183 ceph-mon[32752]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T01:41:05.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:05 smithi052 ceph-mon[27851]: pgmap v329: 97 pgs: 18 creating+peering, 35 unknown, 44 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:05.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:05 smithi052 ceph-mon[27851]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:41:05.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:05 smithi191 ceph-mon[32745]: pgmap v329: 97 pgs: 18 creating+peering, 35 unknown, 44 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:05.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:05 smithi191 ceph-mon[32745]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:41:05.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:05 smithi183 ceph-mon[32752]: pgmap v329: 97 pgs: 18 creating+peering, 35 unknown, 44 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:05.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:05 smithi183 ceph-mon[32752]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T01:41:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 -- 172.21.15.52:0/829360046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638074c70 msgr2=0x7f8638075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 --2- 172.21.15.52:0/829360046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638074c70 0x7f8638075050 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f862c0098f0 tx=0x7f862c009c00 comp rx=0 tx=0).stop 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 -- 172.21.15.52:0/829360046 shutdown_connections 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 --2- 172.21.15.52:0/829360046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 0x7f8638111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 --2- 172.21.15.52:0/829360046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638074c70 0x7f8638075050 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 --2- 172.21.15.52:0/829360046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638075590 0x7f863810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 -- 172.21.15.52:0/829360046 >> 172.21.15.52:0/829360046 conn(0x7f8638079bf0 msgr2=0x7f863807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 -- 172.21.15.52:0/829360046 shutdown_connections 2024-04-26T01:41:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.126+0000 7f8640200700 1 -- 172.21.15.52:0/829360046 wait complete. 2024-04-26T01:41:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.127+0000 7f8640200700 1 Processor -- start 2024-04-26T01:41:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 -- start start 2024-04-26T01:41:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638075590 0x7f863819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 0x7f863819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8638114e90 con 0x7f8638074c70 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8638114d10 con 0x7f863810f440 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f8640200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8638115010 con 0x7f8638075590 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f863d800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f863d800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54272/0 (socket says 172.21.15.52:54272) 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.128+0000 7f863d800700 1 -- 172.21.15.52:0/1854595653 learned_addr learned my addr 172.21.15.52:0/1854595653 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863e200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 0x7f863819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:06.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863ce00700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638075590 0x7f863819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 -- 172.21.15.52:0/1854595653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638075590 msgr2=0x7f863819b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638075590 0x7f863819b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 -- 172.21.15.52:0/1854595653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 msgr2=0x7f863819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 0x7f863819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 -- 172.21.15.52:0/1854595653 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86381a03d0 con 0x7f8638074c70 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.129+0000 7f863d800700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f862c009230 tx=0x7f862c0094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.130+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c005140 con 0x7f8638074c70 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.130+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86381a0660 con 0x7f8638074c70 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.130+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862c00b560 con 0x7f8638074c70 2024-04-26T01:41:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.130+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c003a80 con 0x7f8638074c70 2024-04-26T01:41:06.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.130+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86381a7f40 con 0x7f8638074c70 2024-04-26T01:41:06.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.131+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8638076390 con 0x7f8638074c70 2024-04-26T01:41:06.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.133+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f862c00ba10 con 0x7f8638074c70 2024-04-26T01:41:06.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.135+0000 7f8636000700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 0x7f86200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:06.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.135+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7879+0+0 (secure 0 0 0) 0x7f862c0d1430 con 0x7f8638074c70 2024-04-26T01:41:06.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.135+0000 7f863ce00700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 0x7f86200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:06.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.136+0000 7f863ce00700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 0x7f86200a6100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8624009750 tx=0x7f8624009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:06.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.138+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f862c05eff0 con 0x7f8638074c70 2024-04-26T01:41:06.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:06.422+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f863802df80 con 0x7f8638074c70 2024-04-26T01:41:07.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.297+0000 7f8636000700 1 -- 172.21.15.52:0/1854595653 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f862c013030 con 0x7f8638074c70 2024-04-26T01:41:07.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 msgr2=0x7f86200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 0x7f86200a6100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8624009750 tx=0x7f8624009310 comp rx=0 tx=0).stop 2024-04-26T01:41:07.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 msgr2=0x7f863819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:07.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f862c009230 tx=0x7f862c0094a0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 shutdown_connections 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f86200a3c40 0x7f86200a6100 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f863810f440 0x7f863819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8638074c70 0x7f863819b3c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 --2- 172.21.15.52:0/1854595653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8638075590 0x7f863819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:07.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.303+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 >> 172.21.15.52:0/1854595653 conn(0x7f8638079bf0 msgr2=0x7f86380ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:07.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.304+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 shutdown_connections 2024-04-26T01:41:07.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:07.304+0000 7f8640200700 1 -- 172.21.15.52:0/1854595653 wait complete. 2024-04-26T01:41:07.305 INFO:teuthology.orchestra.run.smithi052.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T01:41:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:07 smithi191 ceph-mon[32745]: pgmap v331: 97 pgs: 15 creating+peering, 15 unknown, 67 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:07 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:41:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:07 smithi191 ceph-mon[32745]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T01:41:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:07 smithi183 ceph-mon[32752]: pgmap v331: 97 pgs: 15 creating+peering, 15 unknown, 67 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:07 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:41:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:07 smithi183 ceph-mon[32752]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T01:41:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:07 smithi052 ceph-mon[27851]: pgmap v331: 97 pgs: 15 creating+peering, 15 unknown, 67 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:41:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:07 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T01:41:07.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:07 smithi052 ceph-mon[27851]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T01:41:08.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:active 2024-04-26T01:41:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:08 smithi052 ceph-mon[27851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:active 2024-04-26T01:41:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:08 smithi191 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:08.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T01:41:08.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1854595653' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T01:41:08.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T01:41:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T01:41:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T01:41:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] up:active 2024-04-26T01:41:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:08 smithi183 ceph-mon[32752]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:08.690 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:09.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:09 smithi052 ceph-mon[27851]: pgmap v333: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 567 B/s wr, 1 op/s 2024-04-26T01:41:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:09 smithi052 ceph-mon[27851]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:41:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:09 smithi052 ceph-mon[27851]: Cluster is now healthy 2024-04-26T01:41:09.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:09 smithi191 ceph-mon[32745]: pgmap v333: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 567 B/s wr, 1 op/s 2024-04-26T01:41:09.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:09 smithi191 ceph-mon[32745]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:41:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:09 smithi191 ceph-mon[32745]: Cluster is now healthy 2024-04-26T01:41:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:09 smithi183 ceph-mon[32752]: pgmap v333: 97 pgs: 11 creating+peering, 86 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 567 B/s wr, 1 op/s 2024-04-26T01:41:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:09 smithi183 ceph-mon[32752]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T01:41:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:09 smithi183 ceph-mon[32752]: Cluster is now healthy 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 -- 172.21.15.52:0/2997733597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2801137f0 msgr2=0x7fd280115be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 --2- 172.21.15.52:0/2997733597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2801137f0 0x7fd280115be0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd27800b0a0 tx=0x7fd27800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 -- 172.21.15.52:0/2997733597 shutdown_connections 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 --2- 172.21.15.52:0/2997733597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2801137f0 0x7fd280115be0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 --2- 172.21.15.52:0/2997733597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280076f70 0x7fd280077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 --2- 172.21.15.52:0/2997733597 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280077890 0x7fd2801132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 -- 172.21.15.52:0/2997733597 >> 172.21.15.52:0/2997733597 conn(0x7fd280100620 msgr2=0x7fd280102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 -- 172.21.15.52:0/2997733597 shutdown_connections 2024-04-26T01:41:10.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.944+0000 7fd27f200700 1 -- 172.21.15.52:0/2997733597 wait complete. 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.945+0000 7fd27f200700 1 Processor -- start 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 -- start start 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd280076f70 0x7fd28019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 0x7fd2801a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd280119150 con 0x7fd280077890 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd280118fd0 con 0x7fd2801137f0 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27f200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd2801192d0 con 0x7fd280076f70 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.946+0000 7fd27d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54292/0 (socket says 172.21.15.52:54292) 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 -- 172.21.15.52:0/2371685988 learned_addr learned my addr 172.21.15.52:0/2371685988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27de00700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd280076f70 0x7fd28019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:10.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27e800700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 0x7fd2801a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 -- 172.21.15.52:0/2371685988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd280076f70 msgr2=0x7fd28019f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd280076f70 0x7fd28019f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 -- 172.21.15.52:0/2371685988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 msgr2=0x7fd2801a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 0x7fd2801a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 -- 172.21.15.52:0/2371685988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2801a4610 con 0x7fd280077890 2024-04-26T01:41:10.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27e800700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 0x7fd2801a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.947+0000 7fd27d400700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9c0 tx=0x7fd26c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.948+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c00cbd0 con 0x7fd280077890 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.948+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd26c00ee60 con 0x7fd280077890 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.948+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2801a4900 con 0x7fd280077890 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.948+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0055c0 con 0x7fd280077890 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.948+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2801ac180 con 0x7fd280077890 2024-04-26T01:41:10.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.949+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd280069730 con 0x7fd280077890 2024-04-26T01:41:10.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.951+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd26c004750 con 0x7fd280077890 2024-04-26T01:41:10.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.953+0000 7fd276a00700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 0x7fd2640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:10.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.953+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fd26c0d04a0 con 0x7fd280077890 2024-04-26T01:41:10.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.953+0000 7fd27de00700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 0x7fd2640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:10.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.954+0000 7fd27de00700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 0x7fd2640a6180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd270009680 tx=0x7fd270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:10.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:10.958+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd26c09cf30 con 0x7fd280077890 2024-04-26T01:41:11.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.190+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2801a0990 con 0x7fd280077890 2024-04-26T01:41:11.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.193+0000 7fd276a00700 1 -- 172.21.15.52:0/2371685988 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22215 (secure 0 0 0) 0x7fd2801a0990 con 0x7fd280077890 2024-04-26T01:41:11.191 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:41:11.191 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":92,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:06.424706+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:41:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 msgr2=0x7fd2640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 0x7fd2640a6180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd270009680 tx=0x7fd270009260 comp rx=0 tx=0).stop 2024-04-26T01:41:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 msgr2=0x7fd28019fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9c0 tx=0x7fd26c00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 shutdown_connections 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2640a3cc0 0x7fd2640a6180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2801137f0 0x7fd2801a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd280076f70 0x7fd28019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 --2- 172.21.15.52:0/2371685988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd280077890 0x7fd28019fb40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.196+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 >> 172.21.15.52:0/2371685988 conn(0x7fd280100620 msgr2=0x7fd280102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.197+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 shutdown_connections 2024-04-26T01:41:11.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:11.197+0000 7fd27f200700 1 -- 172.21.15.52:0/2371685988 wait complete. 2024-04-26T01:41:11.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:11 smithi052 ceph-mon[27851]: pgmap v334: 97 pgs: 97 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 1 op/s 2024-04-26T01:41:11.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:11 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2371685988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:11.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:11 smithi191 ceph-mon[32745]: pgmap v334: 97 pgs: 97 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 1 op/s 2024-04-26T01:41:11.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:11 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2371685988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:11.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:11 smithi183 ceph-mon[32752]: pgmap v334: 97 pgs: 97 active+clean; 578 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 1 op/s 2024-04-26T01:41:11.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:11 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2371685988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:11.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:41:12.443 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:12.673 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:41:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: 2024-04-26T01:41:12.384+0000 7f7e11b24700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T01:41:13.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:13 smithi052 ceph-mon[27851]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:41:13.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:13 smithi191 ceph-mon[32745]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:41:13.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:13 smithi183 ceph-mon[32752]: pgmap v335: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-26T01:41:14.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 -- 172.21.15.52:0/4053820021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 msgr2=0x7f866410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 --2- 172.21.15.52:0/4053820021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866410dcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f86500099c0 tx=0x7f8650009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 -- 172.21.15.52:0/4053820021 shutdown_connections 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 --2- 172.21.15.52:0/4053820021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f866410e3b0 0x7f8664115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 --2- 172.21.15.52:0/4053820021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86641092d0 0x7f86641096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 --2- 172.21.15.52:0/4053820021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866410dcd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.554+0000 7f866c000700 1 -- 172.21.15.52:0/4053820021 >> 172.21.15.52:0/4053820021 conn(0x7f8664076210 msgr2=0x7f8664107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.555+0000 7f866c000700 1 -- 172.21.15.52:0/4053820021 shutdown_connections 2024-04-26T01:41:14.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.555+0000 7f866c000700 1 -- 172.21.15.52:0/4053820021 wait complete. 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.556+0000 7f866c000700 1 Processor -- start 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.556+0000 7f866c000700 1 -- start start 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8664119290 con 0x7f866410e3b0 2024-04-26T01:41:14.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8664119110 con 0x7f86641092d0 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866c000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8664119410 con 0x7f8664109c80 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f8669600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f8668c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f8669600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51786/0 (socket says 172.21.15.52:51786) 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f866a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58358/0 (socket says 172.21.15.52:58358) 2024-04-26T01:41:14.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.557+0000 7f8669600700 1 -- 172.21.15.52:0/3863632561 learned_addr learned my addr 172.21.15.52:0/3863632561 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 -- 172.21.15.52:0/3863632561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 msgr2=0x7f866419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 -- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 msgr2=0x7f86641a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 -- 172.21.15.52:0/3863632561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86641a4730 con 0x7f86641092d0 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f866a000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8669600700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f865800ef10 tx=0x7f865800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865800c9c0 con 0x7f86641092d0 2024-04-26T01:41:14.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8658019070 con 0x7f86641092d0 2024-04-26T01:41:14.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86641a49c0 con 0x7f86641092d0 2024-04-26T01:41:14.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658014480 con 0x7f86641092d0 2024-04-26T01:41:14.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.558+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86641ac2a0 con 0x7f86641092d0 2024-04-26T01:41:14.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.560+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8664069730 con 0x7f86641092d0 2024-04-26T01:41:14.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.561+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8658012050 con 0x7f86641092d0 2024-04-26T01:41:14.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.563+0000 7f8662000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 0x7f864c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:14.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.563+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f86580d01e0 con 0x7f86641092d0 2024-04-26T01:41:14.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.563+0000 7f8668c00700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 0x7f864c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:14.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.564+0000 7f8668c00700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 0x7f864c0a6180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8650000c00 tx=0x7f8650005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:14.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.568+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8658010040 con 0x7f86641092d0 2024-04-26T01:41:14.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.795+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f866402df80 con 0x7f86641092d0 2024-04-26T01:41:14.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.797+0000 7f8662000700 1 -- 172.21.15.52:0/3863632561 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22215 (secure 0 0 0) 0x7f865809cc70 con 0x7f86641092d0 2024-04-26T01:41:14.795 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:41:14.795 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":92,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:06.424706+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:41:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 msgr2=0x7f864c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 0x7f864c0a6180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8650000c00 tx=0x7f8650005fb0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 msgr2=0x7f866419f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f865800ef10 tx=0x7f865800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 shutdown_connections 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f864c0a3cc0 0x7f864c0a6180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f866410e3b0 0x7f86641a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86641092d0 0x7f866419f720 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 --2- 172.21.15.52:0/3863632561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8664109c80 0x7f866419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.801+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 >> 172.21.15.52:0/3863632561 conn(0x7f8664076210 msgr2=0x7f8664076830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.802+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 shutdown_connections 2024-04-26T01:41:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:14.802+0000 7f866c000700 1 -- 172.21.15.52:0/3863632561 wait complete. 2024-04-26T01:41:15.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:15 smithi052 ceph-mon[27851]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-26T01:41:15.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:15 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3863632561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:15.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-26T01:41:15.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:15 smithi191 ceph-mon[32745]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-26T01:41:15.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:15 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3863632561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:15.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:15 smithi183 ceph-mon[32752]: pgmap v336: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-26T01:41:15.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:15 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3863632561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:16.072 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:17.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:17 smithi052 ceph-mon[27851]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:41:17.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:17 smithi191 ceph-mon[32745]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:41:17.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:17 smithi183 ceph-mon[32752]: pgmap v337: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-26T01:41:18.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.316+0000 7f4b9ac00700 1 -- 172.21.15.52:0/4122562141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b94108a30 msgr2=0x7f4b94108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:18.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.316+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/4122562141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b94108a30 0x7f4b94108e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009990 tx=0x7f4b84009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 -- 172.21.15.52:0/4122562141 shutdown_connections 2024-04-26T01:41:18.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/4122562141 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/4122562141 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b94108a30 0x7f4b94108e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/4122562141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b9406a750 0x7f4b9406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 -- 172.21.15.52:0/4122562141 >> 172.21.15.52:0/4122562141 conn(0x7f4b940788f0 msgr2=0x7f4b94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:18.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 -- 172.21.15.52:0/4122562141 shutdown_connections 2024-04-26T01:41:18.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.317+0000 7f4b9ac00700 1 -- 172.21.15.52:0/4122562141 wait complete. 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.318+0000 7f4b9ac00700 1 Processor -- start 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.318+0000 7f4b9ac00700 1 -- start start 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.318+0000 7f4b9ac00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b9ac00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9419b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b9ac00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b94108a30 0x7f4b9419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b9ac00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94112c00 con 0x7f4b94108a30 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b9ac00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94112a80 con 0x7f4b9406b0f0 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b9ac00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94112d80 con 0x7f4b9406a750 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b93400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9419b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b93e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:18.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b98c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b94108a30 0x7f4b9419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:18.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b93400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9419b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51798/0 (socket says 172.21.15.52:51798) 2024-04-26T01:41:18.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b93e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39624/0 (socket says 172.21.15.52:39624) 2024-04-26T01:41:18.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.319+0000 7f4b93400700 1 -- 172.21.15.52:0/3714429129 learned_addr learned my addr 172.21.15.52:0/3714429129 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 -- 172.21.15.52:0/3714429129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 msgr2=0x7f4b9419b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9419b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 -- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b94108a30 msgr2=0x7f4b9419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b94108a30 0x7f4b9419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 -- 172.21.15.52:0/3714429129 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b941a0350 con 0x7f4b9406a750 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b93e00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4b840098e0 tx=0x7f4b840057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84027070 con 0x7f4b9406a750 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b84022470 con 0x7f4b9406a750 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84017650 con 0x7f4b9406a750 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b941a05e0 con 0x7f4b9406a750 2024-04-26T01:41:18.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.320+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b941a7ec0 con 0x7f4b9406a750 2024-04-26T01:41:18.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.322+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b94104170 con 0x7f4b9406a750 2024-04-26T01:41:18.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.325+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b8402a070 con 0x7f4b9406a750 2024-04-26T01:41:18.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.327+0000 7f4b90c00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 0x7f4b780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:18.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.327+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f4b840d1080 con 0x7f4b9406a750 2024-04-26T01:41:18.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.327+0000 7f4b93400700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 0x7f4b780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.328+0000 7f4b93400700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 0x7f4b780a6180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b9410fd40 tx=0x7f4b8000b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.328+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b84013070 con 0x7f4b9406a750 2024-04-26T01:41:18.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:18.610+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4b941a8090 con 0x7f4b9406a750 2024-04-26T01:41:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.326+0000 7f4b90c00700 1 -- 172.21.15.52:0/3714429129 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f4b8409dba0 con 0x7f4b9406a750 2024-04-26T01:41:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.329+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 msgr2=0x7f4b780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.329+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 0x7f4b780a6180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b9410fd40 tx=0x7f4b8000b500 comp rx=0 tx=0).stop 2024-04-26T01:41:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.329+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 msgr2=0x7f4b9419b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.329+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4b840098e0 tx=0x7f4b840057c0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 shutdown_connections 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4b780a3cc0 0x7f4b780a6180 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4b94108a30 0x7f4b9419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b9406a750 0x7f4b9419b390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 --2- 172.21.15.52:0/3714429129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4b9406b0f0 0x7f4b9419b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 >> 172.21.15.52:0/3714429129 conn(0x7f4b940788f0 msgr2=0x7f4b940ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 shutdown_connections 2024-04-26T01:41:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:19.330+0000 7f4b9ac00700 1 -- 172.21.15.52:0/3714429129 wait complete. 2024-04-26T01:41:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:19 smithi191 ceph-mon[32745]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:41:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:19 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3714429129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:19 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:19.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:19 smithi183 ceph-mon[32752]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:41:19.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:19 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3714429129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:19.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:19 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:19 smithi052 ceph-mon[27851]: pgmap v338: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-26T01:41:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:19 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3714429129' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:19 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T01:41:20.101 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T01:41:20.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-26T01:41:20.582 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:20 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:41:20.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:20 smithi191 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:20.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:20 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:41:20.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:20 smithi183 ceph-mon[32752]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:20.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:20 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T01:41:20.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:20 smithi052 ceph-mon[27851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:21.603 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:21 smithi052 ceph-mon[27851]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T01:41:21.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:21 smithi191 ceph-mon[32745]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T01:41:21.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:21 smithi183 ceph-mon[32752]: pgmap v339: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-04-26T01:41:22.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 -- 172.21.15.52:0/227943235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc1092d0 msgr2=0x7ff1fc1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:22.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 --2- 172.21.15.52:0/227943235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1096b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00099c0 tx=0x7ff1f0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 -- 172.21.15.52:0/227943235 shutdown_connections 2024-04-26T01:41:22.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 --2- 172.21.15.52:0/227943235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc10e3b0 0x7ff1fc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 --2- 172.21.15.52:0/227943235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1096b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 --2- 172.21.15.52:0/227943235 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc109c80 0x7ff1fc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 -- 172.21.15.52:0/227943235 >> 172.21.15.52:0/227943235 conn(0x7ff1fc076210 msgr2=0x7ff1fc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 -- 172.21.15.52:0/227943235 shutdown_connections 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.897+0000 7ff203800700 1 -- 172.21.15.52:0/227943235 wait complete. 2024-04-26T01:41:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.898+0000 7ff203800700 1 Processor -- start 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.898+0000 7ff203800700 1 -- start start 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc10e3b0 0x7ff1fc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc1192c0 con 0x7ff1fc10e3b0 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc119140 con 0x7ff1fc1092d0 2024-04-26T01:41:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff203800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc119440 con 0x7ff1fc109c80 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff201800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc10e3b0 0x7ff1fc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39638/0 (socket says 172.21.15.52:39638) 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 -- 172.21.15.52:0/3653909046 learned_addr learned my addr 172.21.15.52:0/3653909046 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff200e00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 -- 172.21.15.52:0/3653909046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc1092d0 msgr2=0x7ff1fc1a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:22.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 -- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc10e3b0 msgr2=0x7ff1fc19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc10e3b0 0x7ff1fc19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.899+0000 7ff1fbe00700 1 -- 172.21.15.52:0/3653909046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc19fd60 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff1fbe00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e970 tx=0x7ff1e800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8004a30 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e800cc90 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8005610 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc1a0050 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.900+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc1ac2c0 con 0x7ff1fc109c80 2024-04-26T01:41:22.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.902+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc069730 con 0x7ff1fc109c80 2024-04-26T01:41:22.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.903+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff1e800c7e0 con 0x7ff1fc109c80 2024-04-26T01:41:22.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.905+0000 7ff1f9600700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 0x7ff1e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:22.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.905+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7ff1e80d04e0 con 0x7ff1fc109c80 2024-04-26T01:41:22.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.905+0000 7ff200e00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 0x7ff1e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:22.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.906+0000 7ff200e00700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 0x7ff1e40a6100 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:22.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:22.909+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1e809cf70 con 0x7ff1fc109c80 2024-04-26T01:41:23.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.195+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7ff1fc02df80 con 0x7ff1fc109c80 2024-04-26T01:41:23.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.345+0000 7ff1f9600700 1 -- 172.21.15.52:0/3653909046 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7ff1e809c870 con 0x7ff1fc109c80 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.349+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 msgr2=0x7ff1e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.349+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 0x7ff1e40a6100 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0005fb0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 msgr2=0x7ff1fc1a5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e970 tx=0x7ff1e800ec80 comp rx=0 tx=0).stop 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 shutdown_connections 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1e40a3c40 0x7ff1e40a6100 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1fc10e3b0 0x7ff1fc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1fc1092d0 0x7ff1fc1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 --2- 172.21.15.52:0/3653909046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1fc109c80 0x7ff1fc1a5930 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 >> 172.21.15.52:0/3653909046 conn(0x7ff1fc076210 msgr2=0x7ff1fc077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.350+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 shutdown_connections 2024-04-26T01:41:23.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:23.351+0000 7ff203800700 1 -- 172.21.15.52:0/3653909046 wait complete. 2024-04-26T01:41:23.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:23 smithi052 ceph-mon[27851]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 2 op/s 2024-04-26T01:41:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:23 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3653909046' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:23 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:23.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:23 smithi191 ceph-mon[32745]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 2 op/s 2024-04-26T01:41:23.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:23 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3653909046' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:23.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:23 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:23.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:23 smithi183 ceph-mon[32752]: pgmap v340: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 2 op/s 2024-04-26T01:41:23.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:23 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3653909046' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:23.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:23 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T01:41:24.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-26T01:41:24.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:41:24.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:41:24.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:41:24.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:24.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T01:41:24.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:24 smithi052 ceph-mon[27851]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T01:41:24.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:41:24.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:41:24.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:41:24.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:24.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T01:41:24.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:24 smithi191 ceph-mon[32745]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T01:41:24.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T01:41:24.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T01:41:24.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T01:41:24.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T01:41:24.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T01:41:24.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:24 smithi183 ceph-mon[32752]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T01:41:24.721 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:active 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: Cluster is now healthy 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T01:41:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T01:41:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:25 smithi191 ceph-mon[32745]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:active 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: Cluster is now healthy 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T01:41:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:25 smithi183 ceph-mon[32752]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T01:41:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: pgmap v341: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-26T01:41:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] up:active 2024-04-26T01:41:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T01:41:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T01:41:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: Cluster is now healthy 2024-04-26T01:41:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T01:41:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T01:41:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:25 smithi052 ceph-mon[27851]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T01:41:26.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:26 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:active 2024-04-26T01:41:26.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:26 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:26.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:26 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:active 2024-04-26T01:41:26.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:26 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:26.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:26 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] up:active 2024-04-26T01:41:26.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:26 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:26.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.961+0000 7f7105200700 1 -- 172.21.15.52:0/3113671151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7100102af0 msgr2=0x7f7100102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.961+0000 7f7105200700 1 --2- 172.21.15.52:0/3113671151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7100102af0 0x7f7100102f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0099c0 tx=0x7f70ec009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 -- 172.21.15.52:0/3113671151 shutdown_connections 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 --2- 172.21.15.52:0/3113671151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f71001034b0 0x7f710010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 --2- 172.21.15.52:0/3113671151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100107940 0x7f7100107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 --2- 172.21.15.52:0/3113671151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7100102af0 0x7f7100102f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 -- 172.21.15.52:0/3113671151 >> 172.21.15.52:0/3113671151 conn(0x7f7100079ba0 msgr2=0x7f7100079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 -- 172.21.15.52:0/3113671151 shutdown_connections 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.962+0000 7f7105200700 1 -- 172.21.15.52:0/3113671151 wait complete. 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 Processor -- start 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 -- start start 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 0x7f7100074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71001034b0 0x7f7100075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7100112c50 con 0x7f71001034b0 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7100112ad0 con 0x7f7100107940 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.963+0000 7f7105200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7100112dd0 con 0x7f7100102af0 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fd400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71001034b0 0x7f7100075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46032/0 (socket says 172.21.15.52:46032) 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 -- 172.21.15.52:0/1513292507 learned_addr learned my addr 172.21.15.52:0/1513292507 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fde00700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 0x7f7100074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 -- 172.21.15.52:0/1513292507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 msgr2=0x7f7100074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 0x7f7100074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 -- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71001034b0 msgr2=0x7f7100075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:26.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71001034b0 0x7f7100075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:26.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fe800700 1 -- 172.21.15.52:0/1513292507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71001a76b0 con 0x7f7100107940 2024-04-26T01:41:26.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.964+0000 7f70fde00700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 0x7f7100074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:26.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f70fe800700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70f800efc0 tx=0x7f70f800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f80092e0 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70f8004bb0 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71001a7940 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8004020 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.965+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71001a7e00 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.966+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7100069730 con 0x7f7100107940 2024-04-26T01:41:26.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.968+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f70f801b030 con 0x7f7100107940 2024-04-26T01:41:26.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.969+0000 7f70f6a00700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 0x7f70e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:26.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.970+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f70f80cfcf0 con 0x7f7100107940 2024-04-26T01:41:26.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.970+0000 7f70fde00700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 0x7f70e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:26.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.970+0000 7f70fde00700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 0x7f70e40a6100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009c90 tx=0x7f70f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:26.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:26.974+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70f8022020 con 0x7f7100107940 2024-04-26T01:41:27.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.254+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f710002df80 con 0x7f7100107940 2024-04-26T01:41:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.390+0000 7f70f6a00700 1 -- 172.21.15.52:0/1513292507 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f70f8010040 con 0x7f7100107940 2024-04-26T01:41:27.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 msgr2=0x7f70e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 0x7f70e40a6100 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009c90 tx=0x7f70f0009260 comp rx=0 tx=0).stop 2024-04-26T01:41:27.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 msgr2=0x7f71000757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:27.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70f800efc0 tx=0x7f70f800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 shutdown_connections 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f70e40a3c40 0x7f70e40a6100 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7100107940 0x7f71000757c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7100102af0 0x7f7100074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 --2- 172.21.15.52:0/1513292507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f71001034b0 0x7f7100075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.395+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 >> 172.21.15.52:0/1513292507 conn(0x7f7100079ba0 msgr2=0x7f7100100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.396+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 shutdown_connections 2024-04-26T01:41:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:27.396+0000 7f7105200700 1 -- 172.21.15.52:0/1513292507 wait complete. 2024-04-26T01:41:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:27 smithi191 ceph-mon[32745]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 2 op/s 2024-04-26T01:41:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:27 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1513292507' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:27 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:27 smithi183 ceph-mon[32752]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 2 op/s 2024-04-26T01:41:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:27 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1513292507' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:27 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:27.674 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:41:27 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: 2024-04-26T01:41:27.385+0000 7f7e11b24700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T01:41:27.674 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 01:41:27 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: 2024-04-26T01:41:27.385+0000 7f7e11b24700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T01:41:27.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:27 smithi052 ceph-mon[27851]: pgmap v342: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 2 op/s 2024-04-26T01:41:27.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:27 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1513292507' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:27.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:27 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T01:41:28.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-26T01:41:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:28 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:41:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:28 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:28 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:28.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:28 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:41:28.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:28 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:28.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:28 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:28.685 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:28.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:28 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T01:41:28.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:28 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T01:41:28.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:28 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:29.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:29 smithi183 ceph-mon[32752]: pgmap v343: 97 pgs: 97 active+clean; 581 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-26T01:41:29.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:29 smithi052 ceph-mon[27851]: pgmap v343: 97 pgs: 97 active+clean; 581 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-26T01:41:29.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:29 smithi191 ceph-mon[32745]: pgmap v343: 97 pgs: 97 active+clean; 581 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.855+0000 7f6c79c00700 1 -- 172.21.15.52:0/3315901362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406b980 msgr2=0x7f6c7410f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.855+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3315901362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406b980 0x7f6c7410f8f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009f90 tx=0x7f6c6c00b330 comp rx=0 tx=0).stop 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 -- 172.21.15.52:0/3315901362 shutdown_connections 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3315901362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406b980 0x7f6c7410f8f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3315901362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c74108a20 0x7f6c74108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3315901362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c7406afe0 0x7f6c7406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 -- 172.21.15.52:0/3315901362 >> 172.21.15.52:0/3315901362 conn(0x7f6c740788b0 msgr2=0x7f6c74078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 -- 172.21.15.52:0/3315901362 shutdown_connections 2024-04-26T01:41:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.856+0000 7f6c79c00700 1 -- 172.21.15.52:0/3315901362 wait complete. 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 Processor -- start 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 -- start start 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 0x7f6c7419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c74108a20 0x7f6c7419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:30.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74112c10 con 0x7f6c7406afe0 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74112a90 con 0x7f6c7406b980 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.857+0000 7f6c79c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74112d90 con 0x7f6c74108a20 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51656/0 (socket says 172.21.15.52:51656) 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 -- 172.21.15.52:0/3579596858 learned_addr learned my addr 172.21.15.52:0/3579596858 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72000700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 0x7f6c7419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c73400700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c74108a20 0x7f6c7419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 -- 172.21.15.52:0/3579596858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c74108a20 msgr2=0x7f6c7419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c74108a20 0x7f6c7419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 -- 172.21.15.52:0/3579596858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 msgr2=0x7f6c7419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 0x7f6c7419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72a00700 1 -- 172.21.15.52:0/3579596858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c741a0370 con 0x7f6c7406afe0 2024-04-26T01:41:30.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.858+0000 7f6c72000700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 0x7f6c7419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:30.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c72a00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9a0 tx=0x7f6c6400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:30.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6400ca10 con 0x7f6c7406afe0 2024-04-26T01:41:30.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6400cbb0 con 0x7f6c7406afe0 2024-04-26T01:41:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64018680 con 0x7f6c7406afe0 2024-04-26T01:41:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c741a0660 con 0x7f6c7406afe0 2024-04-26T01:41:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.859+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c741a7ee0 con 0x7f6c7406afe0 2024-04-26T01:41:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.860+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c74069730 con 0x7f6c7406afe0 2024-04-26T01:41:30.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.862+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c64010430 con 0x7f6c7406afe0 2024-04-26T01:41:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.865+0000 7f6c6b400700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 0x7f6c580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.865+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f6c640d03e0 con 0x7f6c7406afe0 2024-04-26T01:41:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.865+0000 7f6c72000700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 0x7f6c580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.866+0000 7f6c72000700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 0x7f6c580a6180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009750 tx=0x7f6c60009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:30.870+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c6409ce70 con 0x7f6c7406afe0 2024-04-26T01:41:31.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.151+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f6c7402df80 con 0x7f6c7406afe0 2024-04-26T01:41:31.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.413+0000 7f6c6b400700 1 -- 172.21.15.52:0/3579596858 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f6c6409c770 con 0x7f6c7406afe0 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.417+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 msgr2=0x7f6c580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 0x7f6c580a6180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009750 tx=0x7f6c60009310 comp rx=0 tx=0).stop 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 msgr2=0x7f6c7419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9a0 tx=0x7f6c6400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 shutdown_connections 2024-04-26T01:41:31.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6c580a3cc0 0x7f6c580a6180 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c74108a20 0x7f6c7419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c7406afe0 0x7f6c7419b360 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 --2- 172.21.15.52:0/3579596858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c7406b980 0x7f6c7419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 >> 172.21.15.52:0/3579596858 conn(0x7f6c740788b0 msgr2=0x7f6c7410def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.418+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 shutdown_connections 2024-04-26T01:41:31.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:31.419+0000 7f6c79c00700 1 -- 172.21.15.52:0/3579596858 wait complete. 2024-04-26T01:41:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:31 smithi183 ceph-mon[32752]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:41:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:31 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:41:31.703 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:31 smithi191 ceph-mon[32745]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:41:31.703 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:31 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:41:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:31 smithi052 ceph-mon[27851]: pgmap v344: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-26T01:41:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:31 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T01:41:32.215 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T01:41:32.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-26T01:41:32.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:32 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:41:32.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:32 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:32.692 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:32.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:32 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:41:32.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:32 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:32.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:32 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3579596858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T01:41:32.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:32 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:41:33.699 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:33 smithi052 ceph-mon[27851]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:41:33.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:33 smithi191 ceph-mon[32745]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:41:33.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:33 smithi183 ceph-mon[32752]: pgmap v345: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-26T01:41:34.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 -- 172.21.15.52:0/1007352303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc075590 msgr2=0x7f99fc10ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:34.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 --2- 172.21.15.52:0/1007352303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc075590 0x7f99fc10ef30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009990 tx=0x7f99e4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 -- 172.21.15.52:0/1007352303 shutdown_connections 2024-04-26T01:41:34.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 --2- 172.21.15.52:0/1007352303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc10f470 0x7f99fc111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 --2- 172.21.15.52:0/1007352303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc074c70 0x7f99fc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 --2- 172.21.15.52:0/1007352303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc075590 0x7f99fc10ef30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 -- 172.21.15.52:0/1007352303 >> 172.21.15.52:0/1007352303 conn(0x7f99fc079bf0 msgr2=0x7f99fc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 -- 172.21.15.52:0/1007352303 shutdown_connections 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.846+0000 7f9a00e00700 1 -- 172.21.15.52:0/1007352303 wait complete. 2024-04-26T01:41:34.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.847+0000 7f9a00e00700 1 Processor -- start 2024-04-26T01:41:34.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.847+0000 7f9a00e00700 1 -- start start 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 0x7f99fc19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc075590 0x7f99fc19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 0x7f99fc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc114ec0 con 0x7f99fc075590 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc114d40 con 0x7f99fc10f470 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f9a00e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc115040 con 0x7f99fc074c70 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f99f9e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 0x7f99fc19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f99f9e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 0x7f99fc19b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55310/0 (socket says 172.21.15.52:55310) 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f99f9e00700 1 -- 172.21.15.52:0/2015610564 learned_addr learned my addr 172.21.15.52:0/2015610564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f99fa800700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 0x7f99fc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.848+0000 7f99f1400700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc075590 0x7f99fc19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:34.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 -- 172.21.15.52:0/2015610564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 msgr2=0x7f99fc19b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 0x7f99fc19b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 -- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc075590 msgr2=0x7f99fc19b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc075590 0x7f99fc19b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 -- 172.21.15.52:0/2015610564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1a0400 con 0x7f99fc10f470 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99fa800700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 0x7f99fc19fcc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99f400efc0 tx=0x7f99f400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:34.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.849+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f40092e0 con 0x7f99fc10f470 2024-04-26T01:41:34.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.850+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc1a0690 con 0x7f99fc10f470 2024-04-26T01:41:34.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.850+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99f4004bb0 con 0x7f99fc10f470 2024-04-26T01:41:34.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.850+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4004020 con 0x7f99fc10f470 2024-04-26T01:41:34.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.850+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc1a7f70 con 0x7f99fc10f470 2024-04-26T01:41:34.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.851+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99fc069730 con 0x7f99fc10f470 2024-04-26T01:41:34.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.853+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f99f401b030 con 0x7f99fc10f470 2024-04-26T01:41:34.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.854+0000 7f99f3400700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 0x7f99e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:34.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.854+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f99f40cffb0 con 0x7f99fc10f470 2024-04-26T01:41:34.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.855+0000 7f99f9e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 0x7f99e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:34.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.855+0000 7f99f9e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 0x7f99e00a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009d20 tx=0x7f99ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:34.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:34.857+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99f4022020 con 0x7f99fc10f470 2024-04-26T01:41:35.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.121+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f99fc19c750 con 0x7f99e00a3c40 2024-04-26T01:41:35.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.161+0000 7f99f3400700 1 -- 172.21.15.52:0/2015610564 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f99fc19c750 con 0x7f99e00a3c40 2024-04-26T01:41:35.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.164+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 msgr2=0x7f99e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:35.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.164+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 0x7f99e00a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009d20 tx=0x7f99ec009260 comp rx=0 tx=0).stop 2024-04-26T01:41:35.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.164+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 msgr2=0x7f99fc19fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:35.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.164+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 0x7f99fc19fcc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99f400efc0 tx=0x7f99f400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:41:35.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 shutdown_connections 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f99e00a3c40 0x7f99e00a6100 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99fc10f470 0x7f99fc19fcc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99fc074c70 0x7f99fc19b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 --2- 172.21.15.52:0/2015610564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99fc075590 0x7f99fc19b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 >> 172.21.15.52:0/2015610564 conn(0x7f99fc079bf0 msgr2=0x7f99fc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 shutdown_connections 2024-04-26T01:41:35.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:35.165+0000 7f9a00e00700 1 -- 172.21.15.52:0/2015610564 wait complete. 2024-04-26T01:41:35.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:41:35.135+0000 7f925f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:35.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:41:35.135+0000 7f925f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:35.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:41:35.135+0000 7f925f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:35.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:41:35.135+0000 7f925f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:35.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:41:35.135+0000 7f925f600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:41:35.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:35 smithi052 ceph-mon[27851]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:35.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:35 smithi191 ceph-mon[32745]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:35.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T01:41:35.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:35 smithi183 ceph-mon[32752]: pgmap v346: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:36.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:36.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:36 smithi052 ceph-mon[27851]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:36.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:36 smithi191 ceph-mon[32745]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:36.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:36 smithi183 ceph-mon[32752]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:37.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:37 smithi052 ceph-mon[27851]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:37.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:37 smithi052 ceph-mon[27851]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-26T01:41:37.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:37 smithi191 ceph-mon[32745]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:37.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:37 smithi191 ceph-mon[32745]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-26T01:41:37.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:37 smithi183 ceph-mon[32752]: pgmap v347: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-26T01:41:37.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:37 smithi183 ceph-mon[32752]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-26T01:41:38.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 -- 172.21.15.52:0/582492411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 msgr2=0x7f512c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 --2- 172.21.15.52:0/582492411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c06a750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f511c009990 tx=0x7f511c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 -- 172.21.15.52:0/582492411 shutdown_connections 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 --2- 172.21.15.52:0/582492411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 0x7f512c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 --2- 172.21.15.52:0/582492411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c06a750 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 --2- 172.21.15.52:0/582492411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 -- 172.21.15.52:0/582492411 >> 172.21.15.52:0/582492411 conn(0x7f512c0788f0 msgr2=0x7f512c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 -- 172.21.15.52:0/582492411 shutdown_connections 2024-04-26T01:41:38.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.540+0000 7f5131a00700 1 -- 172.21.15.52:0/582492411 wait complete. 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.541+0000 7f5131a00700 1 Processor -- start 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.541+0000 7f5131a00700 1 -- start start 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 0x7f512c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c1a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f512c1191d0 con 0x7f512c107970 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f512c119050 con 0x7f512c105480 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5131a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f512c119350 con 0x7f512c06ac90 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55356/0 (socket says 172.21.15.52:55356) 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512aa00700 1 -- 172.21.15.52:0/2339401735 learned_addr learned my addr 172.21.15.52:0/2339401735 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f5122000700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 0x7f512c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:38.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512b400700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c1a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512aa00700 1 -- 172.21.15.52:0/2339401735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 msgr2=0x7f512c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.542+0000 7f512aa00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 0x7f512c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f512aa00700 1 -- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 msgr2=0x7f512c1a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f512aa00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c1a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f512aa00700 1 -- 172.21.15.52:0/2339401735 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c1a4730 con 0x7f512c06ac90 2024-04-26T01:41:38.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f512aa00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f511c0057c0 tx=0x7f511c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c026070 con 0x7f512c06ac90 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512c1a49c0 con 0x7f512c06ac90 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f511c021410 con 0x7f512c06ac90 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c017650 con 0x7f512c06ac90 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.543+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512c1ac2a0 con 0x7f512c06ac90 2024-04-26T01:41:38.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.544+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f512c069730 con 0x7f512c06ac90 2024-04-26T01:41:38.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.546+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f511c029070 con 0x7f512c06ac90 2024-04-26T01:41:38.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.548+0000 7f5123e00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 0x7f51100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:38.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.548+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f511c0d1120 con 0x7f512c06ac90 2024-04-26T01:41:38.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.548+0000 7f5122000700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 0x7f51100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:38.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.549+0000 7f5122000700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 0x7f51100a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f512c079180 tx=0x7f5114009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:38.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.552+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f511c05dff0 con 0x7f512c06ac90 2024-04-26T01:41:38.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.820+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f512c104280 con 0x7f51100a3d50 2024-04-26T01:41:38.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.860+0000 7f5123e00700 1 -- 172.21.15.52:0/2339401735 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f512c104280 con 0x7f51100a3d50 2024-04-26T01:41:38.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 msgr2=0x7f51100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 0x7f51100a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f512c079180 tx=0x7f5114009260 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 msgr2=0x7f512c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f511c0057c0 tx=0x7f511c009580 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 shutdown_connections 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f51100a3d50 0x7f51100a6210 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f512c107970 0x7f512c1a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f512c06ac90 0x7f512c19f720 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 --2- 172.21.15.52:0/2339401735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f512c105480 0x7f512c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.863+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 >> 172.21.15.52:0/2339401735 conn(0x7f512c0788f0 msgr2=0x7f512c103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.864+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 shutdown_connections 2024-04-26T01:41:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:38.864+0000 7f5131a00700 1 -- 172.21.15.52:0/2339401735 wait complete. 2024-04-26T01:41:39.626 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T01:41:39.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:39 smithi052 ceph-mon[27851]: pgmap v348: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:41:39.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:39 smithi191 ceph-mon[32745]: pgmap v348: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:41:39.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:39 smithi183 ceph-mon[32752]: pgmap v348: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-26T01:41:40.119 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:40.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:40 smithi052 ceph-mon[27851]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:40.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:40 smithi191 ceph-mon[32745]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:40.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:40 smithi183 ceph-mon[32752]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:41.682 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:41 smithi052 ceph-mon[27851]: pgmap v349: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-26T01:41:41.825 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:41 smithi183 ceph-mon[32752]: pgmap v349: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-26T01:41:41.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:41 smithi191 ceph-mon[32745]: pgmap v349: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 9 op/s 2024-04-26T01:41:42.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 -- 172.21.15.52:0/1172568738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc108a20 msgr2=0x7fc5cc108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/1172568738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc108a20 0x7fc5cc108e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009990 tx=0x7fc5bc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 -- 172.21.15.52:0/1172568738 shutdown_connections 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/1172568738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06b980 0x7fc5cc10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/1172568738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc108a20 0x7fc5cc108e00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/1172568738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 -- 172.21.15.52:0/1172568738 >> 172.21.15.52:0/1172568738 conn(0x7fc5cc0788b0 msgr2=0x7fc5cc078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 -- 172.21.15.52:0/1172568738 shutdown_connections 2024-04-26T01:41:42.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.202+0000 7fc5d1e00700 1 -- 172.21.15.52:0/1172568738 wait complete. 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.203+0000 7fc5d1e00700 1 Processor -- start 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.203+0000 7fc5d1e00700 1 -- start start 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 0x7fc5cc101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc108a20 0x7fc5cc102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc112c10 con 0x7fc5cc06afe0 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc112a90 con 0x7fc5cc06b980 2024-04-26T01:41:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5d1e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc112d90 con 0x7fc5cc108a20 2024-04-26T01:41:42.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5caa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5c3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 0x7fc5cc101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5caa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44432/0 (socket says 172.21.15.52:44432) 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5caa00700 1 -- 172.21.15.52:0/3207994811 learned_addr learned my addr 172.21.15.52:0/3207994811 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.204+0000 7fc5cb400700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc108a20 0x7fc5cc102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 -- 172.21.15.52:0/3207994811 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc108a20 msgr2=0x7fc5cc102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc108a20 0x7fc5cc102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 -- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 msgr2=0x7fc5cc101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 -- 172.21.15.52:0/3207994811 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc1a76f0 con 0x7fc5cc06b980 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5c3e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 0x7fc5cc101f40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e9f0 tx=0x7fc5b400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.205+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4004220 con 0x7fc5cc06b980 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.206+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b40043c0 con 0x7fc5cc06b980 2024-04-26T01:41:42.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.206+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc1a79e0 con 0x7fc5cc06b980 2024-04-26T01:41:42.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.206+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400ce60 con 0x7fc5cc06b980 2024-04-26T01:41:42.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.206+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1a7f20 con 0x7fc5cc06b980 2024-04-26T01:41:42.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.207+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5cc069730 con 0x7fc5cc06b980 2024-04-26T01:41:42.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.209+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc5b4005290 con 0x7fc5cc06b980 2024-04-26T01:41:42.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.210+0000 7fc5d0a00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 0x7fc5b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:42.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.211+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fc5b40d0580 con 0x7fc5cc06b980 2024-04-26T01:41:42.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.211+0000 7fc5caa00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 0x7fc5b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:42.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.212+0000 7fc5caa00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 0x7fc5b00a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006010 tx=0x7fc5bc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:42.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.215+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5b409ceb0 con 0x7fc5cc06b980 2024-04-26T01:41:42.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.487+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fc5cc02e090 con 0x7fc5b00a3d50 2024-04-26T01:41:42.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.495+0000 7fc5d0a00700 1 -- 172.21.15.52:0/3207994811 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc5cc02e090 con 0x7fc5b00a3d50 2024-04-26T01:41:42.493 INFO:teuthology.orchestra.run.smithi052.stdout:/volumes/qa/sv_0/2e53ae82-4210-4b56-9099-65be4881b343 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 msgr2=0x7fc5b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 0x7fc5b00a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006010 tx=0x7fc5bc00b260 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 msgr2=0x7fc5cc101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 0x7fc5cc101f40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e9f0 tx=0x7fc5b400ed00 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 shutdown_connections 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc5b00a3d50 0x7fc5b00a6210 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5cc108a20 0x7fc5cc102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc5cc06afe0 0x7fc5cc101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 --2- 172.21.15.52:0/3207994811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5cc06b980 0x7fc5cc101f40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.498+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 >> 172.21.15.52:0/3207994811 conn(0x7fc5cc0788b0 msgr2=0x7fc5cc0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.499+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 shutdown_connections 2024-04-26T01:41:42.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:42.499+0000 7fc5d1e00700 1 -- 172.21.15.52:0/3207994811 wait complete. 2024-04-26T01:41:43.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T01:41:43.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:43 smithi052 ceph-mon[27851]: pgmap v350: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:41:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:43 smithi052 ceph-mon[27851]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:43.800 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:43.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:43 smithi191 ceph-mon[32745]: pgmap v350: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:41:43.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:43 smithi191 ceph-mon[32745]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:43.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:43 smithi183 ceph-mon[32752]: pgmap v350: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:41:43.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:43 smithi183 ceph-mon[32752]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:45 smithi052 ceph-mon[27851]: pgmap v351: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:41:45.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:45 smithi191 ceph-mon[32745]: pgmap v351: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:41:45.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:45 smithi183 ceph-mon[32752]: pgmap v351: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:41:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 -- 172.21.15.52:0/3891452706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494105870 msgr2=0x7fa4941098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 --2- 172.21.15.52:0/3891452706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494105870 0x7fa4941098c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0099c0 tx=0x7fa47c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 -- 172.21.15.52:0/3891452706 shutdown_connections 2024-04-26T01:41:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 --2- 172.21.15.52:0/3891452706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 0x7fa494111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 --2- 172.21.15.52:0/3891452706 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494104ec0 0x7fa4941052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 --2- 172.21.15.52:0/3891452706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494105870 0x7fa4941098c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 -- 172.21.15.52:0/3891452706 >> 172.21.15.52:0/3891452706 conn(0x7fa494100750 msgr2=0x7fa494102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 -- 172.21.15.52:0/3891452706 shutdown_connections 2024-04-26T01:41:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.968+0000 7fa49a600700 1 -- 172.21.15.52:0/3891452706 wait complete. 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.969+0000 7fa49a600700 1 Processor -- start 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 -- start start 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494105870 0x7fa49419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 0x7fa49419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa494114e10 con 0x7fa494104ec0 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa494114c90 con 0x7fa494105870 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa49a600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa494114f90 con 0x7fa494109fa0 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa493400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa493400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56208/0 (socket says 172.21.15.52:56208) 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa493400700 1 -- 172.21.15.52:0/3653820212 learned_addr learned my addr 172.21.15.52:0/3653820212 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa493e00700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 0x7fa49419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.970+0000 7fa48be00700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494105870 0x7fa49419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 -- 172.21.15.52:0/3653820212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 msgr2=0x7fa49419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 0x7fa49419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 -- 172.21.15.52:0/3653820212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494105870 msgr2=0x7fa49419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494105870 0x7fa49419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 -- 172.21.15.52:0/3653820212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4941a0320 con 0x7fa494104ec0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa493400700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa48400e9a0 tx=0x7fa48400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.971+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48400ca10 con 0x7fa494104ec0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.972+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa48400cbb0 con 0x7fa494104ec0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.972+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4941a0610 con 0x7fa494104ec0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.972+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484018680 con 0x7fa494104ec0 2024-04-26T01:41:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.972+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4941a7e90 con 0x7fa494104ec0 2024-04-26T01:41:45.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.973+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa494069730 con 0x7fa494104ec0 2024-04-26T01:41:45.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.975+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa484010430 con 0x7fa494104ec0 2024-04-26T01:41:45.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.976+0000 7fa490c00700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 0x7fa4780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:45.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.976+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fa4840d0660 con 0x7fa494104ec0 2024-04-26T01:41:45.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.977+0000 7fa48be00700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 0x7fa4780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:45.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.977+0000 7fa48be00700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 0x7fa4780a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:45.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:45.981+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa48409cf90 con 0x7fa494104ec0 2024-04-26T01:41:46.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.245+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa494062e80 con 0x7fa4780a3cd0 2024-04-26T01:41:46.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.291+0000 7fa490c00700 1 -- 172.21.15.52:0/3653820212 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa494062e80 con 0x7fa4780a3cd0 2024-04-26T01:41:46.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 msgr2=0x7fa4780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:46.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 0x7fa4780a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa47c000c00 tx=0x7fa47c005fb0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 msgr2=0x7fa49419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:46.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa48400e9a0 tx=0x7fa48400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 shutdown_connections 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.294+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa4780a3cd0 0x7fa4780a6190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa494109fa0 0x7fa49419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa494104ec0 0x7fa49419b360 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 --2- 172.21.15.52:0/3653820212 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa494105870 0x7fa49419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 >> 172.21.15.52:0/3653820212 conn(0x7fa494100750 msgr2=0x7fa494102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 shutdown_connections 2024-04-26T01:41:46.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:46.295+0000 7fa49a600700 1 -- 172.21.15.52:0/3653820212 wait complete. 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:46.686 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:46 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2024-04-26T01:41:46.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-26T01:41:46.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:46.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:46 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:46.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:46 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-26T01:41:47.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T01:41:47.588 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:47.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: pgmap v352: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-26T01:41:47.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:47 smithi052 ceph-mon[27851]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:41:47.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: pgmap v352: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-26T01:41:47.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:47 smithi191 ceph-mon[32745]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:41:47.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: pgmap v352: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-26T01:41:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:47 smithi183 ceph-mon[32752]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T01:41:48.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:48 smithi052 ceph-mon[27851]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:41:48.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:48 smithi052 ceph-mon[27851]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 KiB/s wr, 9 op/s 2024-04-26T01:41:48.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:48 smithi191 ceph-mon[32745]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:41:48.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:48 smithi191 ceph-mon[32745]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 KiB/s wr, 9 op/s 2024-04-26T01:41:48.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:48 smithi183 ceph-mon[32752]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T01:41:48.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:48 smithi183 ceph-mon[32752]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 KiB/s wr, 9 op/s 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 -- 172.21.15.52:0/637266367 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 msgr2=0x7f74181052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 --2- 172.21.15.52:0/637266367 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f74181052a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f740c0099c0 tx=0x7f740c009cd0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 -- 172.21.15.52:0/637266367 shutdown_connections 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 --2- 172.21.15.52:0/637266367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 0x7f7418111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 --2- 172.21.15.52:0/637266367 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f74181052a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 --2- 172.21.15.52:0/637266367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f74181098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 -- 172.21.15.52:0/637266367 >> 172.21.15.52:0/637266367 conn(0x7f7418100750 msgr2=0x7f7418102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:49.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 -- 172.21.15.52:0/637266367 shutdown_connections 2024-04-26T01:41:49.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.766+0000 7f741fe00700 1 -- 172.21.15.52:0/637266367 wait complete. 2024-04-26T01:41:49.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.767+0000 7f741fe00700 1 Processor -- start 2024-04-26T01:41:49.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 -- start start 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 0x7f741819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7418114e80 con 0x7f7418104ec0 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7418114d00 con 0x7f7418105870 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741fe00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7418115000 con 0x7f7418109fa0 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741ca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741ca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57744/0 (socket says 172.21.15.52:57744) 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56230/0 (socket says 172.21.15.52:56230) 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741ca00700 1 -- 172.21.15.52:0/107122180 learned_addr learned my addr 172.21.15.52:0/107122180 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.768+0000 7f741de00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 0x7f741819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 -- 172.21.15.52:0/107122180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 msgr2=0x7f741819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 0x7f741819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 -- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 msgr2=0x7f741819b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 -- 172.21.15.52:0/107122180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74181a0390 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741d400700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.769+0000 7f741ca00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f740400c8f0 tx=0x7f740400cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.770+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74040078d0 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.770+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f740400f450 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.770+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74181a0680 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.770+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404018660 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.770+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74181a7f00 con 0x7f7418105870 2024-04-26T01:41:49.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.771+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7418069730 con 0x7f7418105870 2024-04-26T01:41:49.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.773+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f740400f900 con 0x7f7418105870 2024-04-26T01:41:49.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.774+0000 7f7416000700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 0x7f74000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:49.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.774+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f74040d0580 con 0x7f7418105870 2024-04-26T01:41:49.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.775+0000 7f741d400700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 0x7f74000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:49.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.775+0000 7f741d400700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 0x7f74000a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f740c009990 tx=0x7f740c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:49.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:49.777+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7404014030 con 0x7f7418105870 2024-04-26T01:41:50.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.047+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f7418062e80 con 0x7f74000a3cd0 2024-04-26T01:41:50.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.059+0000 7f7416000700 1 -- 172.21.15.52:0/107122180 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7418062e80 con 0x7f74000a3cd0 2024-04-26T01:41:50.056 INFO:teuthology.orchestra.run.smithi052.stdout:/volumes/qa/sv_1/95eafff3-c868-445a-b0d3-cf6c55ba5d86 2024-04-26T01:41:50.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 msgr2=0x7f74000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:50.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 0x7f74000a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f740c009990 tx=0x7f740c000bc0 comp rx=0 tx=0).stop 2024-04-26T01:41:50.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 msgr2=0x7f741819b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f740400c8f0 tx=0x7f740400cc00 comp rx=0 tx=0).stop 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 shutdown_connections 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f74000a3cd0 0x7f74000a6190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7418109fa0 0x7f741819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7418104ec0 0x7f741819b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 --2- 172.21.15.52:0/107122180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7418105870 0x7f741819b910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 >> 172.21.15.52:0/107122180 conn(0x7f7418100750 msgr2=0x7f7418102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 shutdown_connections 2024-04-26T01:41:50.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:50.062+0000 7f741fe00700 1 -- 172.21.15.52:0/107122180 wait complete. 2024-04-26T01:41:50.808 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:41:51.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:50 smithi191 ceph-mon[32745]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.4 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:51.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:50 smithi191 ceph-mon[32745]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:51.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:50 smithi183 ceph-mon[32752]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.4 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:51.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:50 smithi183 ceph-mon[32752]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:50 smithi052 ceph-mon[27851]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.4 KiB/s wr, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:50 smithi052 ceph-mon[27851]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:41:51.273 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:52.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:52 smithi052 ceph-mon[27851]: pgmap v357: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:53.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:52 smithi191 ceph-mon[32745]: pgmap v357: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:53.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:52 smithi183 ceph-mon[32752]: pgmap v357: 97 pgs: 1 peering, 96 active+clean; 625 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:53.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.532+0000 7fdf47200700 1 -- 172.21.15.52:0/4023616614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 msgr2=0x7fdf401096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.532+0000 7fdf47200700 1 --2- 172.21.15.52:0/4023616614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401096b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b0a0 tx=0x7fdf3000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 -- 172.21.15.52:0/4023616614 shutdown_connections 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 --2- 172.21.15.52:0/4023616614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 0x7fdf40115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 --2- 172.21.15.52:0/4023616614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401096b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 --2- 172.21.15.52:0/4023616614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf4010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 -- 172.21.15.52:0/4023616614 >> 172.21.15.52:0/4023616614 conn(0x7fdf40076210 msgr2=0x7fdf40107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 -- 172.21.15.52:0/4023616614 shutdown_connections 2024-04-26T01:41:53.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.533+0000 7fdf47200700 1 -- 172.21.15.52:0/4023616614 wait complete. 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.534+0000 7fdf47200700 1 Processor -- start 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.534+0000 7fdf47200700 1 -- start start 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.534+0000 7fdf47200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf47200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf401a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf47200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 0x7fdf4019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf47200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf40119290 con 0x7fdf40109c80 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf47200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf40119110 con 0x7fdf401092d0 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf47200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf40119410 con 0x7fdf4010e3b0 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57752/0 (socket says 172.21.15.52:57752) 2024-04-26T01:41:53.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 -- 172.21.15.52:0/1549455383 learned_addr learned my addr 172.21.15.52:0/1549455383 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:53.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3f400700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf401a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf45200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 0x7fdf4019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 -- 172.21.15.52:0/1549455383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 msgr2=0x7fdf4019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 0x7fdf4019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 -- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 msgr2=0x7fdf401a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf401a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3fe00700 1 -- 172.21.15.52:0/1549455383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4019fd10 con 0x7fdf401092d0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.535+0000 7fdf3f400700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf401a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf3fe00700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b070 tx=0x7fdf30004d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30021070 con 0x7fdf401092d0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf30007c00 con 0x7fdf401092d0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4019ffa0 con 0x7fdf401092d0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300047e0 con 0x7fdf401092d0 2024-04-26T01:41:53.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.536+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf401ac2c0 con 0x7fdf401092d0 2024-04-26T01:41:53.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.537+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf40069730 con 0x7fdf401092d0 2024-04-26T01:41:53.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.539+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf30003a00 con 0x7fdf401092d0 2024-04-26T01:41:53.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.541+0000 7fdf3cc00700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 0x7fdf240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:53.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.541+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fdf3000f020 con 0x7fdf401092d0 2024-04-26T01:41:53.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.541+0000 7fdf3f400700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 0x7fdf240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:53.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.542+0000 7fdf3f400700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 0x7fdf240a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf400782a0 tx=0x7fdf28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:53.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.545+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf3002a020 con 0x7fdf401092d0 2024-04-26T01:41:53.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.829+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf4002df80 con 0x7fdf401092d0 2024-04-26T01:41:53.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.830+0000 7fdf3cc00700 1 -- 172.21.15.52:0/1549455383 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fdf3009e360 con 0x7fdf401092d0 2024-04-26T01:41:53.827 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:41:53.827 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:41:53.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 msgr2=0x7fdf240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 0x7fdf240a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf400782a0 tx=0x7fdf28009260 comp rx=0 tx=0).stop 2024-04-26T01:41:53.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 msgr2=0x7fdf401a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b070 tx=0x7fdf30004d10 comp rx=0 tx=0).stop 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 shutdown_connections 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdf240a3cd0 0x7fdf240a6190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf4010e3b0 0x7fdf4019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf401092d0 0x7fdf401a53f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 --2- 172.21.15.52:0/1549455383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf40109c80 0x7fdf401a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 >> 172.21.15.52:0/1549455383 conn(0x7fdf40076210 msgr2=0x7fdf4010cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.833+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 shutdown_connections 2024-04-26T01:41:53.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:53.834+0000 7fdf47200700 1 -- 172.21.15.52:0/1549455383 wait complete. 2024-04-26T01:41:53.832 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:41:54.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:41:54.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:54 smithi052 ceph-mon[27851]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.9 KiB/s wr, 8 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:54.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:54 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1549455383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:54 smithi191 ceph-mon[32745]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.9 KiB/s wr, 8 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:55.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:54 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1549455383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:55.161 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:41:55.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:54 smithi183 ceph-mon[32752]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.9 KiB/s wr, 8 op/s; 72 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:55.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:54 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1549455383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:41:56.910 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:56 smithi052 ceph-mon[27851]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 63 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:57.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:56 smithi191 ceph-mon[32745]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 63 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:57.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:56 smithi183 ceph-mon[32752]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.5 KiB/s wr, 7 op/s; 63 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 -- 172.21.15.52:0/3665386938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806a750 msgr2=0x7fc98806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 --2- 172.21.15.52:0/3665386938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806a750 0x7fc98806ab30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009990 tx=0x7fc97c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 -- 172.21.15.52:0/3665386938 shutdown_connections 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 --2- 172.21.15.52:0/3665386938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98810f4f0 0x7fc9881118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 --2- 172.21.15.52:0/3665386938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806a750 0x7fc98806ab30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 --2- 172.21.15.52:0/3665386938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98806b070 0x7fc98810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.423+0000 7fc98f800700 1 -- 172.21.15.52:0/3665386938 >> 172.21.15.52:0/3665386938 conn(0x7fc9880788f0 msgr2=0x7fc988078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:57.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.424+0000 7fc98f800700 1 -- 172.21.15.52:0/3665386938 shutdown_connections 2024-04-26T01:41:57.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.424+0000 7fc98f800700 1 -- 172.21.15.52:0/3665386938 wait complete. 2024-04-26T01:41:57.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 Processor -- start 2024-04-26T01:41:57.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 -- start start 2024-04-26T01:41:57.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98806a750 0x7fc98819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:57.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:57.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 0x7fc98819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:57.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc988114e50 con 0x7fc98810f4f0 2024-04-26T01:41:57.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc988114cd0 con 0x7fc98806b070 2024-04-26T01:41:57.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.425+0000 7fc98f800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc988114fd0 con 0x7fc98806a750 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc98d800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 0x7fc98819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc98ce00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98806a750 0x7fc98819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:52060/0 (socket says 172.21.15.52:52060) 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc98d800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 0x7fc98819fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52030/0 (socket says 172.21.15.52:52030) 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 -- 172.21.15.52:0/1032154062 learned_addr learned my addr 172.21.15.52:0/1032154062 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 -- 172.21.15.52:0/1032154062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98806a750 msgr2=0x7fc98819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:57.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98806a750 0x7fc98819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 -- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 msgr2=0x7fc98819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 0x7fc98819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 -- 172.21.15.52:0/1032154062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9881a0370 con 0x7fc98806b070 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.426+0000 7fc987e00700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9a0 tx=0x7fc97400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.427+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97400ca10 con 0x7fc98806b070 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.427+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97400cbb0 con 0x7fc98806b070 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.427+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974018680 con 0x7fc98806b070 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.427+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9881a0660 con 0x7fc98806b070 2024-04-26T01:41:57.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.427+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881a7ee0 con 0x7fc98806b070 2024-04-26T01:41:57.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.428+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc988069730 con 0x7fc98806b070 2024-04-26T01:41:57.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.433+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc974010430 con 0x7fc98806b070 2024-04-26T01:41:57.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.434+0000 7fc985600700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 0x7fc9700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:41:57.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.434+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fc9740d0950 con 0x7fc98806b070 2024-04-26T01:41:57.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.434+0000 7fc98ce00700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 0x7fc9700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:41:57.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.435+0000 7fc98ce00700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 0x7fc9700a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b2e0 tx=0x7fc97c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:41:57.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.438+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc97409d1e0 con 0x7fc98806b070 2024-04-26T01:41:57.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.667+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc98802df80 con 0x7fc98806b070 2024-04-26T01:41:57.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.669+0000 7fc985600700 1 -- 172.21.15.52:0/1032154062 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22667 (secure 0 0 0) 0x7fc97409cae0 con 0x7fc98806b070 2024-04-26T01:41:57.667 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:41:57.667 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":94,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:47.490242+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 msgr2=0x7fc9700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 0x7fc9700a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b2e0 tx=0x7fc97c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 msgr2=0x7fc98819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9a0 tx=0x7fc97400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 shutdown_connections 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc9700a3d50 0x7fc9700a6210 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98810f4f0 0x7fc98819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc98806a750 0x7fc98819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 --2- 172.21.15.52:0/1032154062 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98806b070 0x7fc98819b8a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 >> 172.21.15.52:0/1032154062 conn(0x7fc9880788f0 msgr2=0x7fc9881109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 shutdown_connections 2024-04-26T01:41:57.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:41:57.673+0000 7fc98f800700 1 -- 172.21.15.52:0/1032154062 wait complete. 2024-04-26T01:41:57.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:57 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:57.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:57 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1032154062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:58.064 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:57 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:58.064 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:57 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1032154062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:58.139 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:57 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:41:58.139 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:57 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1032154062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:41:58.559 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:41:58.560 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:41:59.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:58 smithi052 ceph-mon[27851]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 57 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:41:59.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:58 smithi191 ceph-mon[32745]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 57 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:41:59.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:58 smithi183 ceph-mon[32752]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 57 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:41:59.633 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:41:59 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:41:59 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.173 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:00.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:41:59 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:01.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:00 smithi052 ceph-mon[27851]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:42:01.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:00 smithi191 ceph-mon[32745]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:42:01.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:00 smithi183 ceph-mon[32752]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T01:42:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.715+0000 7ff12aa00700 1 -- 172.21.15.52:0/3341696789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124074c70 msgr2=0x7ff124075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.715+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3341696789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124074c70 0x7ff124075050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff114009990 tx=0x7ff114009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 -- 172.21.15.52:0/3341696789 shutdown_connections 2024-04-26T01:42:01.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3341696789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff124111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3341696789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124074c70 0x7ff124075050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3341696789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124075590 0x7ff12410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 -- 172.21.15.52:0/3341696789 >> 172.21.15.52:0/3341696789 conn(0x7ff124079ba0 msgr2=0x7ff124079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:01.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 -- 172.21.15.52:0/3341696789 shutdown_connections 2024-04-26T01:42:01.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.716+0000 7ff12aa00700 1 -- 172.21.15.52:0/3341696789 wait complete. 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.717+0000 7ff12aa00700 1 Processor -- start 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.717+0000 7ff12aa00700 1 -- start start 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.717+0000 7ff12aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124074c70 0x7ff12419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff12aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff12aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff12419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff12aa00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff124114e70 con 0x7ff12410f440 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff12aa00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff124114cf0 con 0x7ff124074c70 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff12aa00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff124114ff0 con 0x7ff124075590 2024-04-26T01:42:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:01.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37344/0 (socket says 172.21.15.52:37344) 2024-04-26T01:42:01.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 -- 172.21.15.52:0/3476334555 learned_addr learned my addr 172.21.15.52:0/3476334555 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:01.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124074c70 0x7ff12419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:01.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff128a00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff12419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 -- 172.21.15.52:0/3476334555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124074c70 msgr2=0x7ff12419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124074c70 0x7ff12419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 -- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 msgr2=0x7ff12419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff12419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff123400700 1 -- 172.21.15.52:0/3476334555 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1241a0310 con 0x7ff124075590 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.718+0000 7ff128a00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff12419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff123400700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d980 tx=0x7ff11000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff124075590 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff110009de0 con 0x7ff124075590 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1241a0600 con 0x7ff124075590 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11000f3c0 con 0x7ff124075590 2024-04-26T01:42:01.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.719+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1241a7e80 con 0x7ff124075590 2024-04-26T01:42:01.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.720+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff124069730 con 0x7ff124075590 2024-04-26T01:42:01.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.723+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff110009930 con 0x7ff124075590 2024-04-26T01:42:01.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.726+0000 7ff120c00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:01.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.726+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7ff1100d0110 con 0x7ff124075590 2024-04-26T01:42:01.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.726+0000 7ff123e00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:01.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.727+0000 7ff123e00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 0x7ff1080a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff11400b2e0 tx=0x7ff114005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:01.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.727+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff11009c9c0 con 0x7ff124075590 2024-04-26T01:42:01.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.974+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff124062e80 con 0x7ff1080a3d50 2024-04-26T01:42:01.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.984+0000 7ff120c00700 1 -- 172.21.15.52:0/3476334555 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258739 (secure 0 0 0) 0x7ff1140672d0 con 0x7ff1080a3d50 2024-04-26T01:42:01.987 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:42:01.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 msgr2=0x7ff1080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 0x7ff1080a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff11400b2e0 tx=0x7ff114005fd0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 msgr2=0x7ff12419b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff11000d980 tx=0x7ff11000dc90 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 shutdown_connections 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff12410f440 0x7ff12419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff124074c70 0x7ff12419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 --2- 172.21.15.52:0/3476334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff124075590 0x7ff12419b840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.989+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 >> 172.21.15.52:0/3476334555 conn(0x7ff124079ba0 msgr2=0x7ff1240ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.990+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 shutdown_connections 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:01.990+0000 7ff12aa00700 1 -- 172.21.15.52:0/3476334555 wait complete. 2024-04-26T01:42:01.990 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:42:02.745 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-04-26T01:42:01.683157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657675,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":356,"num_read_kb":296,"num_write":349,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388236,"kb_used_data":14088,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553588,"statfs":{"total":1151940427776,"available":1151542874112,"internally_reserved":0,"allocated":14426112,"data_stored":7938765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17561,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":47,"num_write":16,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004720"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497027+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:41:47.497027+0000","last_peered":"2024-04-26T01:41:47.497027+0000","last_clean":"2024-04-26T01:41:47.497027+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:41:47.497027+0000","last_undegraded":"2024-04-26T01:41:47.497027+0000","last_fullsized":"2024-04-26T01:41:47.497027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504566+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:41:47.504566+0000","last_peered":"2024-04-26T01:41:47.504566+0000","last_clean":"2024-04-26T01:41:47.504566+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:41:47.504566+0000","last_undegraded":"2024-04-26T01:41:47.504566+0000","last_fullsized":"2024-04-26T01:41:47.504566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504433+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:41:47.504433+0000","last_peered":"2024-04-26T01:41:47.504433+0000","last_clean":"2024-04-26T01:41:47.504433+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:41:47.504433+0000","last_undegraded":"2024-04-26T01:41:47.504433+0000","last_fullsized":"2024-04-26T01:41:47.504433+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503142+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:41:47.503142+0000","last_peered":"2024-04-26T01:41:47.503142+0000","last_clean":"2024-04-26T01:41:47.503142+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:41:47.503142+0000","last_undegraded":"2024-04-26T01:41:47.503142+0000","last_fullsized":"2024-04-26T01:41:47.503142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499067+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:41:47.499067+0000","last_peered":"2024-04-26T01:41:47.499067+0000","last_clean":"2024-04-26T01:41:47.499067+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:41:47.499067+0000","last_undegraded":"2024-04-26T01:41:47.499067+0000","last_fullsized":"2024-04-26T01:41:47.499067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508633+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:41:47.508633+0000","last_peered":"2024-04-26T01:41:47.508633+0000","last_clean":"2024-04-26T01:41:47.508633+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:41:47.508633+0000","last_undegraded":"2024-04-26T01:41:47.508633+0000","last_fullsized":"2024-04-26T01:41:47.508633+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499821+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:41:47.499821+0000","last_peered":"2024-04-26T01:41:47.499821+0000","last_clean":"2024-04-26T01:41:47.499821+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:41:47.499821+0000","last_undegraded":"2024-04-26T01:41:47.499821+0000","last_fullsized":"2024-04-26T01:41:47.499821+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509232+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:41:47.509232+0000","last_peered":"2024-04-26T01:41:47.509232+0000","last_clean":"2024-04-26T01:41:47.509232+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:41:47.509232+0000","last_undegraded":"2024-04-26T01:41:47.509232+0000","last_fullsized":"2024-04-26T01:41:47.509232+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509390+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:41:47.509390+0000","last_peered":"2024-04-26T01:41:47.509390+0000","last_clean":"2024-04-26T01:41:47.509390+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:41:47.509390+0000","last_undegraded":"2024-04-26T01:41:47.509390+0000","last_fullsized":"2024-04-26T01:41:47.509390+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508813+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:41:47.508813+0000","last_peered":"2024-04-26T01:41:47.508813+0000","last_clean":"2024-04-26T01:41:47.508813+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:41:47.508813+0000","last_undegraded":"2024-04-26T01:41:47.508813+0000","last_fullsized":"2024-04-26T01:41:47.508813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499519+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:41:47.499519+0000","last_peered":"2024-04-26T01:41:47.499519+0000","last_clean":"2024-04-26T01:41:47.499519+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:41:47.499519+0000","last_undegraded":"2024-04-26T01:41:47.499519+0000","last_fullsized":"2024-04-26T01:41:47.499519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497259+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:41:47.497259+0000","last_peered":"2024-04-26T01:41:47.497259+0000","last_clean":"2024-04-26T01:41:47.497259+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:41:47.497259+0000","last_undegraded":"2024-04-26T01:41:47.497259+0000","last_fullsized":"2024-04-26T01:41:47.497259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494766+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:41:47.494766+0000","last_peered":"2024-04-26T01:41:47.494766+0000","last_clean":"2024-04-26T01:41:47.494766+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:41:47.494766+0000","last_undegraded":"2024-04-26T01:41:47.494766+0000","last_fullsized":"2024-04-26T01:41:47.494766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503984+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:41:47.503984+0000","last_peered":"2024-04-26T01:41:47.503984+0000","last_clean":"2024-04-26T01:41:47.503984+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:41:47.503984+0000","last_undegraded":"2024-04-26T01:41:47.503984+0000","last_fullsized":"2024-04-26T01:41:47.503984+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503750+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:41:47.503750+0000","last_peered":"2024-04-26T01:41:47.503750+0000","last_clean":"2024-04-26T01:41:47.503750+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:41:47.503750+0000","last_undegraded":"2024-04-26T01:41:47.503750+0000","last_fullsized":"2024-04-26T01:41:47.503750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508483+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:41:47.508483+0000","last_peered":"2024-04-26T01:41:47.508483+0000","last_clean":"2024-04-26T01:41:47.508483+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:41:47.508483+0000","last_undegraded":"2024-04-26T01:41:47.508483+0000","last_fullsized":"2024-04-26T01:41:47.508483+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497244+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:41:47.497244+0000","last_peered":"2024-04-26T01:41:47.497244+0000","last_clean":"2024-04-26T01:41:47.497244+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:41:47.497244+0000","last_undegraded":"2024-04-26T01:41:47.497244+0000","last_fullsized":"2024-04-26T01:41:47.497244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498488+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:41:47.498488+0000","last_peered":"2024-04-26T01:41:47.498488+0000","last_clean":"2024-04-26T01:41:47.498488+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:41:47.498488+0000","last_undegraded":"2024-04-26T01:41:47.498488+0000","last_fullsized":"2024-04-26T01:41:47.498488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494907+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:41:47.494907+0000","last_peered":"2024-04-26T01:41:47.494907+0000","last_clean":"2024-04-26T01:41:47.494907+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:41:47.494907+0000","last_undegraded":"2024-04-26T01:41:47.494907+0000","last_fullsized":"2024-04-26T01:41:47.494907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498685+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:41:47.498685+0000","last_peered":"2024-04-26T01:41:47.498685+0000","last_clean":"2024-04-26T01:41:47.498685+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:41:47.498685+0000","last_undegraded":"2024-04-26T01:41:47.498685+0000","last_fullsized":"2024-04-26T01:41:47.498685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510113+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:41:47.510113+0000","last_peered":"2024-04-26T01:41:47.510113+0000","last_clean":"2024-04-26T01:41:47.510113+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:41:47.510113+0000","last_undegraded":"2024-04-26T01:41:47.510113+0000","last_fullsized":"2024-04-26T01:41:47.510113+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510446+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:41:47.510446+0000","last_peered":"2024-04-26T01:41:47.510446+0000","last_clean":"2024-04-26T01:41:47.510446+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:41:47.510446+0000","last_undegraded":"2024-04-26T01:41:47.510446+0000","last_fullsized":"2024-04-26T01:41:47.510446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504783+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:41:47.504783+0000","last_peered":"2024-04-26T01:41:47.504783+0000","last_clean":"2024-04-26T01:41:47.504783+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:41:47.504783+0000","last_undegraded":"2024-04-26T01:41:47.504783+0000","last_fullsized":"2024-04-26T01:41:47.504783+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.506427+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:41:47.506427+0000","last_peered":"2024-04-26T01:41:47.506427+0000","last_clean":"2024-04-26T01:41:47.506427+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:41:47.506427+0000","last_undegraded":"2024-04-26T01:41:47.506427+0000","last_fullsized":"2024-04-26T01:41:47.506427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508949+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:41:47.508949+0000","last_peered":"2024-04-26T01:41:47.508949+0000","last_clean":"2024-04-26T01:41:47.508949+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:41:47.508949+0000","last_undegraded":"2024-04-26T01:41:47.508949+0000","last_fullsized":"2024-04-26T01:41:47.508949+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.500059+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:41:47.500059+0000","last_peered":"2024-04-26T01:41:47.500059+0000","last_clean":"2024-04-26T01:41:47.500059+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:41:47.500059+0000","last_undegraded":"2024-04-26T01:41:47.500059+0000","last_fullsized":"2024-04-26T01:41:47.500059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508654+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:41:47.508654+0000","last_peered":"2024-04-26T01:41:47.508654+0000","last_clean":"2024-04-26T01:41:47.508654+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:41:47.508654+0000","last_undegraded":"2024-04-26T01:41:47.508654+0000","last_fullsized":"2024-04-26T01:41:47.508654+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:42:00.453918+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:42:00.453918+0000","last_peered":"2024-04-26T01:42:00.453918+0000","last_clean":"2024-04-26T01:42:00.453918+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:42:00.453918+0000","last_undegraded":"2024-04-26T01:42:00.453918+0000","last_fullsized":"2024-04-26T01:42:00.453918+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499158+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:41:47.499158+0000","last_peered":"2024-04-26T01:41:47.499158+0000","last_clean":"2024-04-26T01:41:47.499158+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:41:47.499158+0000","last_undegraded":"2024-04-26T01:41:47.499158+0000","last_fullsized":"2024-04-26T01:41:47.499158+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508594+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:41:47.508594+0000","last_peered":"2024-04-26T01:41:47.508594+0000","last_clean":"2024-04-26T01:41:47.508594+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:41:47.508594+0000","last_undegraded":"2024-04-26T01:41:47.508594+0000","last_fullsized":"2024-04-26T01:41:47.508594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508357+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:41:47.508357+0000","last_peered":"2024-04-26T01:41:47.508357+0000","last_clean":"2024-04-26T01:41:47.508357+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:41:47.508357+0000","last_undegraded":"2024-04-26T01:41:47.508357+0000","last_fullsized":"2024-04-26T01:41:47.508357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494651+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:41:47.494651+0000","last_peered":"2024-04-26T01:41:47.494651+0000","last_clean":"2024-04-26T01:41:47.494651+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:41:47.494651+0000","last_undegraded":"2024-04-26T01:41:47.494651+0000","last_fullsized":"2024-04-26T01:41:47.494651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509161+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:41:47.509161+0000","last_peered":"2024-04-26T01:41:47.509161+0000","last_clean":"2024-04-26T01:41:47.509161+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:41:47.509161+0000","last_undegraded":"2024-04-26T01:41:47.509161+0000","last_fullsized":"2024-04-26T01:41:47.509161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508108+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:41:47.508108+0000","last_peered":"2024-04-26T01:41:47.508108+0000","last_clean":"2024-04-26T01:41:47.508108+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:41:47.508108+0000","last_undegraded":"2024-04-26T01:41:47.508108+0000","last_fullsized":"2024-04-26T01:41:47.508108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510772+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:41:47.510772+0000","last_peered":"2024-04-26T01:41:47.510772+0000","last_clean":"2024-04-26T01:41:47.510772+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:41:47.510772+0000","last_undegraded":"2024-04-26T01:41:47.510772+0000","last_fullsized":"2024-04-26T01:41:47.510772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:56.448145+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:41:56.448145+0000","last_peered":"2024-04-26T01:41:56.448145+0000","last_clean":"2024-04-26T01:41:56.448145+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:41:56.448145+0000","last_undegraded":"2024-04-26T01:41:56.448145+0000","last_fullsized":"2024-04-26T01:41:56.448145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509329+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:41:47.509329+0000","last_peered":"2024-04-26T01:41:47.509329+0000","last_clean":"2024-04-26T01:41:47.509329+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:41:47.509329+0000","last_undegraded":"2024-04-26T01:41:47.509329+0000","last_fullsized":"2024-04-26T01:41:47.509329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499597+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:41:47.499597+0000","last_peered":"2024-04-26T01:41:47.499597+0000","last_clean":"2024-04-26T01:41:47.499597+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:41:47.499597+0000","last_undegraded":"2024-04-26T01:41:47.499597+0000","last_fullsized":"2024-04-26T01:41:47.499597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.505535+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:41:47.505535+0000","last_peered":"2024-04-26T01:41:47.505535+0000","last_clean":"2024-04-26T01:41:47.505535+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:41:47.505535+0000","last_undegraded":"2024-04-26T01:41:47.505535+0000","last_fullsized":"2024-04-26T01:41:47.505535+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508753+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:41:47.508753+0000","last_peered":"2024-04-26T01:41:47.508753+0000","last_clean":"2024-04-26T01:41:47.508753+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:41:47.508753+0000","last_undegraded":"2024-04-26T01:41:47.508753+0000","last_fullsized":"2024-04-26T01:41:47.508753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509086+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:41:47.509086+0000","last_peered":"2024-04-26T01:41:47.509086+0000","last_clean":"2024-04-26T01:41:47.509086+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:41:47.509086+0000","last_undegraded":"2024-04-26T01:41:47.509086+0000","last_fullsized":"2024-04-26T01:41:47.509086+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'30","reported_seq":126,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:57.450223+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:41:57.450223+0000","last_peered":"2024-04-26T01:41:57.450223+0000","last_clean":"2024-04-26T01:41:57.450223+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:41:57.450223+0000","last_undegraded":"2024-04-26T01:41:57.450223+0000","last_fullsized":"2024-04-26T01:41:57.450223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63072,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":96,"num_write":31,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499210+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:41:47.499210+0000","last_peered":"2024-04-26T01:41:47.499210+0000","last_clean":"2024-04-26T01:41:47.499210+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:41:47.499210+0000","last_undegraded":"2024-04-26T01:41:47.499210+0000","last_fullsized":"2024-04-26T01:41:47.499210+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494848+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:41:47.494848+0000","last_peered":"2024-04-26T01:41:47.494848+0000","last_clean":"2024-04-26T01:41:47.494848+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:41:47.494848+0000","last_undegraded":"2024-04-26T01:41:47.494848+0000","last_fullsized":"2024-04-26T01:41:47.494848+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494661+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:41:47.494661+0000","last_peered":"2024-04-26T01:41:47.494661+0000","last_clean":"2024-04-26T01:41:47.494661+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:41:47.494661+0000","last_undegraded":"2024-04-26T01:41:47.494661+0000","last_fullsized":"2024-04-26T01:41:47.494661+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508326+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:41:47.508326+0000","last_peered":"2024-04-26T01:41:47.508326+0000","last_clean":"2024-04-26T01:41:47.508326+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:41:47.508326+0000","last_undegraded":"2024-04-26T01:41:47.508326+0000","last_fullsized":"2024-04-26T01:41:47.508326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499423+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:41:47.499423+0000","last_peered":"2024-04-26T01:41:47.499423+0000","last_clean":"2024-04-26T01:41:47.499423+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:41:47.499423+0000","last_undegraded":"2024-04-26T01:41:47.499423+0000","last_fullsized":"2024-04-26T01:41:47.499423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'4","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:59.477767+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:41:59.477767+0000","last_peered":"2024-04-26T01:41:59.477767+0000","last_clean":"2024-04-26T01:41:59.477767+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:41:59.477767+0000","last_undegraded":"2024-04-26T01:41:59.477767+0000","last_fullsized":"2024-04-26T01:41:59.477767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499162+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:41:47.499162+0000","last_peered":"2024-04-26T01:41:47.499162+0000","last_clean":"2024-04-26T01:41:47.499162+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:41:47.499162+0000","last_undegraded":"2024-04-26T01:41:47.499162+0000","last_fullsized":"2024-04-26T01:41:47.499162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498976+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:41:47.498976+0000","last_peered":"2024-04-26T01:41:47.498976+0000","last_clean":"2024-04-26T01:41:47.498976+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:41:47.498976+0000","last_undegraded":"2024-04-26T01:41:47.498976+0000","last_fullsized":"2024-04-26T01:41:47.498976+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.511194+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:41:47.511194+0000","last_peered":"2024-04-26T01:41:47.511194+0000","last_clean":"2024-04-26T01:41:47.511194+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:41:47.511194+0000","last_undegraded":"2024-04-26T01:41:47.511194+0000","last_fullsized":"2024-04-26T01:41:47.511194+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499598+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:41:47.499598+0000","last_peered":"2024-04-26T01:41:47.499598+0000","last_clean":"2024-04-26T01:41:47.499598+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:41:47.499598+0000","last_undegraded":"2024-04-26T01:41:47.499598+0000","last_fullsized":"2024-04-26T01:41:47.499598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496846+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:41:47.496846+0000","last_peered":"2024-04-26T01:41:47.496846+0000","last_clean":"2024-04-26T01:41:47.496846+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:41:47.496846+0000","last_undegraded":"2024-04-26T01:41:47.496846+0000","last_fullsized":"2024-04-26T01:41:47.496846+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508155+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:41:47.508155+0000","last_peered":"2024-04-26T01:41:47.508155+0000","last_clean":"2024-04-26T01:41:47.508155+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:41:47.508155+0000","last_undegraded":"2024-04-26T01:41:47.508155+0000","last_fullsized":"2024-04-26T01:41:47.508155+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509597+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:41:47.509597+0000","last_peered":"2024-04-26T01:41:47.509597+0000","last_clean":"2024-04-26T01:41:47.509597+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:41:47.509597+0000","last_undegraded":"2024-04-26T01:41:47.509597+0000","last_fullsized":"2024-04-26T01:41:47.509597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499417+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:41:47.499417+0000","last_peered":"2024-04-26T01:41:47.499417+0000","last_clean":"2024-04-26T01:41:47.499417+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:41:47.499417+0000","last_undegraded":"2024-04-26T01:41:47.499417+0000","last_fullsized":"2024-04-26T01:41:47.499417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508774+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:41:47.508774+0000","last_peered":"2024-04-26T01:41:47.508774+0000","last_clean":"2024-04-26T01:41:47.508774+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:41:47.508774+0000","last_undegraded":"2024-04-26T01:41:47.508774+0000","last_fullsized":"2024-04-26T01:41:47.508774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508586+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:41:47.508586+0000","last_peered":"2024-04-26T01:41:47.508586+0000","last_clean":"2024-04-26T01:41:47.508586+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:41:47.508586+0000","last_undegraded":"2024-04-26T01:41:47.508586+0000","last_fullsized":"2024-04-26T01:41:47.508586+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'4","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:57.473258+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:41:57.473258+0000","last_peered":"2024-04-26T01:41:57.473258+0000","last_clean":"2024-04-26T01:41:57.473258+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:41:57.473258+0000","last_undegraded":"2024-04-26T01:41:57.473258+0000","last_fullsized":"2024-04-26T01:41:57.473258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509636+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:41:47.509636+0000","last_peered":"2024-04-26T01:41:47.509636+0000","last_clean":"2024-04-26T01:41:47.509636+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:41:47.509636+0000","last_undegraded":"2024-04-26T01:41:47.509636+0000","last_fullsized":"2024-04-26T01:41:47.509636+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497435+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:41:47.497435+0000","last_peered":"2024-04-26T01:41:47.497435+0000","last_clean":"2024-04-26T01:41:47.497435+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:41:47.497435+0000","last_undegraded":"2024-04-26T01:41:47.497435+0000","last_fullsized":"2024-04-26T01:41:47.497435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499373+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:41:47.499373+0000","last_peered":"2024-04-26T01:41:47.499373+0000","last_clean":"2024-04-26T01:41:47.499373+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:41:47.499373+0000","last_undegraded":"2024-04-26T01:41:47.499373+0000","last_fullsized":"2024-04-26T01:41:47.499373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494519+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:41:47.494519+0000","last_peered":"2024-04-26T01:41:47.494519+0000","last_clean":"2024-04-26T01:41:47.494519+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:41:47.494519+0000","last_undegraded":"2024-04-26T01:41:47.494519+0000","last_fullsized":"2024-04-26T01:41:47.494519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499489+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:41:47.499489+0000","last_peered":"2024-04-26T01:41:47.499489+0000","last_clean":"2024-04-26T01:41:47.499489+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:41:47.499489+0000","last_undegraded":"2024-04-26T01:41:47.499489+0000","last_fullsized":"2024-04-26T01:41:47.499489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499392+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:41:47.499392+0000","last_peered":"2024-04-26T01:41:47.499392+0000","last_clean":"2024-04-26T01:41:47.499392+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:41:47.499392+0000","last_undegraded":"2024-04-26T01:41:47.499392+0000","last_fullsized":"2024-04-26T01:41:47.499392+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495185+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:41:47.495185+0000","last_peered":"2024-04-26T01:41:47.495185+0000","last_clean":"2024-04-26T01:41:47.495185+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:41:47.495185+0000","last_undegraded":"2024-04-26T01:41:47.495185+0000","last_fullsized":"2024-04-26T01:41:47.495185+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'12","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:52.316162+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:41:52.316162+0000","last_peered":"2024-04-26T01:41:52.316162+0000","last_clean":"2024-04-26T01:41:52.316162+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:41:52.316162+0000","last_undegraded":"2024-04-26T01:41:52.316162+0000","last_fullsized":"2024-04-26T01:41:52.316162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509948+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:41:47.509948+0000","last_peered":"2024-04-26T01:41:47.509948+0000","last_clean":"2024-04-26T01:41:47.509948+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:41:47.509948+0000","last_undegraded":"2024-04-26T01:41:47.509948+0000","last_fullsized":"2024-04-26T01:41:47.509948+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497049+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:41:47.497049+0000","last_peered":"2024-04-26T01:41:47.497049+0000","last_clean":"2024-04-26T01:41:47.497049+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:41:47.497049+0000","last_undegraded":"2024-04-26T01:41:47.497049+0000","last_fullsized":"2024-04-26T01:41:47.497049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504585+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:41:47.504585+0000","last_peered":"2024-04-26T01:41:47.504585+0000","last_clean":"2024-04-26T01:41:47.504585+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:41:47.504585+0000","last_undegraded":"2024-04-26T01:41:47.504585+0000","last_fullsized":"2024-04-26T01:41:47.504585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508399+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:41:47.508399+0000","last_peered":"2024-04-26T01:41:47.508399+0000","last_clean":"2024-04-26T01:41:47.508399+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:41:47.508399+0000","last_undegraded":"2024-04-26T01:41:47.508399+0000","last_fullsized":"2024-04-26T01:41:47.508399+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499009+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:41:47.499009+0000","last_peered":"2024-04-26T01:41:47.499009+0000","last_clean":"2024-04-26T01:41:47.499009+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:41:47.499009+0000","last_undegraded":"2024-04-26T01:41:47.499009+0000","last_fullsized":"2024-04-26T01:41:47.499009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509372+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:41:47.509372+0000","last_peered":"2024-04-26T01:41:47.509372+0000","last_clean":"2024-04-26T01:41:47.509372+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:41:47.509372+0000","last_undegraded":"2024-04-26T01:41:47.509372+0000","last_fullsized":"2024-04-26T01:41:47.509372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496629+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:41:47.496629+0000","last_peered":"2024-04-26T01:41:47.496629+0000","last_clean":"2024-04-26T01:41:47.496629+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:41:47.496629+0000","last_undegraded":"2024-04-26T01:41:47.496629+0000","last_fullsized":"2024-04-26T01:41:47.496629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494637+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:41:47.494637+0000","last_peered":"2024-04-26T01:41:47.494637+0000","last_clean":"2024-04-26T01:41:47.494637+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:41:47.494637+0000","last_undegraded":"2024-04-26T01:41:47.494637+0000","last_fullsized":"2024-04-26T01:41:47.494637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510042+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:41:47.510042+0000","last_peered":"2024-04-26T01:41:47.510042+0000","last_clean":"2024-04-26T01:41:47.510042+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:41:47.510042+0000","last_undegraded":"2024-04-26T01:41:47.510042+0000","last_fullsized":"2024-04-26T01:41:47.510042+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499216+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:41:47.499216+0000","last_peered":"2024-04-26T01:41:47.499216+0000","last_clean":"2024-04-26T01:41:47.499216+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:41:47.499216+0000","last_undegraded":"2024-04-26T01:41:47.499216+0000","last_fullsized":"2024-04-26T01:41:47.499216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499650+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:41:47.499650+0000","last_peered":"2024-04-26T01:41:47.499650+0000","last_clean":"2024-04-26T01:41:47.499650+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:41:47.499650+0000","last_undegraded":"2024-04-26T01:41:47.499650+0000","last_fullsized":"2024-04-26T01:41:47.499650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.513560+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:41:47.513560+0000","last_peered":"2024-04-26T01:41:47.513560+0000","last_clean":"2024-04-26T01:41:47.513560+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:41:47.513560+0000","last_undegraded":"2024-04-26T01:41:47.513560+0000","last_fullsized":"2024-04-26T01:41:47.513560+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499680+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:41:47.499680+0000","last_peered":"2024-04-26T01:41:47.499680+0000","last_clean":"2024-04-26T01:41:47.499680+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:41:47.499680+0000","last_undegraded":"2024-04-26T01:41:47.499680+0000","last_fullsized":"2024-04-26T01:41:47.499680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508517+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:41:47.508517+0000","last_peered":"2024-04-26T01:41:47.508517+0000","last_clean":"2024-04-26T01:41:47.508517+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:41:47.508517+0000","last_undegraded":"2024-04-26T01:41:47.508517+0000","last_fullsized":"2024-04-26T01:41:47.508517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499469+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:41:47.499469+0000","last_peered":"2024-04-26T01:41:47.499469+0000","last_clean":"2024-04-26T01:41:47.499469+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:41:47.499469+0000","last_undegraded":"2024-04-26T01:41:47.499469+0000","last_fullsized":"2024-04-26T01:41:47.499469+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495205+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:41:47.495205+0000","last_peered":"2024-04-26T01:41:47.495205+0000","last_clean":"2024-04-26T01:41:47.495205+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:41:47.495205+0000","last_undegraded":"2024-04-26T01:41:47.495205+0000","last_fullsized":"2024-04-26T01:41:47.495205+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499774+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:41:47.499774+0000","last_peered":"2024-04-26T01:41:47.499774+0000","last_clean":"2024-04-26T01:41:47.499774+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:41:47.499774+0000","last_undegraded":"2024-04-26T01:41:47.499774+0000","last_fullsized":"2024-04-26T01:41:47.499774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496746+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:41:47.496746+0000","last_peered":"2024-04-26T01:41:47.496746+0000","last_clean":"2024-04-26T01:41:47.496746+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:41:47.496746+0000","last_undegraded":"2024-04-26T01:41:47.496746+0000","last_fullsized":"2024-04-26T01:41:47.496746+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494606+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:41:47.494606+0000","last_peered":"2024-04-26T01:41:47.494606+0000","last_clean":"2024-04-26T01:41:47.494606+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:41:47.494606+0000","last_undegraded":"2024-04-26T01:41:47.494606+0000","last_fullsized":"2024-04-26T01:41:47.494606+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494578+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:41:47.494578+0000","last_peered":"2024-04-26T01:41:47.494578+0000","last_clean":"2024-04-26T01:41:47.494578+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:41:47.494578+0000","last_undegraded":"2024-04-26T01:41:47.494578+0000","last_fullsized":"2024-04-26T01:41:47.494578+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504187+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:41:47.504187+0000","last_peered":"2024-04-26T01:41:47.504187+0000","last_clean":"2024-04-26T01:41:47.504187+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:41:47.504187+0000","last_undegraded":"2024-04-26T01:41:47.504187+0000","last_fullsized":"2024-04-26T01:41:47.504187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495110+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:41:47.495110+0000","last_peered":"2024-04-26T01:41:47.495110+0000","last_clean":"2024-04-26T01:41:47.495110+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:41:47.495110+0000","last_undegraded":"2024-04-26T01:41:47.495110+0000","last_fullsized":"2024-04-26T01:41:47.495110+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497370+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:41:47.497370+0000","last_peered":"2024-04-26T01:41:47.497370+0000","last_clean":"2024-04-26T01:41:47.497370+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:41:47.497370+0000","last_undegraded":"2024-04-26T01:41:47.497370+0000","last_fullsized":"2024-04-26T01:41:47.497370+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495075+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:41:47.495075+0000","last_peered":"2024-04-26T01:41:47.495075+0000","last_clean":"2024-04-26T01:41:47.495075+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:41:47.495075+0000","last_undegraded":"2024-04-26T01:41:47.495075+0000","last_fullsized":"2024-04-26T01:41:47.495075+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509844+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:41:47.509844+0000","last_peered":"2024-04-26T01:41:47.509844+0000","last_clean":"2024-04-26T01:41:47.509844+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:41:47.509844+0000","last_undegraded":"2024-04-26T01:41:47.509844+0000","last_fullsized":"2024-04-26T01:41:47.509844+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508177+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:41:47.508177+0000","last_peered":"2024-04-26T01:41:47.508177+0000","last_clean":"2024-04-26T01:41:47.508177+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:41:47.508177+0000","last_undegraded":"2024-04-26T01:41:47.508177+0000","last_fullsized":"2024-04-26T01:41:47.508177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509234+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:41:47.509234+0000","last_peered":"2024-04-26T01:41:47.509234+0000","last_clean":"2024-04-26T01:41:47.509234+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:41:47.509234+0000","last_undegraded":"2024-04-26T01:41:47.509234+0000","last_fullsized":"2024-04-26T01:41:47.509234+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508921+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:41:47.508921+0000","last_peered":"2024-04-26T01:41:47.508921+0000","last_clean":"2024-04-26T01:41:47.508921+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:41:47.508921+0000","last_undegraded":"2024-04-26T01:41:47.508921+0000","last_fullsized":"2024-04-26T01:41:47.508921+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494966+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:41:47.494966+0000","last_peered":"2024-04-26T01:41:47.494966+0000","last_clean":"2024-04-26T01:41:47.494966+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:41:47.494966+0000","last_undegraded":"2024-04-26T01:41:47.494966+0000","last_fullsized":"2024-04-26T01:41:47.494966+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.502054+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:41:47.502054+0000","last_peered":"2024-04-26T01:41:47.502054+0000","last_clean":"2024-04-26T01:41:47.502054+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:41:47.502054+0000","last_undegraded":"2024-04-26T01:41:47.502054+0000","last_fullsized":"2024-04-26T01:41:47.502054+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67073,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":220,"num_read_kb":177,"num_write":166,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":201219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252893,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":497727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514531,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32856,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712296,"statfs":{"total":95995035648,"available":95961391104,"internally_reserved":0,"allocated":1720320,"data_stored":1152671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":497963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972398,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":497883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234034,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1630208,"data_stored":1089813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397560,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":497352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":560217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":498623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117064,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":497663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855430,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":498762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659199,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1642496,"data_stored":1089106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:42:02.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:42:03.203 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:03.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:03 smithi052 ceph-mon[27851]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-26T01:42:03.356 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:03 smithi052 ceph-mon[27851]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:03.356 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:03.356 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:03 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:03 smithi191 ceph-mon[32745]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-26T01:42:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:03 smithi191 ceph-mon[32745]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:03 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:03.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:03 smithi183 ceph-mon[32752]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.9 KiB/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-04-26T01:42:03.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:03 smithi183 ceph-mon[32752]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:03 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:04.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:04.423 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:04.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:42:04.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:42:04.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:42:05.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 -- 172.21.15.52:0/1393882865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98102300 msgr2=0x7fcb9810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:05.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 --2- 172.21.15.52:0/1393882865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98102300 0x7fcb9810e7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000b0a0 tx=0x7fcb9000b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 -- 172.21.15.52:0/1393882865 shutdown_connections 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 --2- 172.21.15.52:0/1393882865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98102300 0x7fcb9810e7f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 --2- 172.21.15.52:0/1393882865 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98107990 0x7fcb98107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 --2- 172.21.15.52:0/1393882865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98101940 0x7fcb98101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.447+0000 7fcba0400700 1 -- 172.21.15.52:0/1393882865 >> 172.21.15.52:0/1393882865 conn(0x7fcb980788f0 msgr2=0x7fcb98078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.448+0000 7fcba0400700 1 -- 172.21.15.52:0/1393882865 shutdown_connections 2024-04-26T01:42:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.448+0000 7fcba0400700 1 -- 172.21.15.52:0/1393882865 wait complete. 2024-04-26T01:42:05.446 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:05 smithi052 ceph-mon[27851]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:42:05.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 Processor -- start 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 -- start start 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98102300 0x7fcb981982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 0x7fcb98198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcb98111b30 con 0x7fcb98101940 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcb981119b0 con 0x7fcb98107990 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.449+0000 7fcba0400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcb98111cb0 con 0x7fcb98102300 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58186/0 (socket says 172.21.15.52:58186) 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 -- 172.21.15.52:0/1620078375 learned_addr learned my addr 172.21.15.52:0/1620078375 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9e400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 0x7fcb98198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:05.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9d000700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98102300 0x7fcb981982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 -- 172.21.15.52:0/1620078375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98102300 msgr2=0x7fcb981982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98102300 0x7fcb981982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 -- 172.21.15.52:0/1620078375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 msgr2=0x7fcb98198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 0x7fcb98198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9da00700 1 -- 172.21.15.52:0/1620078375 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb981a7620 con 0x7fcb98101940 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.450+0000 7fcb9e400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 0x7fcb98198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:05.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcb9da00700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00e9a0 tx=0x7fcb8c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00ca10 con 0x7fcb98101940 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c00cbb0 con 0x7fcb98101940 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb981a7910 con 0x7fcb98101940 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c018680 con 0x7fcb98101940 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.451+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb981a7e50 con 0x7fcb98101940 2024-04-26T01:42:05.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.452+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb98069730 con 0x7fcb98101940 2024-04-26T01:42:05.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.455+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcb8c010430 con 0x7fcb98101940 2024-04-26T01:42:05.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.456+0000 7fcb96000700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 0x7fcb800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:05.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.456+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fcb8c0d09c0 con 0x7fcb98101940 2024-04-26T01:42:05.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.457+0000 7fcb9d000700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 0x7fcb800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:05.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.457+0000 7fcb9d000700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 0x7fcb800a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb9806afa0 tx=0x7fcb84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:05.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.459+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb8c09d1f0 con 0x7fcb98101940 2024-04-26T01:42:05.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:05 smithi191 ceph-mon[32745]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:42:05.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:05 smithi183 ceph-mon[32752]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-26T01:42:05.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.684+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb98062e80 con 0x7fcb800a3cd0 2024-04-26T01:42:05.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.693+0000 7fcb96000700 1 -- 172.21.15.52:0/1620078375 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258737 (secure 0 0 0) 0x7fcb98062e80 con 0x7fcb800a3cd0 2024-04-26T01:42:05.697 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:42:05.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 msgr2=0x7fcb800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 0x7fcb800a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb9806afa0 tx=0x7fcb84009260 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 msgr2=0x7fcb98197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00e9a0 tx=0x7fcb8c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 shutdown_connections 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb800a3cd0 0x7fcb800a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb98107990 0x7fcb98198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb98101940 0x7fcb98197da0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 --2- 172.21.15.52:0/1620078375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb98102300 0x7fcb981982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 >> 172.21.15.52:0/1620078375 conn(0x7fcb980788f0 msgr2=0x7fcb980ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.701+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 shutdown_connections 2024-04-26T01:42:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:05.702+0000 7fcba0400700 1 -- 172.21.15.52:0/1620078375 wait complete. 2024-04-26T01:42:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:42:06.491 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-26T01:42:05.684728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657675,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":376,"num_read_kb":311,"num_write":349,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388236,"kb_used_data":14088,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553588,"statfs":{"total":1151940427776,"available":1151542874112,"internally_reserved":0,"allocated":14426112,"data_stored":7938765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8705,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":44,"num_write":10,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004639"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497027+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:41:47.497027+0000","last_peered":"2024-04-26T01:41:47.497027+0000","last_clean":"2024-04-26T01:41:47.497027+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:41:47.497027+0000","last_undegraded":"2024-04-26T01:41:47.497027+0000","last_fullsized":"2024-04-26T01:41:47.497027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504566+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:41:47.504566+0000","last_peered":"2024-04-26T01:41:47.504566+0000","last_clean":"2024-04-26T01:41:47.504566+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:41:47.504566+0000","last_undegraded":"2024-04-26T01:41:47.504566+0000","last_fullsized":"2024-04-26T01:41:47.504566+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504433+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:41:47.504433+0000","last_peered":"2024-04-26T01:41:47.504433+0000","last_clean":"2024-04-26T01:41:47.504433+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:41:47.504433+0000","last_undegraded":"2024-04-26T01:41:47.504433+0000","last_fullsized":"2024-04-26T01:41:47.504433+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503142+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:41:47.503142+0000","last_peered":"2024-04-26T01:41:47.503142+0000","last_clean":"2024-04-26T01:41:47.503142+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:41:47.503142+0000","last_undegraded":"2024-04-26T01:41:47.503142+0000","last_fullsized":"2024-04-26T01:41:47.503142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499067+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:41:47.499067+0000","last_peered":"2024-04-26T01:41:47.499067+0000","last_clean":"2024-04-26T01:41:47.499067+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:41:47.499067+0000","last_undegraded":"2024-04-26T01:41:47.499067+0000","last_fullsized":"2024-04-26T01:41:47.499067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508633+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:41:47.508633+0000","last_peered":"2024-04-26T01:41:47.508633+0000","last_clean":"2024-04-26T01:41:47.508633+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:41:47.508633+0000","last_undegraded":"2024-04-26T01:41:47.508633+0000","last_fullsized":"2024-04-26T01:41:47.508633+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499821+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:41:47.499821+0000","last_peered":"2024-04-26T01:41:47.499821+0000","last_clean":"2024-04-26T01:41:47.499821+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:41:47.499821+0000","last_undegraded":"2024-04-26T01:41:47.499821+0000","last_fullsized":"2024-04-26T01:41:47.499821+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509232+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:41:47.509232+0000","last_peered":"2024-04-26T01:41:47.509232+0000","last_clean":"2024-04-26T01:41:47.509232+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:41:47.509232+0000","last_undegraded":"2024-04-26T01:41:47.509232+0000","last_fullsized":"2024-04-26T01:41:47.509232+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509390+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:41:47.509390+0000","last_peered":"2024-04-26T01:41:47.509390+0000","last_clean":"2024-04-26T01:41:47.509390+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:41:47.509390+0000","last_undegraded":"2024-04-26T01:41:47.509390+0000","last_fullsized":"2024-04-26T01:41:47.509390+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508813+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:41:47.508813+0000","last_peered":"2024-04-26T01:41:47.508813+0000","last_clean":"2024-04-26T01:41:47.508813+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:41:47.508813+0000","last_undegraded":"2024-04-26T01:41:47.508813+0000","last_fullsized":"2024-04-26T01:41:47.508813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499519+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:41:47.499519+0000","last_peered":"2024-04-26T01:41:47.499519+0000","last_clean":"2024-04-26T01:41:47.499519+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:41:47.499519+0000","last_undegraded":"2024-04-26T01:41:47.499519+0000","last_fullsized":"2024-04-26T01:41:47.499519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497259+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:41:47.497259+0000","last_peered":"2024-04-26T01:41:47.497259+0000","last_clean":"2024-04-26T01:41:47.497259+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:41:47.497259+0000","last_undegraded":"2024-04-26T01:41:47.497259+0000","last_fullsized":"2024-04-26T01:41:47.497259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494766+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:41:47.494766+0000","last_peered":"2024-04-26T01:41:47.494766+0000","last_clean":"2024-04-26T01:41:47.494766+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:41:47.494766+0000","last_undegraded":"2024-04-26T01:41:47.494766+0000","last_fullsized":"2024-04-26T01:41:47.494766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503984+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:41:47.503984+0000","last_peered":"2024-04-26T01:41:47.503984+0000","last_clean":"2024-04-26T01:41:47.503984+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:41:47.503984+0000","last_undegraded":"2024-04-26T01:41:47.503984+0000","last_fullsized":"2024-04-26T01:41:47.503984+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.503750+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:41:47.503750+0000","last_peered":"2024-04-26T01:41:47.503750+0000","last_clean":"2024-04-26T01:41:47.503750+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:41:47.503750+0000","last_undegraded":"2024-04-26T01:41:47.503750+0000","last_fullsized":"2024-04-26T01:41:47.503750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508483+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:41:47.508483+0000","last_peered":"2024-04-26T01:41:47.508483+0000","last_clean":"2024-04-26T01:41:47.508483+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:41:47.508483+0000","last_undegraded":"2024-04-26T01:41:47.508483+0000","last_fullsized":"2024-04-26T01:41:47.508483+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497244+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:41:47.497244+0000","last_peered":"2024-04-26T01:41:47.497244+0000","last_clean":"2024-04-26T01:41:47.497244+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:41:47.497244+0000","last_undegraded":"2024-04-26T01:41:47.497244+0000","last_fullsized":"2024-04-26T01:41:47.497244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498488+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:41:47.498488+0000","last_peered":"2024-04-26T01:41:47.498488+0000","last_clean":"2024-04-26T01:41:47.498488+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:41:47.498488+0000","last_undegraded":"2024-04-26T01:41:47.498488+0000","last_fullsized":"2024-04-26T01:41:47.498488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494907+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:41:47.494907+0000","last_peered":"2024-04-26T01:41:47.494907+0000","last_clean":"2024-04-26T01:41:47.494907+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:41:47.494907+0000","last_undegraded":"2024-04-26T01:41:47.494907+0000","last_fullsized":"2024-04-26T01:41:47.494907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498685+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:41:47.498685+0000","last_peered":"2024-04-26T01:41:47.498685+0000","last_clean":"2024-04-26T01:41:47.498685+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:41:47.498685+0000","last_undegraded":"2024-04-26T01:41:47.498685+0000","last_fullsized":"2024-04-26T01:41:47.498685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510113+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:41:47.510113+0000","last_peered":"2024-04-26T01:41:47.510113+0000","last_clean":"2024-04-26T01:41:47.510113+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:41:47.510113+0000","last_undegraded":"2024-04-26T01:41:47.510113+0000","last_fullsized":"2024-04-26T01:41:47.510113+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510446+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:41:47.510446+0000","last_peered":"2024-04-26T01:41:47.510446+0000","last_clean":"2024-04-26T01:41:47.510446+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:41:47.510446+0000","last_undegraded":"2024-04-26T01:41:47.510446+0000","last_fullsized":"2024-04-26T01:41:47.510446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504783+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:41:47.504783+0000","last_peered":"2024-04-26T01:41:47.504783+0000","last_clean":"2024-04-26T01:41:47.504783+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:41:47.504783+0000","last_undegraded":"2024-04-26T01:41:47.504783+0000","last_fullsized":"2024-04-26T01:41:47.504783+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.506427+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:41:47.506427+0000","last_peered":"2024-04-26T01:41:47.506427+0000","last_clean":"2024-04-26T01:41:47.506427+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:41:47.506427+0000","last_undegraded":"2024-04-26T01:41:47.506427+0000","last_fullsized":"2024-04-26T01:41:47.506427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508949+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:41:47.508949+0000","last_peered":"2024-04-26T01:41:47.508949+0000","last_clean":"2024-04-26T01:41:47.508949+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:41:47.508949+0000","last_undegraded":"2024-04-26T01:41:47.508949+0000","last_fullsized":"2024-04-26T01:41:47.508949+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.500059+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:41:47.500059+0000","last_peered":"2024-04-26T01:41:47.500059+0000","last_clean":"2024-04-26T01:41:47.500059+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:41:47.500059+0000","last_undegraded":"2024-04-26T01:41:47.500059+0000","last_fullsized":"2024-04-26T01:41:47.500059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508654+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:41:47.508654+0000","last_peered":"2024-04-26T01:41:47.508654+0000","last_clean":"2024-04-26T01:41:47.508654+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:41:47.508654+0000","last_undegraded":"2024-04-26T01:41:47.508654+0000","last_fullsized":"2024-04-26T01:41:47.508654+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:42:00.453918+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:42:00.453918+0000","last_peered":"2024-04-26T01:42:00.453918+0000","last_clean":"2024-04-26T01:42:00.453918+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:42:00.453918+0000","last_undegraded":"2024-04-26T01:42:00.453918+0000","last_fullsized":"2024-04-26T01:42:00.453918+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499158+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:41:47.499158+0000","last_peered":"2024-04-26T01:41:47.499158+0000","last_clean":"2024-04-26T01:41:47.499158+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:41:47.499158+0000","last_undegraded":"2024-04-26T01:41:47.499158+0000","last_fullsized":"2024-04-26T01:41:47.499158+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508594+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:41:47.508594+0000","last_peered":"2024-04-26T01:41:47.508594+0000","last_clean":"2024-04-26T01:41:47.508594+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:41:47.508594+0000","last_undegraded":"2024-04-26T01:41:47.508594+0000","last_fullsized":"2024-04-26T01:41:47.508594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508357+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:41:47.508357+0000","last_peered":"2024-04-26T01:41:47.508357+0000","last_clean":"2024-04-26T01:41:47.508357+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:41:47.508357+0000","last_undegraded":"2024-04-26T01:41:47.508357+0000","last_fullsized":"2024-04-26T01:41:47.508357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494651+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:41:47.494651+0000","last_peered":"2024-04-26T01:41:47.494651+0000","last_clean":"2024-04-26T01:41:47.494651+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:41:47.494651+0000","last_undegraded":"2024-04-26T01:41:47.494651+0000","last_fullsized":"2024-04-26T01:41:47.494651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509161+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:41:47.509161+0000","last_peered":"2024-04-26T01:41:47.509161+0000","last_clean":"2024-04-26T01:41:47.509161+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:41:47.509161+0000","last_undegraded":"2024-04-26T01:41:47.509161+0000","last_fullsized":"2024-04-26T01:41:47.509161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508108+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:41:47.508108+0000","last_peered":"2024-04-26T01:41:47.508108+0000","last_clean":"2024-04-26T01:41:47.508108+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:41:47.508108+0000","last_undegraded":"2024-04-26T01:41:47.508108+0000","last_fullsized":"2024-04-26T01:41:47.508108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510772+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:41:47.510772+0000","last_peered":"2024-04-26T01:41:47.510772+0000","last_clean":"2024-04-26T01:41:47.510772+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:41:47.510772+0000","last_undegraded":"2024-04-26T01:41:47.510772+0000","last_fullsized":"2024-04-26T01:41:47.510772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:42:01.454536+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:42:01.454536+0000","last_peered":"2024-04-26T01:42:01.454536+0000","last_clean":"2024-04-26T01:42:01.454536+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:42:01.454536+0000","last_undegraded":"2024-04-26T01:42:01.454536+0000","last_fullsized":"2024-04-26T01:42:01.454536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509329+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:41:47.509329+0000","last_peered":"2024-04-26T01:41:47.509329+0000","last_clean":"2024-04-26T01:41:47.509329+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:41:47.509329+0000","last_undegraded":"2024-04-26T01:41:47.509329+0000","last_fullsized":"2024-04-26T01:41:47.509329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499597+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:41:47.499597+0000","last_peered":"2024-04-26T01:41:47.499597+0000","last_clean":"2024-04-26T01:41:47.499597+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:41:47.499597+0000","last_undegraded":"2024-04-26T01:41:47.499597+0000","last_fullsized":"2024-04-26T01:41:47.499597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.505535+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:41:47.505535+0000","last_peered":"2024-04-26T01:41:47.505535+0000","last_clean":"2024-04-26T01:41:47.505535+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:41:47.505535+0000","last_undegraded":"2024-04-26T01:41:47.505535+0000","last_fullsized":"2024-04-26T01:41:47.505535+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508753+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:41:47.508753+0000","last_peered":"2024-04-26T01:41:47.508753+0000","last_clean":"2024-04-26T01:41:47.508753+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:41:47.508753+0000","last_undegraded":"2024-04-26T01:41:47.508753+0000","last_fullsized":"2024-04-26T01:41:47.508753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509086+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:41:47.509086+0000","last_peered":"2024-04-26T01:41:47.509086+0000","last_clean":"2024-04-26T01:41:47.509086+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:41:47.509086+0000","last_undegraded":"2024-04-26T01:41:47.509086+0000","last_fullsized":"2024-04-26T01:41:47.509086+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'30","reported_seq":136,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:42:02.455796+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:42:02.455796+0000","last_peered":"2024-04-26T01:42:02.455796+0000","last_clean":"2024-04-26T01:42:02.455796+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:42:02.455796+0000","last_undegraded":"2024-04-26T01:42:02.455796+0000","last_fullsized":"2024-04-26T01:42:02.455796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63072,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":80,"num_read_kb":101,"num_write":31,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499210+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:41:47.499210+0000","last_peered":"2024-04-26T01:41:47.499210+0000","last_clean":"2024-04-26T01:41:47.499210+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:41:47.499210+0000","last_undegraded":"2024-04-26T01:41:47.499210+0000","last_fullsized":"2024-04-26T01:41:47.499210+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494848+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:41:47.494848+0000","last_peered":"2024-04-26T01:41:47.494848+0000","last_clean":"2024-04-26T01:41:47.494848+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:41:47.494848+0000","last_undegraded":"2024-04-26T01:41:47.494848+0000","last_fullsized":"2024-04-26T01:41:47.494848+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494661+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:41:47.494661+0000","last_peered":"2024-04-26T01:41:47.494661+0000","last_clean":"2024-04-26T01:41:47.494661+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:41:47.494661+0000","last_undegraded":"2024-04-26T01:41:47.494661+0000","last_fullsized":"2024-04-26T01:41:47.494661+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508326+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:41:47.508326+0000","last_peered":"2024-04-26T01:41:47.508326+0000","last_clean":"2024-04-26T01:41:47.508326+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:41:47.508326+0000","last_undegraded":"2024-04-26T01:41:47.508326+0000","last_fullsized":"2024-04-26T01:41:47.508326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499423+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:41:47.499423+0000","last_peered":"2024-04-26T01:41:47.499423+0000","last_clean":"2024-04-26T01:41:47.499423+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:41:47.499423+0000","last_undegraded":"2024-04-26T01:41:47.499423+0000","last_fullsized":"2024-04-26T01:41:47.499423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'4","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:59.477767+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:41:59.477767+0000","last_peered":"2024-04-26T01:41:59.477767+0000","last_clean":"2024-04-26T01:41:59.477767+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:41:59.477767+0000","last_undegraded":"2024-04-26T01:41:59.477767+0000","last_fullsized":"2024-04-26T01:41:59.477767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499162+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:41:47.499162+0000","last_peered":"2024-04-26T01:41:47.499162+0000","last_clean":"2024-04-26T01:41:47.499162+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:41:47.499162+0000","last_undegraded":"2024-04-26T01:41:47.499162+0000","last_fullsized":"2024-04-26T01:41:47.499162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498976+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:41:47.498976+0000","last_peered":"2024-04-26T01:41:47.498976+0000","last_clean":"2024-04-26T01:41:47.498976+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:41:47.498976+0000","last_undegraded":"2024-04-26T01:41:47.498976+0000","last_fullsized":"2024-04-26T01:41:47.498976+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.511194+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:41:47.511194+0000","last_peered":"2024-04-26T01:41:47.511194+0000","last_clean":"2024-04-26T01:41:47.511194+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:41:47.511194+0000","last_undegraded":"2024-04-26T01:41:47.511194+0000","last_fullsized":"2024-04-26T01:41:47.511194+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499598+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:41:47.499598+0000","last_peered":"2024-04-26T01:41:47.499598+0000","last_clean":"2024-04-26T01:41:47.499598+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:41:47.499598+0000","last_undegraded":"2024-04-26T01:41:47.499598+0000","last_fullsized":"2024-04-26T01:41:47.499598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496846+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:41:47.496846+0000","last_peered":"2024-04-26T01:41:47.496846+0000","last_clean":"2024-04-26T01:41:47.496846+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:41:47.496846+0000","last_undegraded":"2024-04-26T01:41:47.496846+0000","last_fullsized":"2024-04-26T01:41:47.496846+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508155+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:41:47.508155+0000","last_peered":"2024-04-26T01:41:47.508155+0000","last_clean":"2024-04-26T01:41:47.508155+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:41:47.508155+0000","last_undegraded":"2024-04-26T01:41:47.508155+0000","last_fullsized":"2024-04-26T01:41:47.508155+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509597+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:41:47.509597+0000","last_peered":"2024-04-26T01:41:47.509597+0000","last_clean":"2024-04-26T01:41:47.509597+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:41:47.509597+0000","last_undegraded":"2024-04-26T01:41:47.509597+0000","last_fullsized":"2024-04-26T01:41:47.509597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499417+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:41:47.499417+0000","last_peered":"2024-04-26T01:41:47.499417+0000","last_clean":"2024-04-26T01:41:47.499417+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:41:47.499417+0000","last_undegraded":"2024-04-26T01:41:47.499417+0000","last_fullsized":"2024-04-26T01:41:47.499417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508774+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:41:47.508774+0000","last_peered":"2024-04-26T01:41:47.508774+0000","last_clean":"2024-04-26T01:41:47.508774+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:41:47.508774+0000","last_undegraded":"2024-04-26T01:41:47.508774+0000","last_fullsized":"2024-04-26T01:41:47.508774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508586+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:41:47.508586+0000","last_peered":"2024-04-26T01:41:47.508586+0000","last_clean":"2024-04-26T01:41:47.508586+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:41:47.508586+0000","last_undegraded":"2024-04-26T01:41:47.508586+0000","last_fullsized":"2024-04-26T01:41:47.508586+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'4","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:42:02.482979+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:42:02.482979+0000","last_peered":"2024-04-26T01:42:02.482979+0000","last_clean":"2024-04-26T01:42:02.482979+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:42:02.482979+0000","last_undegraded":"2024-04-26T01:42:02.482979+0000","last_fullsized":"2024-04-26T01:42:02.482979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509636+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:41:47.509636+0000","last_peered":"2024-04-26T01:41:47.509636+0000","last_clean":"2024-04-26T01:41:47.509636+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:41:47.509636+0000","last_undegraded":"2024-04-26T01:41:47.509636+0000","last_fullsized":"2024-04-26T01:41:47.509636+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497435+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:41:47.497435+0000","last_peered":"2024-04-26T01:41:47.497435+0000","last_clean":"2024-04-26T01:41:47.497435+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:41:47.497435+0000","last_undegraded":"2024-04-26T01:41:47.497435+0000","last_fullsized":"2024-04-26T01:41:47.497435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499373+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:41:47.499373+0000","last_peered":"2024-04-26T01:41:47.499373+0000","last_clean":"2024-04-26T01:41:47.499373+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:41:47.499373+0000","last_undegraded":"2024-04-26T01:41:47.499373+0000","last_fullsized":"2024-04-26T01:41:47.499373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494519+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:41:47.494519+0000","last_peered":"2024-04-26T01:41:47.494519+0000","last_clean":"2024-04-26T01:41:47.494519+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:41:47.494519+0000","last_undegraded":"2024-04-26T01:41:47.494519+0000","last_fullsized":"2024-04-26T01:41:47.494519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499489+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:41:47.499489+0000","last_peered":"2024-04-26T01:41:47.499489+0000","last_clean":"2024-04-26T01:41:47.499489+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:41:47.499489+0000","last_undegraded":"2024-04-26T01:41:47.499489+0000","last_fullsized":"2024-04-26T01:41:47.499489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499392+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:41:47.499392+0000","last_peered":"2024-04-26T01:41:47.499392+0000","last_clean":"2024-04-26T01:41:47.499392+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:41:47.499392+0000","last_undegraded":"2024-04-26T01:41:47.499392+0000","last_fullsized":"2024-04-26T01:41:47.499392+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495185+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:41:47.495185+0000","last_peered":"2024-04-26T01:41:47.495185+0000","last_clean":"2024-04-26T01:41:47.495185+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:41:47.495185+0000","last_undegraded":"2024-04-26T01:41:47.495185+0000","last_fullsized":"2024-04-26T01:41:47.495185+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'12","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:52.316162+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:41:52.316162+0000","last_peered":"2024-04-26T01:41:52.316162+0000","last_clean":"2024-04-26T01:41:52.316162+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:41:52.316162+0000","last_undegraded":"2024-04-26T01:41:52.316162+0000","last_fullsized":"2024-04-26T01:41:52.316162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509948+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:41:47.509948+0000","last_peered":"2024-04-26T01:41:47.509948+0000","last_clean":"2024-04-26T01:41:47.509948+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:41:47.509948+0000","last_undegraded":"2024-04-26T01:41:47.509948+0000","last_fullsized":"2024-04-26T01:41:47.509948+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497049+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:41:47.497049+0000","last_peered":"2024-04-26T01:41:47.497049+0000","last_clean":"2024-04-26T01:41:47.497049+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:41:47.497049+0000","last_undegraded":"2024-04-26T01:41:47.497049+0000","last_fullsized":"2024-04-26T01:41:47.497049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504585+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:41:47.504585+0000","last_peered":"2024-04-26T01:41:47.504585+0000","last_clean":"2024-04-26T01:41:47.504585+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:41:47.504585+0000","last_undegraded":"2024-04-26T01:41:47.504585+0000","last_fullsized":"2024-04-26T01:41:47.504585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508399+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:41:47.508399+0000","last_peered":"2024-04-26T01:41:47.508399+0000","last_clean":"2024-04-26T01:41:47.508399+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:41:47.508399+0000","last_undegraded":"2024-04-26T01:41:47.508399+0000","last_fullsized":"2024-04-26T01:41:47.508399+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499009+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:41:47.499009+0000","last_peered":"2024-04-26T01:41:47.499009+0000","last_clean":"2024-04-26T01:41:47.499009+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:41:47.499009+0000","last_undegraded":"2024-04-26T01:41:47.499009+0000","last_fullsized":"2024-04-26T01:41:47.499009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509372+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:41:47.509372+0000","last_peered":"2024-04-26T01:41:47.509372+0000","last_clean":"2024-04-26T01:41:47.509372+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:41:47.509372+0000","last_undegraded":"2024-04-26T01:41:47.509372+0000","last_fullsized":"2024-04-26T01:41:47.509372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496629+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:41:47.496629+0000","last_peered":"2024-04-26T01:41:47.496629+0000","last_clean":"2024-04-26T01:41:47.496629+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:41:47.496629+0000","last_undegraded":"2024-04-26T01:41:47.496629+0000","last_fullsized":"2024-04-26T01:41:47.496629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494637+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:41:47.494637+0000","last_peered":"2024-04-26T01:41:47.494637+0000","last_clean":"2024-04-26T01:41:47.494637+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:41:47.494637+0000","last_undegraded":"2024-04-26T01:41:47.494637+0000","last_fullsized":"2024-04-26T01:41:47.494637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510042+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:41:47.510042+0000","last_peered":"2024-04-26T01:41:47.510042+0000","last_clean":"2024-04-26T01:41:47.510042+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:41:47.510042+0000","last_undegraded":"2024-04-26T01:41:47.510042+0000","last_fullsized":"2024-04-26T01:41:47.510042+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499216+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:41:47.499216+0000","last_peered":"2024-04-26T01:41:47.499216+0000","last_clean":"2024-04-26T01:41:47.499216+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:41:47.499216+0000","last_undegraded":"2024-04-26T01:41:47.499216+0000","last_fullsized":"2024-04-26T01:41:47.499216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499650+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:41:47.499650+0000","last_peered":"2024-04-26T01:41:47.499650+0000","last_clean":"2024-04-26T01:41:47.499650+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:41:47.499650+0000","last_undegraded":"2024-04-26T01:41:47.499650+0000","last_fullsized":"2024-04-26T01:41:47.499650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.513560+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:41:47.513560+0000","last_peered":"2024-04-26T01:41:47.513560+0000","last_clean":"2024-04-26T01:41:47.513560+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:41:47.513560+0000","last_undegraded":"2024-04-26T01:41:47.513560+0000","last_fullsized":"2024-04-26T01:41:47.513560+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499680+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:41:47.499680+0000","last_peered":"2024-04-26T01:41:47.499680+0000","last_clean":"2024-04-26T01:41:47.499680+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:41:47.499680+0000","last_undegraded":"2024-04-26T01:41:47.499680+0000","last_fullsized":"2024-04-26T01:41:47.499680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508517+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:41:47.508517+0000","last_peered":"2024-04-26T01:41:47.508517+0000","last_clean":"2024-04-26T01:41:47.508517+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:41:47.508517+0000","last_undegraded":"2024-04-26T01:41:47.508517+0000","last_fullsized":"2024-04-26T01:41:47.508517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499469+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:41:47.499469+0000","last_peered":"2024-04-26T01:41:47.499469+0000","last_clean":"2024-04-26T01:41:47.499469+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:41:47.499469+0000","last_undegraded":"2024-04-26T01:41:47.499469+0000","last_fullsized":"2024-04-26T01:41:47.499469+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495205+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:41:47.495205+0000","last_peered":"2024-04-26T01:41:47.495205+0000","last_clean":"2024-04-26T01:41:47.495205+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:41:47.495205+0000","last_undegraded":"2024-04-26T01:41:47.495205+0000","last_fullsized":"2024-04-26T01:41:47.495205+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499774+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:41:47.499774+0000","last_peered":"2024-04-26T01:41:47.499774+0000","last_clean":"2024-04-26T01:41:47.499774+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:41:47.499774+0000","last_undegraded":"2024-04-26T01:41:47.499774+0000","last_fullsized":"2024-04-26T01:41:47.499774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.496746+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:41:47.496746+0000","last_peered":"2024-04-26T01:41:47.496746+0000","last_clean":"2024-04-26T01:41:47.496746+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:41:47.496746+0000","last_undegraded":"2024-04-26T01:41:47.496746+0000","last_fullsized":"2024-04-26T01:41:47.496746+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494606+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:41:47.494606+0000","last_peered":"2024-04-26T01:41:47.494606+0000","last_clean":"2024-04-26T01:41:47.494606+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:41:47.494606+0000","last_undegraded":"2024-04-26T01:41:47.494606+0000","last_fullsized":"2024-04-26T01:41:47.494606+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494578+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:41:47.494578+0000","last_peered":"2024-04-26T01:41:47.494578+0000","last_clean":"2024-04-26T01:41:47.494578+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:41:47.494578+0000","last_undegraded":"2024-04-26T01:41:47.494578+0000","last_fullsized":"2024-04-26T01:41:47.494578+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.504187+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:41:47.504187+0000","last_peered":"2024-04-26T01:41:47.504187+0000","last_clean":"2024-04-26T01:41:47.504187+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:41:47.504187+0000","last_undegraded":"2024-04-26T01:41:47.504187+0000","last_fullsized":"2024-04-26T01:41:47.504187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495110+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:41:47.495110+0000","last_peered":"2024-04-26T01:41:47.495110+0000","last_clean":"2024-04-26T01:41:47.495110+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:41:47.495110+0000","last_undegraded":"2024-04-26T01:41:47.495110+0000","last_fullsized":"2024-04-26T01:41:47.495110+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497370+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:41:47.497370+0000","last_peered":"2024-04-26T01:41:47.497370+0000","last_clean":"2024-04-26T01:41:47.497370+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:41:47.497370+0000","last_undegraded":"2024-04-26T01:41:47.497370+0000","last_fullsized":"2024-04-26T01:41:47.497370+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495075+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:41:47.495075+0000","last_peered":"2024-04-26T01:41:47.495075+0000","last_clean":"2024-04-26T01:41:47.495075+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:41:47.495075+0000","last_undegraded":"2024-04-26T01:41:47.495075+0000","last_fullsized":"2024-04-26T01:41:47.495075+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509844+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:41:47.509844+0000","last_peered":"2024-04-26T01:41:47.509844+0000","last_clean":"2024-04-26T01:41:47.509844+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:41:47.509844+0000","last_undegraded":"2024-04-26T01:41:47.509844+0000","last_fullsized":"2024-04-26T01:41:47.509844+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508177+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:41:47.508177+0000","last_peered":"2024-04-26T01:41:47.508177+0000","last_clean":"2024-04-26T01:41:47.508177+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:41:47.508177+0000","last_undegraded":"2024-04-26T01:41:47.508177+0000","last_fullsized":"2024-04-26T01:41:47.508177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509234+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:41:47.509234+0000","last_peered":"2024-04-26T01:41:47.509234+0000","last_clean":"2024-04-26T01:41:47.509234+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:41:47.509234+0000","last_undegraded":"2024-04-26T01:41:47.509234+0000","last_fullsized":"2024-04-26T01:41:47.509234+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508921+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:41:47.508921+0000","last_peered":"2024-04-26T01:41:47.508921+0000","last_clean":"2024-04-26T01:41:47.508921+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:41:47.508921+0000","last_undegraded":"2024-04-26T01:41:47.508921+0000","last_fullsized":"2024-04-26T01:41:47.508921+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494966+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:41:47.494966+0000","last_peered":"2024-04-26T01:41:47.494966+0000","last_clean":"2024-04-26T01:41:47.494966+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:41:47.494966+0000","last_undegraded":"2024-04-26T01:41:47.494966+0000","last_fullsized":"2024-04-26T01:41:47.494966+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.502054+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:41:47.502054+0000","last_peered":"2024-04-26T01:41:47.502054+0000","last_clean":"2024-04-26T01:41:47.502054+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:41:47.502054+0000","last_undegraded":"2024-04-26T01:41:47.502054+0000","last_fullsized":"2024-04-26T01:41:47.502054+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67073,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":240,"num_read_kb":192,"num_write":166,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":201219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252894,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":497727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32856,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712296,"statfs":{"total":95995035648,"available":95961391104,"internally_reserved":0,"allocated":1720320,"data_stored":1152671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":497963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972399,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":497883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1630208,"data_stored":1089813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397561,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":497352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":560217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":498623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":497663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855430,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":498762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659199,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1642496,"data_stored":1089106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:42:06.497 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:42:06.497 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T01:42:06.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T01:42:07.007 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:07 smithi052 ceph-mon[27851]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:42:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:07 smithi052 ceph-mon[27851]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:07 smithi191 ceph-mon[32745]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:42:07.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:07 smithi191 ceph-mon[32745]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:07 smithi183 ceph-mon[32752]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:42:07.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:07 smithi183 ceph-mon[32752]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:42:09.423 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:09 smithi052 ceph-mon[27851]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 -- 172.21.15.52:0/464150903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35701019c0 msgr2=0x7f3570101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 --2- 172.21.15.52:0/464150903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35701019c0 0x7f3570101de0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f35600099c0 tx=0x7f3560009cd0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 -- 172.21.15.52:0/464150903 shutdown_connections 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 --2- 172.21.15.52:0/464150903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3570103560 0x7f3570109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 --2- 172.21.15.52:0/464150903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35701019c0 0x7f3570101de0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 --2- 172.21.15.52:0/464150903 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 0x7f3570103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.447+0000 7f3576e00700 1 -- 172.21.15.52:0/464150903 >> 172.21.15.52:0/464150903 conn(0x7f35700fcf60 msgr2=0x7f35700ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.448+0000 7f3576e00700 1 -- 172.21.15.52:0/464150903 shutdown_connections 2024-04-26T01:42:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.448+0000 7f3576e00700 1 -- 172.21.15.52:0/464150903 wait complete. 2024-04-26T01:42:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 Processor -- start 2024-04-26T01:42:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 -- start start 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35701019c0 0x7f357019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 0x7f357019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f357010d450 con 0x7f3570103560 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f357010d2d0 con 0x7f3570102bc0 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.449+0000 7f3576e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f357010d5d0 con 0x7f35701019c0 2024-04-26T01:42:09.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58198/0 (socket says 172.21.15.52:58198) 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 -- 172.21.15.52:0/1686980323 learned_addr learned my addr 172.21.15.52:0/1686980323 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f356fe00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35701019c0 0x7f357019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f356f400700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 0x7f357019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 -- 172.21.15.52:0/1686980323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35701019c0 msgr2=0x7f357019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35701019c0 0x7f357019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 -- 172.21.15.52:0/1686980323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 msgr2=0x7f357019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 0x7f357019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 -- 172.21.15.52:0/1686980323 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35701a29b0 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.450+0000 7f3574e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f356800ef60 tx=0x7f356800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.451+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35680092e0 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.451+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35701a2c40 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.451+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3568004bb0 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.451+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568004020 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.451+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35701a3180 con 0x7f3570103560 2024-04-26T01:42:09.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.452+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3570050f80 con 0x7f3570103560 2024-04-26T01:42:09.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.454+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f356801b030 con 0x7f3570103560 2024-04-26T01:42:09.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.455+0000 7f356cc00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 0x7f35540a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:09.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.456+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f35680cff60 con 0x7f3570103560 2024-04-26T01:42:09.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.456+0000 7f356fe00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 0x7f35540a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:09.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.457+0000 7f356fe00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 0x7f35540a62c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f35600036f0 tx=0x7f3560005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:09.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.461+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3568022020 con 0x7f3570103560 2024-04-26T01:42:09.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:09 smithi191 ceph-mon[32745]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:09.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:09 smithi183 ceph-mon[32752]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-26T01:42:09.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.789+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f35701a35b0 con 0x7f3570103560 2024-04-26T01:42:09.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.794+0000 7f356cc00700 1 -- 172.21.15.52:0/1686980323 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3568010040 con 0x7f3570103560 2024-04-26T01:42:09.791 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-04-26T01:42:09.791 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBxBitmHcMmLxAAKdXYP3Vecbl0txUOQ1hx0A== 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 msgr2=0x7f35540a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 0x7f35540a62c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f35600036f0 tx=0x7f3560005fb0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 msgr2=0x7f35701a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f356800ef60 tx=0x7f356800ef90 comp rx=0 tx=0).stop 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 shutdown_connections 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f35540a3e00 0x7f35540a62c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3570103560 0x7f35701a2310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35701019c0 0x7f357019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 --2- 172.21.15.52:0/1686980323 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3570102bc0 0x7f357019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:09.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 >> 172.21.15.52:0/1686980323 conn(0x7f35700fcf60 msgr2=0x7f35700fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:09.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 shutdown_connections 2024-04-26T01:42:09.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:09.797+0000 7f3576e00700 1 -- 172.21.15.52:0/1686980323 wait complete. 2024-04-26T01:42:10.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:10 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:42:10.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:10 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:42:10.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:10 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:42:10.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:10 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:42:10.674 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:42:10.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T01:42:10.674 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T01:42:10.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:10 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T01:42:10.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:10 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1686980323' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T01:42:10.823 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T01:42:10.823 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T01:42:10.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T01:42:11.347 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:11.557 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:11 smithi052 ceph-mon[27851]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:11.575 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:11 smithi183 ceph-mon[32752]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:11.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:11 smithi191 ceph-mon[32745]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:13.384 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:13 smithi052 ceph-mon[27851]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 -- 172.21.15.52:0/3265396460 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 msgr2=0x7f097410f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 --2- 172.21.15.52:0/3265396460 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097410f490 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f096c00b0a0 tx=0x7f096c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 -- 172.21.15.52:0/3265396460 shutdown_connections 2024-04-26T01:42:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 --2- 172.21.15.52:0/3265396460 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097410f490 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 --2- 172.21.15.52:0/3265396460 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 0x7f0974103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 --2- 172.21.15.52:0/3265396460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 0x7f097410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 -- 172.21.15.52:0/3265396460 >> 172.21.15.52:0/3265396460 conn(0x7f09740fd3f0 msgr2=0x7f09740ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 -- 172.21.15.52:0/3265396460 shutdown_connections 2024-04-26T01:42:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.561+0000 7f0973200700 1 -- 172.21.15.52:0/3265396460 wait complete. 2024-04-26T01:42:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.562+0000 7f0973200700 1 Processor -- start 2024-04-26T01:42:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 -- start start 2024-04-26T01:42:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 0x7f097419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 0x7f097419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0974112be0 con 0x7f097410cfd0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0974112a60 con 0x7f09741044c0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0973200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0974112d60 con 0x7f0974103ba0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.563+0000 7f0972800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58224/0 (socket says 172.21.15.52:58224) 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 -- 172.21.15.52:0/4120465131 learned_addr learned my addr 172.21.15.52:0/4120465131 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0971e00700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 0x7f097419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0971400700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 0x7f097419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 -- 172.21.15.52:0/4120465131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 msgr2=0x7f097419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 0x7f097419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 -- 172.21.15.52:0/4120465131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 msgr2=0x7f097419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 0x7f097419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 -- 172.21.15.52:0/4120465131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09741a02c0 con 0x7f097410cfd0 2024-04-26T01:42:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.564+0000 7f0972800700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f096c00b780 tx=0x7f096c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.565+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c021070 con 0x7f097410cfd0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.565+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096c007e30 con 0x7f097410cfd0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.565+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c004700 con 0x7f097410cfd0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.565+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09741a0550 con 0x7f097410cfd0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.565+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09741a7e30 con 0x7f097410cfd0 2024-04-26T01:42:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.566+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097406b890 con 0x7f097410cfd0 2024-04-26T01:42:13.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.568+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f096c007980 con 0x7f097410cfd0 2024-04-26T01:42:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.570+0000 7f096aa00700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 0x7f09580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.570+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f096c00f020 con 0x7f097410cfd0 2024-04-26T01:42:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.570+0000 7f0971e00700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 0x7f09580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:13.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.571+0000 7f0971e00700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 0x7f09580a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0964009c60 tx=0x7f0964009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:13.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.575+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f096c02a020 con 0x7f097410cfd0 2024-04-26T01:42:13.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:13 smithi191 ceph-mon[32745]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:13.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:13 smithi183 ceph-mon[32752]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:13.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.876+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f09741a8000 con 0x7f097410cfd0 2024-04-26T01:42:13.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.892+0000 7f096aa00700 1 -- 172.21.15.52:0/4120465131 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7f096c09e100 con 0x7f097410cfd0 2024-04-26T01:42:13.891 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:42:13.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 msgr2=0x7f09580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 0x7f09580a6210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0964009c60 tx=0x7f0964009260 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 msgr2=0x7f097419fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f096c00b780 tx=0x7f096c004d00 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 shutdown_connections 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f09580a3d50 0x7f09580a6210 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f097410cfd0 0x7f097419fb80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0974103ba0 0x7f097419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 --2- 172.21.15.52:0/4120465131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09741044c0 0x7f097419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 >> 172.21.15.52:0/4120465131 conn(0x7f09740fd3f0 msgr2=0x7f09740ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.900+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 shutdown_connections 2024-04-26T01:42:13.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:13.901+0000 7f0973200700 1 -- 172.21.15.52:0/4120465131 wait complete. 2024-04-26T01:42:14.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:14 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4120465131' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:42:14.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:14 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4120465131' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:42:14.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:14 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4120465131' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T01:42:14.880 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":2415060450},{"type":"v1","addr":"172.21.15.52:6801","nonce":2415060450}]},"active_addr":"172.21.15.52:6801/2415060450","active_change":"2024-04-26T01:32:45.509999+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":243168308}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1613611936}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2463815619}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2167000831}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3813587674}]}]} 2024-04-26T01:42:14.886 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T01:42:14.886 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T01:42:14.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:42:15.403 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:15.552 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:15 smithi052 ceph-mon[27851]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:15.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:15 smithi191 ceph-mon[32745]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:15 smithi183 ceph-mon[32752]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T01:42:17.419 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:17 smithi052 ceph-mon[27851]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:17.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:17 smithi191 ceph-mon[32745]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:17.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:17 smithi183 ceph-mon[32752]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 -- 172.21.15.52:0/1808119751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4654074c70 msgr2=0x7f4654075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 --2- 172.21.15.52:0/1808119751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4654074c70 0x7f4654075050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4644009990 tx=0x7f4644009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 -- 172.21.15.52:0/1808119751 shutdown_connections 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 --2- 172.21.15.52:0/1808119751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f465410f440 0x7f4654111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 --2- 172.21.15.52:0/1808119751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4654074c70 0x7f4654075050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 --2- 172.21.15.52:0/1808119751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 0x7f465410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 -- 172.21.15.52:0/1808119751 >> 172.21.15.52:0/1808119751 conn(0x7f4654079ba0 msgr2=0x7f4654079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 -- 172.21.15.52:0/1808119751 shutdown_connections 2024-04-26T01:42:17.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.703+0000 7f465a600700 1 -- 172.21.15.52:0/1808119751 wait complete. 2024-04-26T01:42:17.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.704+0000 7f465a600700 1 Processor -- start 2024-04-26T01:42:17.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 -- start start 2024-04-26T01:42:17.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:17.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 0x7f465419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:17.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 0x7f465419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:17.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4654114e40 con 0x7f465410f440 2024-04-26T01:42:17.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4654114cc0 con 0x7f4654075590 2024-04-26T01:42:17.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f465a600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4654114fc0 con 0x7f4654074c70 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f4653400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f4653e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 0x7f465419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f4652a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 0x7f465419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f4653400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54278/0 (socket says 172.21.15.52:54278) 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.705+0000 7f4653400700 1 -- 172.21.15.52:0/1572647371 learned_addr learned my addr 172.21.15.52:0/1572647371 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 -- 172.21.15.52:0/1572647371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 msgr2=0x7f465419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 0x7f465419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 -- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 msgr2=0x7f465419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 0x7f465419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 -- 172.21.15.52:0/1572647371 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46541a02d0 con 0x7f4654074c70 2024-04-26T01:42:17.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653e00700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 0x7f465419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.706+0000 7f4653400700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f46440057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.707+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644027070 con 0x7f4654074c70 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.707+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46541a0560 con 0x7f4654074c70 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.707+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4644022470 con 0x7f4654074c70 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.707+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644017650 con 0x7f4654074c70 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.707+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46541a7e40 con 0x7f4654074c70 2024-04-26T01:42:17.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.708+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4654076360 con 0x7f4654074c70 2024-04-26T01:42:17.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.710+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f464402a070 con 0x7f4654074c70 2024-04-26T01:42:17.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.711+0000 7f464be00700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 0x7f46380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:17.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.712+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f46440d1380 con 0x7f4654074c70 2024-04-26T01:42:17.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.712+0000 7f4652a00700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 0x7f46380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:17.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.712+0000 7f4652a00700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 0x7f46380a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4640009680 tx=0x7f4640009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:17.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.714+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4644013070 con 0x7f4654074c70 2024-04-26T01:42:17.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.950+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f465402df80 con 0x7f4654074c70 2024-04-26T01:42:17.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.952+0000 7f464be00700 1 -- 172.21.15.52:0/1572647371 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22667 (secure 0 0 0) 0x7f464409dc40 con 0x7f4654074c70 2024-04-26T01:42:17.950 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:42:17.950 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":94,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:47.490242+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:42:17.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.955+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 msgr2=0x7f46380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:17.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.955+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 0x7f46380a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4640009680 tx=0x7f4640009260 comp rx=0 tx=0).stop 2024-04-26T01:42:17.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.955+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 msgr2=0x7f465419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:17.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.955+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f46440057c0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 shutdown_connections 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f46380a3cd0 0x7f46380a6190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f465410f440 0x7f465419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4654074c70 0x7f465419b310 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 --2- 172.21.15.52:0/1572647371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4654075590 0x7f465419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 >> 172.21.15.52:0/1572647371 conn(0x7f4654079ba0 msgr2=0x7f46540ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 shutdown_connections 2024-04-26T01:42:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:17.956+0000 7f465a600700 1 -- 172.21.15.52:0/1572647371 wait complete. 2024-04-26T01:42:18.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:18 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1572647371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:18.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:18 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1572647371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:18.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:18 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1572647371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:18.763 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T01:42:18.764 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:42:19.204 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:19.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:19 smithi052 ceph-mon[27851]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:42:19.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:19 smithi191 ceph-mon[32745]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:42:19.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:19 smithi183 ceph-mon[32752]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T01:42:21.332 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:21 smithi052 ceph-mon[27851]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 -- 172.21.15.52:0/3806683599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910109fa0 msgr2=0x7f0910111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 --2- 172.21.15.52:0/3806683599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910109fa0 0x7f0910111b70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f090800b0d0 tx=0x7f090800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 -- 172.21.15.52:0/3806683599 shutdown_connections 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 --2- 172.21.15.52:0/3806683599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910109fa0 0x7f0910111b70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 --2- 172.21.15.52:0/3806683599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 0x7f09101052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 --2- 172.21.15.52:0/3806683599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910105870 0x7f09101098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.628+0000 7f0915c00700 1 -- 172.21.15.52:0/3806683599 >> 172.21.15.52:0/3806683599 conn(0x7f0910100710 msgr2=0x7f0910102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.629+0000 7f0915c00700 1 -- 172.21.15.52:0/3806683599 shutdown_connections 2024-04-26T01:42:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.629+0000 7f0915c00700 1 -- 172.21.15.52:0/3806683599 wait complete. 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 Processor -- start 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 -- start start 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 0x7f091019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 0x7f091019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0910114e90 con 0x7f0910105870 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0910114d10 con 0x7f0910104ec0 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.630+0000 7f0915c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0910115010 con 0x7f0910109fa0 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53286/0 (socket says 172.21.15.52:53286) 2024-04-26T01:42:21.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 -- 172.21.15.52:0/488033947 learned_addr learned my addr 172.21.15.52:0/488033947 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:21.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090ea00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 0x7f091019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:21.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090f400700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 0x7f091019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 -- 172.21.15.52:0/488033947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 msgr2=0x7f091019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 0x7f091019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 -- 172.21.15.52:0/488033947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 msgr2=0x7f091019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 0x7f091019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090e000700 1 -- 172.21.15.52:0/488033947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09101a0310 con 0x7f0910105870 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.631+0000 7f090f400700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 0x7f091019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f090e000700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e970 tx=0x7f08fc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc004a30 con 0x7f0910105870 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08fc00cc90 con 0x7f0910105870 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09101a0600 con 0x7f0910105870 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc005610 con 0x7f0910105870 2024-04-26T01:42:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.632+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09101a7e80 con 0x7f0910105870 2024-04-26T01:42:21.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.634+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0910069730 con 0x7f0910105870 2024-04-26T01:42:21.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.636+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f08fc00c7e0 con 0x7f0910105870 2024-04-26T01:42:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.637+0000 7f0907400700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 0x7f08f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.637+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f08fc0d06a0 con 0x7f0910105870 2024-04-26T01:42:21.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.638+0000 7f090ea00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 0x7f08f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:21.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.638+0000 7f090ea00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 0x7f08f40a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0900005950 tx=0x7f090000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:21.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.642+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08fc09ced0 con 0x7f0910105870 2024-04-26T01:42:21.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:21 smithi191 ceph-mon[32745]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:21.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:21 smithi183 ceph-mon[32752]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:21.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.881+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f091002df80 con 0x7f0910105870 2024-04-26T01:42:21.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.883+0000 7f0907400700 1 -- 172.21.15.52:0/488033947 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22667 (secure 0 0 0) 0x7f08fc09c7d0 con 0x7f0910105870 2024-04-26T01:42:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:42:21.881 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":94,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:47.490242+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:42:21.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.886+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 msgr2=0x7f08f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.886+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 0x7f08f40a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0900005950 tx=0x7f090000b410 comp rx=0 tx=0).stop 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.886+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 msgr2=0x7f091019b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.886+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f08fc00e970 tx=0x7f08fc00ec80 comp rx=0 tx=0).stop 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 shutdown_connections 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f08f40a3cd0 0x7f08f40a6190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0910109fa0 0x7f091019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0910104ec0 0x7f091019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 --2- 172.21.15.52:0/488033947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0910105870 0x7f091019b840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:21.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 >> 172.21.15.52:0/488033947 conn(0x7f0910100710 msgr2=0x7f09101029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:21.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 shutdown_connections 2024-04-26T01:42:21.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:21.887+0000 7f0915c00700 1 -- 172.21.15.52:0/488033947 wait complete. 2024-04-26T01:42:22.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:22 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/488033947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:22.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:22 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/488033947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:22.705 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:22 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/488033947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:42:22.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T01:42:22.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T01:42:22.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T01:42:22.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T01:42:22.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T01:42:22.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T01:42:22.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T01:42:22.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T01:42:22.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T01:42:22.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T01:42:22.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T01:42:22.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T01:42:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:23 smithi191 ceph-mon[32745]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:23.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:23 smithi183 ceph-mon[32752]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:23.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:23 smithi052 ceph-mon[27851]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:23.789 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.797 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.836 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.839 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.986 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.986 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:23.987 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:24.927 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:25.406 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:25 smithi052 ceph-mon[27851]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:25.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:25 smithi191 ceph-mon[32745]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:25.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:25 smithi183 ceph-mon[32752]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:27.143 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:27.567 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:27 smithi052 ceph-mon[27851]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:27 smithi191 ceph-mon[32745]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:27.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:27 smithi183 ceph-mon[32752]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:27.803 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:27.804 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:29 smithi052 ceph-mon[27851]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:42:29.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:29 smithi191 ceph-mon[32745]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:42:29.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:29 smithi183 ceph-mon[32752]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:42:31.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:31 smithi052 ceph-mon[27851]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:31.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:31 smithi191 ceph-mon[32745]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:31.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:31 smithi183 ceph-mon[32752]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:33.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:33 smithi191 ceph-mon[32745]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:33.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:33 smithi183 ceph-mon[32752]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:33.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:33 smithi052 ceph-mon[27851]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:42:35.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 -- 172.21.15.52:0/1194644953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c074c70 msgr2=0x7f5f4c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 --2- 172.21.15.52:0/1194644953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c074c70 0x7f5f4c075050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5f400098f0 tx=0x7f5f40009c00 comp rx=0 tx=0).stop 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 -- 172.21.15.52:0/1194644953 shutdown_connections 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 --2- 172.21.15.52:0/1194644953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f4c113c10 0x7f5f4c116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 --2- 172.21.15.52:0/1194644953 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c074c70 0x7f5f4c075050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 --2- 172.21.15.52:0/1194644953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c075590 0x7f5f4c1136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 -- 172.21.15.52:0/1194644953 >> 172.21.15.52:0/1194644953 conn(0x7f5f4c079ba0 msgr2=0x7f5f4c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 -- 172.21.15.52:0/1194644953 shutdown_connections 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.842+0000 7f5f53e00700 1 -- 172.21.15.52:0/1194644953 wait complete. 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.843+0000 7f5f53e00700 1 Processor -- start 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.843+0000 7f5f53e00700 1 -- start start 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 0x7f5f4c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:35.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f4c075590 0x7f5f4c1a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c113c10 0x7f5f4c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c119270 con 0x7f5f4c075590 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1190f0 con 0x7f5f4c113c10 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f53e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1193f0 con 0x7f5f4c074c70 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f51400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 0x7f5f4c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f50a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f4c075590 0x7f5f4c1a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f51400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 0x7f5f4c1a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54572/0 (socket says 172.21.15.52:54572) 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.844+0000 7f5f51400700 1 -- 172.21.15.52:0/2333463025 learned_addr learned my addr 172.21.15.52:0/2333463025 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c113c10 0x7f5f4c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 -- 172.21.15.52:0/2333463025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c113c10 msgr2=0x7f5f4c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c113c10 0x7f5f4c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 -- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f4c075590 msgr2=0x7f5f4c1a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f4c075590 0x7f5f4c1a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c19ff10 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f4c113c10 0x7f5f4c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f51400700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 0x7f5f4c1a5560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5f40009230 tx=0x7f5f4000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40005140 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4000b560 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.845+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c1a01a0 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.846+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40003ae0 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.846+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4c1ac4e0 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.847+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f4c100b00 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.849+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5f40003c80 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.851+0000 7f5f4a000700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5f340a3d50 0x7f5f340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.851+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f5f400d1950 con 0x7f5f4c074c70 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.851+0000 7f5f50a00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5f340a3d50 0x7f5f340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:35.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.851+0000 7f5f4a000700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] conn(0x7f5f340ad760 0x7f5f340afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.852+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f40001d30 con 0x7f5f340ad760 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.852+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f5f400d1d00 con 0x7f5f4c074c70 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.852+0000 7f5f51e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] conn(0x7f5f340ad760 0x7f5f340afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.852+0000 7f5f50a00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5f340a3d50 0x7f5f340a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c1a0df0 tx=0x7f5f38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.853+0000 7f5f51e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] conn(0x7f5f340ad760 0x7f5f340afc00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.854+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== osd.6 v2:172.21.15.52:6818/915546379 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f40001d30 con 0x7f5f340ad760 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.873+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 --> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5f4c076360 con 0x7f5f340ad760 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.873+0000 7f5f4a000700 1 -- 172.21.15.52:0/2333463025 <== osd.6 v2:172.21.15.52:6818/915546379 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5f4c076360 con 0x7f5f340ad760 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] conn(0x7f5f340ad760 msgr2=0x7f5f340afc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] conn(0x7f5f340ad760 0x7f5f340afc00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5f340a3d50 msgr2=0x7f5f340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5f340a3d50 0x7f5f340a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c1a0df0 tx=0x7f5f38009260 comp rx=0 tx=0).stop 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 msgr2=0x7f5f4c1a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 --2- 172.21.15.52:0/2333463025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f4c074c70 0x7f5f4c1a5560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5f40009230 tx=0x7f5f4000fa70 comp rx=0 tx=0).stop 2024-04-26T01:42:35.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f51e00700 1 -- 172.21.15.52:0/2333463025 reap_dead start 2024-04-26T01:42:35.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 shutdown_connections 2024-04-26T01:42:35.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 >> 172.21.15.52:0/2333463025 conn(0x7f5f4c079ba0 msgr2=0x7f5f4c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:35.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 shutdown_connections 2024-04-26T01:42:35.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:34.874+0000 7f5f53e00700 1 -- 172.21.15.52:0/2333463025 wait complete. 2024-04-26T01:42:35.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:35 smithi052 ceph-mon[27851]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:35 smithi191 ceph-mon[32745]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:35.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:35 smithi183 ceph-mon[32752]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 -- 172.21.15.52:0/1503421104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe8010e3b0 msgr2=0x7fbe80115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 --2- 172.21.15.52:0/1503421104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe8010e3b0 0x7fbe80115f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b0d0 tx=0x7fbe7800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 -- 172.21.15.52:0/1503421104 shutdown_connections 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 --2- 172.21.15.52:0/1503421104 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe8010e3b0 0x7fbe80115f80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 --2- 172.21.15.52:0/1503421104 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe801092d0 0x7fbe801096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 --2- 172.21.15.52:0/1503421104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe80109c80 0x7fbe8010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 -- 172.21.15.52:0/1503421104 >> 172.21.15.52:0/1503421104 conn(0x7fbe80076210 msgr2=0x7fbe80078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 -- 172.21.15.52:0/1503421104 shutdown_connections 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.827+0000 7fbe87000700 1 -- 172.21.15.52:0/1503421104 wait complete. 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.828+0000 7fbe87000700 1 Processor -- start 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.828+0000 7fbe87000700 1 -- start start 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe801092d0 0x7fbe8019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 0x7fbe8019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe8010e3b0 0x7fbe801a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe801192a0 con 0x7fbe80109c80 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80119120 con 0x7fbe8010e3b0 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe87000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80119420 con 0x7fbe801092d0 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe7f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 0x7fbe8019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe7f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 0x7fbe8019fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55708/0 (socket says 172.21.15.52:55708) 2024-04-26T01:42:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe7f400700 1 -- 172.21.15.52:0/2896209688 learned_addr learned my addr 172.21.15.52:0/2896209688 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe7fe00700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe801092d0 0x7fbe8019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.829+0000 7fbe85000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe8010e3b0 0x7fbe801a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 -- 172.21.15.52:0/2896209688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe801092d0 msgr2=0x7fbe8019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe801092d0 0x7fbe8019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 -- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe8010e3b0 msgr2=0x7fbe801a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe8010e3b0 0x7fbe801a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe801a4710 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7f400700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 0x7fbe8019fc90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e970 tx=0x7fbe6c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c004a30 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe6c00cc90 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c005610 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe801a4a00 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.830+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe801ac280 con 0x7fbe80109c80 2024-04-26T01:42:37.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.831+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe800631d0 con 0x7fbe80109c80 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.834+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe6c00c7e0 con 0x7fbe80109c80 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7cc00700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbe640a3d50 0x7fbe640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fbe6c0d0a90 con 0x7fbe80109c80 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7fe00700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbe640a3d50 0x7fbe640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7cc00700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] conn(0x7fbe640ad890 0x7fbe640afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe640b0400 con 0x7fbe640ad890 2024-04-26T01:42:37.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.835+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fbe6c0d0e40 con 0x7fbe80109c80 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.836+0000 7fbe85000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] conn(0x7fbe640ad890 0x7fbe640afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.836+0000 7fbe7fe00700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbe640a3d50 0x7fbe640a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009680 tx=0x7fbe70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.836+0000 7fbe85000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] conn(0x7fbe640ad890 0x7fbe640afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.837+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== osd.4 v2:172.21.15.183:6808/2309146877 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbe640b0400 con 0x7fbe640ad890 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.856+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 --> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe800635d0 con 0x7fbe640ad890 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.856+0000 7fbe7cc00700 1 -- 172.21.15.52:0/2896209688 <== osd.4 v2:172.21.15.183:6808/2309146877 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbe800635d0 con 0x7fbe640ad890 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] conn(0x7fbe640ad890 msgr2=0x7fbe640afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] conn(0x7fbe640ad890 0x7fbe640afd30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbe640a3d50 msgr2=0x7fbe640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbe640a3d50 0x7fbe640a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009680 tx=0x7fbe70009260 comp rx=0 tx=0).stop 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 msgr2=0x7fbe8019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 --2- 172.21.15.52:0/2896209688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe80109c80 0x7fbe8019fc90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e970 tx=0x7fbe6c00ec80 comp rx=0 tx=0).stop 2024-04-26T01:42:37.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe85000700 1 -- 172.21.15.52:0/2896209688 reap_dead start 2024-04-26T01:42:37.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 shutdown_connections 2024-04-26T01:42:37.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.857+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 >> 172.21.15.52:0/2896209688 conn(0x7fbe80076210 msgr2=0x7fbe80076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:37.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.858+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 shutdown_connections 2024-04-26T01:42:37.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:36.858+0000 7fbe87000700 1 -- 172.21.15.52:0/2896209688 wait complete. 2024-04-26T01:42:37.634 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:37 smithi052 ceph-mon[27851]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:42:37.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:37 smithi191 ceph-mon[32745]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:42:37.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:37 smithi183 ceph-mon[32752]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 -- 172.21.15.52:0/1189001214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd0075590 msgr2=0x7f3cd010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 --2- 172.21.15.52:0/1189001214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd0075590 0x7f3cd010ef00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc009990 tx=0x7f3cbc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 -- 172.21.15.52:0/1189001214 shutdown_connections 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 --2- 172.21.15.52:0/1189001214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd010f440 0x7f3cd0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 --2- 172.21.15.52:0/1189001214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0074c70 0x7f3cd0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 --2- 172.21.15.52:0/1189001214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd0075590 0x7f3cd010ef00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 -- 172.21.15.52:0/1189001214 >> 172.21.15.52:0/1189001214 conn(0x7f3cd0079ba0 msgr2=0x7f3cd0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 -- 172.21.15.52:0/1189001214 shutdown_connections 2024-04-26T01:42:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.833+0000 7f3cd7000700 1 -- 172.21.15.52:0/1189001214 wait complete. 2024-04-26T01:42:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.834+0000 7f3cd7000700 1 Processor -- start 2024-04-26T01:42:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 -- start start 2024-04-26T01:42:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd0074c70 0x7f3cd019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0075590 0x7f3cd019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd010f440 0x7f3cd019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0114e70 con 0x7f3cd0075590 2024-04-26T01:42:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0114cf0 con 0x7f3cd0074c70 2024-04-26T01:42:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3cd7000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0114ff0 con 0x7f3cd010f440 2024-04-26T01:42:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3ccf400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0075590 0x7f3cd019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd0074c70 0x7f3cd019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.835+0000 7f3ccf400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0075590 0x7f3cd019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55736/0 (socket says 172.21.15.52:55736) 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3cd5000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd010f440 0x7f3cd019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccf400700 1 -- 172.21.15.52:0/3285066263 learned_addr learned my addr 172.21.15.52:0/3285066263 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 -- 172.21.15.52:0/3285066263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd010f440 msgr2=0x7f3cd019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3cd010f440 0x7f3cd019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 -- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0075590 msgr2=0x7f3cd019b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3cd0075590 0x7f3cd019b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd01a0350 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.836+0000 7f3ccfe00700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd0074c70 0x7f3cd019b340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d750 tx=0x7f3cc000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.837+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0014030 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.837+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd01a0640 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.837+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc00044d0 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.837+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0005440 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.837+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd01a7ec0 con 0x7f3cd0074c70 2024-04-26T01:42:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.838+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3cd0063150 con 0x7f3cd0074c70 2024-04-26T01:42:38.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.840+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3cc00055e0 con 0x7f3cd0074c70 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.841+0000 7f3cccc00700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f3cb40a3d50 0x7f3cb40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f3cc00d0240 con 0x7f3cd0074c70 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3ccf400700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f3cb40a3d50 0x7f3cb40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3cccc00700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] conn(0x7f3cb40ad890 0x7f3cb40afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3cc0021e50 con 0x7f3cb40ad890 2024-04-26T01:42:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f3cc00d05f0 con 0x7f3cd0074c70 2024-04-26T01:42:38.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3cd5000700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] conn(0x7f3cb40ad890 0x7f3cb40afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:38.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.842+0000 7f3ccf400700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f3cb40a3d50 0x7f3cb40a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0111100 tx=0x7f3cbc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:38.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.843+0000 7f3cd5000700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] conn(0x7f3cb40ad890 0x7f3cb40afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:38.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.844+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== osd.3 v2:172.21.15.52:6810/1589015031 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3cc0021e50 con 0x7f3cb40ad890 2024-04-26T01:42:38.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.862+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 --> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3cd0063550 con 0x7f3cb40ad890 2024-04-26T01:42:38.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.862+0000 7f3cccc00700 1 -- 172.21.15.52:0/3285066263 <== osd.3 v2:172.21.15.52:6810/1589015031 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3cd0063550 con 0x7f3cb40ad890 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] conn(0x7f3cb40ad890 msgr2=0x7f3cb40afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] conn(0x7f3cb40ad890 0x7f3cb40afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f3cb40a3d50 msgr2=0x7f3cb40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f3cb40a3d50 0x7f3cb40a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0111100 tx=0x7f3cbc005fd0 comp rx=0 tx=0).stop 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd0074c70 msgr2=0x7f3cd019b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 --2- 172.21.15.52:0/3285066263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cd0074c70 0x7f3cd019b340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d750 tx=0x7f3cc000da60 comp rx=0 tx=0).stop 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd5000700 1 -- 172.21.15.52:0/3285066263 reap_dead start 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 shutdown_connections 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 >> 172.21.15.52:0/3285066263 conn(0x7f3cd0079ba0 msgr2=0x7f3cd00ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 shutdown_connections 2024-04-26T01:42:38.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.863+0000 7f3cd7000700 1 -- 172.21.15.52:0/3285066263 wait complete. 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 -- 172.21.15.52:0/2059528608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70074c70 msgr2=0x7f1a70075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 --2- 172.21.15.52:0/2059528608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70074c70 0x7f1a70075050 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009990 tx=0x7f1a64009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 -- 172.21.15.52:0/2059528608 shutdown_connections 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 --2- 172.21.15.52:0/2059528608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7010f470 0x7f1a70111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 --2- 172.21.15.52:0/2059528608 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70074c70 0x7f1a70075050 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 --2- 172.21.15.52:0/2059528608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70075590 0x7f1a7010ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 -- 172.21.15.52:0/2059528608 >> 172.21.15.52:0/2059528608 conn(0x7f1a70079b10 msgr2=0x7f1a70079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 -- 172.21.15.52:0/2059528608 shutdown_connections 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.737+0000 7f1a77c00700 1 -- 172.21.15.52:0/2059528608 wait complete. 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.738+0000 7f1a77c00700 1 Processor -- start 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 -- start start 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 0x7f1a7019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70075590 0x7f1a7019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7010f470 0x7f1a7019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70114e20 con 0x7f1a7010f470 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70114ca0 con 0x7f1a70075590 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a77c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70114fa0 con 0x7f1a70074c70 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a75c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7010f470 0x7f1a7019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a75200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 0x7f1a7019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a6fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70075590 0x7f1a7019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a75200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 0x7f1a7019b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54614/0 (socket says 172.21.15.52:54614) 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.739+0000 7f1a75200700 1 -- 172.21.15.52:0/324081746 learned_addr learned my addr 172.21.15.52:0/324081746 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 -- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70075590 msgr2=0x7f1a7019b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70075590 0x7f1a7019b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 -- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7010f470 msgr2=0x7f1a7019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7010f470 0x7f1a7019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a701a0280 con 0x7f1a70074c70 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a6fe00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70075590 0x7f1a7019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.740+0000 7f1a75200700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 0x7f1a7019b270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a6400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:39.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.741+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64027070 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.741+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a64022470 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.741+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64017650 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.741+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a701a0510 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.741+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a701a7df0 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.742+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a701006a0 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.744+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1a6402a070 con 0x7f1a70074c70 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6d600700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1a580a3d50 0x7f1a580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f1a640d19e0 con 0x7f1a70074c70 2024-04-26T01:42:39.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6fe00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1a580a3d50 0x7f1a580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6d600700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] conn(0x7f1a580ad890 0x7f1a580afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:39.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a640095d0 con 0x7f1a580ad890 2024-04-26T01:42:39.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.746+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f1a640d1d90 con 0x7f1a70074c70 2024-04-26T01:42:39.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.747+0000 7f1a6fe00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1a580a3d50 0x7f1a580a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a70111180 tx=0x7f1a5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.747+0000 7f1a75c00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] conn(0x7f1a580ad890 0x7f1a580afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.747+0000 7f1a75c00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] conn(0x7f1a580ad890 0x7f1a580afd30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.748+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== osd.10 v2:172.21.15.183:6824/2192270029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a640095d0 con 0x7f1a580ad890 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.767+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 --> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1a70076360 con 0x7f1a580ad890 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.767+0000 7f1a6d600700 1 -- 172.21.15.52:0/324081746 <== osd.10 v2:172.21.15.183:6824/2192270029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1a70076360 con 0x7f1a580ad890 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] conn(0x7f1a580ad890 msgr2=0x7f1a580afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] conn(0x7f1a580ad890 0x7f1a580afd30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1a580a3d50 msgr2=0x7f1a580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1a580a3d50 0x7f1a580a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a70111180 tx=0x7f1a5c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:39.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 msgr2=0x7f1a7019b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 --2- 172.21.15.52:0/324081746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a70074c70 0x7f1a7019b270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a6400bfd0 comp rx=0 tx=0).stop 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a75c00700 1 -- 172.21.15.52:0/324081746 reap_dead start 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 shutdown_connections 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.768+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 >> 172.21.15.52:0/324081746 conn(0x7f1a70079b10 msgr2=0x7f1a700ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.769+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 shutdown_connections 2024-04-26T01:42:39.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:38.769+0000 7f1a77c00700 1 -- 172.21.15.52:0/324081746 wait complete. 2024-04-26T01:42:39.558 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:39 smithi052 ceph-mon[27851]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-04-26T01:42:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:39 smithi191 ceph-mon[32745]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-04-26T01:42:39.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:39 smithi183 ceph-mon[32752]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-04-26T01:42:40.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.928+0000 7efeb2400700 1 -- 172.21.15.52:0/1839589689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac10e3b0 msgr2=0x7efeac115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.928+0000 7efeb2400700 1 --2- 172.21.15.52:0/1839589689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac10e3b0 0x7efeac115f80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efea400b0d0 tx=0x7efea400b3e0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 -- 172.21.15.52:0/1839589689 shutdown_connections 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 --2- 172.21.15.52:0/1839589689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac10e3b0 0x7efeac115f80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 --2- 172.21.15.52:0/1839589689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1092d0 0x7efeac1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 --2- 172.21.15.52:0/1839589689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac109c80 0x7efeac10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 -- 172.21.15.52:0/1839589689 >> 172.21.15.52:0/1839589689 conn(0x7efeac076210 msgr2=0x7efeac107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 -- 172.21.15.52:0/1839589689 shutdown_connections 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.929+0000 7efeb2400700 1 -- 172.21.15.52:0/1839589689 wait complete. 2024-04-26T01:42:40.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 Processor -- start 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 -- start start 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1092d0 0x7efeac19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 0x7efeac19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1192c0 con 0x7efeac10e3b0 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119140 con 0x7efeac1092d0 2024-04-26T01:42:40.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.930+0000 7efeb2400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119440 con 0x7efeac109c80 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeabe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 0x7efeac19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeab400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1092d0 0x7efeac19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 0x7efeac19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54670/0 (socket says 172.21.15.52:54670) 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 -- 172.21.15.52:0/230753025 learned_addr learned my addr 172.21.15.52:0/230753025 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeabe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55772/0 (socket says 172.21.15.52:55772) 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 -- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1092d0 msgr2=0x7efeac19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efeac1092d0 0x7efeac19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 -- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 msgr2=0x7efeac1a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeaaa00700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac1a4700 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.931+0000 7efeabe00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efeaaa00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 0x7efeac19fc30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e970 tx=0x7efe9800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98004a30 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9800cc90 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac1a49f0 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98005610 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.932+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac1ac270 con 0x7efeac109c80 2024-04-26T01:42:40.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.933+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efeac0631d0 con 0x7efeac109c80 2024-04-26T01:42:40.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.935+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efe9800c7e0 con 0x7efeac109c80 2024-04-26T01:42:40.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efea3e00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efe900a3d50 0x7efe900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7efe980d0a90 con 0x7efeac109c80 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efeab400700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efe900a3d50 0x7efe900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efea3e00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] conn(0x7efe900ad890 0x7efe900afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe900b0400 con 0x7efe900ad890 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.937+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7efe980d0e40 con 0x7efeac109c80 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.938+0000 7efeabe00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] conn(0x7efe900ad890 0x7efe900afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:40.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.938+0000 7efeab400700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efe900a3d50 0x7efe900a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009c60 tx=0x7efe9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:40.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.938+0000 7efeabe00700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] conn(0x7efe900ad890 0x7efe900afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:40.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.939+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== osd.1 v2:172.21.15.183:6800/3104160515 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe900b0400 con 0x7efe900ad890 2024-04-26T01:42:40.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.957+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 --> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efeac063600 con 0x7efe900ad890 2024-04-26T01:42:40.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.958+0000 7efea3e00700 1 -- 172.21.15.52:0/230753025 <== osd.1 v2:172.21.15.183:6800/3104160515 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efeac063600 con 0x7efe900ad890 2024-04-26T01:42:40.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] conn(0x7efe900ad890 msgr2=0x7efe900afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] conn(0x7efe900ad890 0x7efe900afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efe900a3d50 msgr2=0x7efe900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efe900a3d50 0x7efe900a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009c60 tx=0x7efe9c009260 comp rx=0 tx=0).stop 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 msgr2=0x7efeac19fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 --2- 172.21.15.52:0/230753025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efeac109c80 0x7efeac19fc30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e970 tx=0x7efe9800ec80 comp rx=0 tx=0).stop 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeabe00700 1 -- 172.21.15.52:0/230753025 reap_dead start 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 shutdown_connections 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.959+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 >> 172.21.15.52:0/230753025 conn(0x7efeac076210 msgr2=0x7efeac076730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.960+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 shutdown_connections 2024-04-26T01:42:40.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:40.960+0000 7efeb2400700 1 -- 172.21.15.52:0/230753025 wait complete. 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.828+0000 7f7543e00700 1 -- 172.21.15.52:0/1123281524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c102af0 msgr2=0x7f753c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.828+0000 7f7543e00700 1 --2- 172.21.15.52:0/1123281524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c102af0 0x7f753c102f70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7528009990 tx=0x7f7528009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 -- 172.21.15.52:0/1123281524 shutdown_connections 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 --2- 172.21.15.52:0/1123281524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c1034b0 0x7f753c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 --2- 172.21.15.52:0/1123281524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f753c107940 0x7f753c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 --2- 172.21.15.52:0/1123281524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c102af0 0x7f753c102f70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 -- 172.21.15.52:0/1123281524 >> 172.21.15.52:0/1123281524 conn(0x7f753c079ba0 msgr2=0x7f753c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:41.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 -- 172.21.15.52:0/1123281524 shutdown_connections 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.829+0000 7f7543e00700 1 -- 172.21.15.52:0/1123281524 wait complete. 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.830+0000 7f7543e00700 1 Processor -- start 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.830+0000 7f7543e00700 1 -- start start 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 0x7f753c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f753c1034b0 0x7f753c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c107940 0x7f753c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f753c112c90 con 0x7f753c102af0 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f753c112b10 con 0x7f753c1034b0 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7543e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f753c112e10 con 0x7f753c107940 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 0x7f753c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 0x7f753c19b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55756/0 (socket says 172.21.15.52:55756) 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 -- 172.21.15.52:0/744183208 learned_addr learned my addr 172.21.15.52:0/744183208 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7540a00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f753c1034b0 0x7f753c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c107940 0x7f753c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:41.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 -- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c107940 msgr2=0x7f753c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f753c107940 0x7f753c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 -- 172.21.15.52:0/744183208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f753c1034b0 msgr2=0x7f753c19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f753c1034b0 0x7f753c19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.831+0000 7f7541400700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c1a0380 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f7541400700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 0x7f753c19b370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f753000e9a0 tx=0x7f753000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753000ca10 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753000cbb0 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c1a0670 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7530018680 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.832+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a7ef0 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.833+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f753c101340 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.835+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7530010430 con 0x7f753c102af0 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f753a000700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f75240a3cd0 0x7f75240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:41.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f75300d0ca0 con 0x7f753c102af0 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f7540a00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f75240a3cd0 0x7f75240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f753a000700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] conn(0x7f75240ad810 0x7f75240afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7530016ec0 con 0x7f75240ad810 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.837+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f75300d1050 con 0x7f753c102af0 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.838+0000 7f7540a00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f75240a3cd0 0x7f75240a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f752800b790 tx=0x7f752800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:41.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.838+0000 7f7541e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] conn(0x7f75240ad810 0x7f75240afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.838+0000 7f7541e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] conn(0x7f75240ad810 0x7f75240afcb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.839+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== osd.2 v2:172.21.15.191:6800/932162512 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7530016ec0 con 0x7f75240ad810 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.859+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 --> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f753c1041b0 con 0x7f75240ad810 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.860+0000 7f753a000700 1 -- 172.21.15.52:0/744183208 <== osd.2 v2:172.21.15.191:6800/932162512 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f753c1041b0 con 0x7f75240ad810 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.860+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] conn(0x7f75240ad810 msgr2=0x7f75240afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.860+0000 7f7543e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] conn(0x7f75240ad810 0x7f75240afcb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f75240a3cd0 msgr2=0x7f75240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f75240a3cd0 0x7f75240a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f752800b790 tx=0x7f752800b260 comp rx=0 tx=0).stop 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 msgr2=0x7f753c19b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 --2- 172.21.15.52:0/744183208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f753c102af0 0x7f753c19b370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f753000e9a0 tx=0x7f753000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7541e00700 1 -- 172.21.15.52:0/744183208 reap_dead start 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 shutdown_connections 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 >> 172.21.15.52:0/744183208 conn(0x7f753c079ba0 msgr2=0x7f753c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:41.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 shutdown_connections 2024-04-26T01:42:41.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:39.861+0000 7f7543e00700 1 -- 172.21.15.52:0/744183208 wait complete. 2024-04-26T01:42:41.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:41 smithi052 ceph-mon[27851]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-26T01:42:41.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:41 smithi191 ceph-mon[32745]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-26T01:42:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:41 smithi183 ceph-mon[32752]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-26T01:42:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 -- 172.21.15.52:0/3196398064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 msgr2=0x7fe25806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 --2- 172.21.15.52:0/3196398064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe25806b600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2400098f0 tx=0x7fe240009c00 comp rx=0 tx=0).stop 2024-04-26T01:42:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 -- 172.21.15.52:0/3196398064 shutdown_connections 2024-04-26T01:42:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 --2- 172.21.15.52:0/3196398064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe25806bb40 0x7fe25810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 --2- 172.21.15.52:0/3196398064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 --2- 172.21.15.52:0/3196398064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe25806b600 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 -- 172.21.15.52:0/3196398064 >> 172.21.15.52:0/3196398064 conn(0x7fe258078740 msgr2=0x7fe258078b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.274+0000 7fe25cc00700 1 -- 172.21.15.52:0/3196398064 shutdown_connections 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.275+0000 7fe25cc00700 1 -- 172.21.15.52:0/3196398064 wait complete. 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.276+0000 7fe25cc00700 1 Processor -- start 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.276+0000 7fe25cc00700 1 -- start start 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe258101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe25806bb40 0x7fe258102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe258112cc0 con 0x7fe258108b50 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe258112b40 con 0x7fe25806bb40 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe25cc00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe258112e40 con 0x7fe25806b180 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe256800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe255e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe258101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe24fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe25806bb40 0x7fe258102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe256800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258102790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55800/0 (socket says 172.21.15.52:55800) 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe256800700 1 -- 172.21.15.52:0/1981301239 learned_addr learned my addr 172.21.15.52:0/1981301239 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.277+0000 7fe255e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe258101d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54686/0 (socket says 172.21.15.52:54686) 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 -- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 msgr2=0x7fe258101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe258101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 -- 172.21.15.52:0/1981301239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe25806bb40 msgr2=0x7fe258102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe25806bb40 0x7fe258102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2581a7740 con 0x7fe258108b50 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe255e00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe25806b180 0x7fe258101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe256800700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258102790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe25000c340 tx=0x7fe25000c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25000cb30 con 0x7fe258108b50 2024-04-26T01:42:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2581a7a30 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe25000ccd0 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.278+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe250007ce0 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.279+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2581a7f40 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.279+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe2580630d0 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.282+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe250012070 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.283+0000 7fe24ea00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe23c0a3cd0 0x7fe23c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.283+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fe2500d0760 con 0x7fe258108b50 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe255e00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe23c0a3cd0 0x7fe23c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe24ea00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] conn(0x7fe23c0ad810 0x7fe23c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe250023e50 con 0x7fe23c0ad810 2024-04-26T01:42:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fe2500d0b10 con 0x7fe258108b50 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe24fe00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] conn(0x7fe23c0ad810 0x7fe23c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.284+0000 7fe255e00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe23c0a3cd0 0x7fe23c0a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.285+0000 7fe24fe00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] conn(0x7fe23c0ad810 0x7fe23c0afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.286+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== osd.5 v2:172.21.15.191:6808/3556847761 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe250023e50 con 0x7fe23c0ad810 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.305+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 --> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe258063550 con 0x7fe23c0ad810 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.306+0000 7fe24ea00700 1 -- 172.21.15.52:0/1981301239 <== osd.5 v2:172.21.15.191:6808/3556847761 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe258063550 con 0x7fe23c0ad810 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.306+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] conn(0x7fe23c0ad810 msgr2=0x7fe23c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.306+0000 7fe25cc00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] conn(0x7fe23c0ad810 0x7fe23c0afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.306+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe23c0a3cd0 msgr2=0x7fe23c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.306+0000 7fe25cc00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe23c0a3cd0 0x7fe23c0a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe248009fd0 tx=0x7fe248009260 comp rx=0 tx=0).stop 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 msgr2=0x7fe258102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 --2- 172.21.15.52:0/1981301239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe258108b50 0x7fe258102790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe25000c340 tx=0x7fe25000c650 comp rx=0 tx=0).stop 2024-04-26T01:42:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe256800700 1 -- 172.21.15.52:0/1981301239 reap_dead start 2024-04-26T01:42:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 shutdown_connections 2024-04-26T01:42:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 >> 172.21.15.52:0/1981301239 conn(0x7fe258078740 msgr2=0x7fe258100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 shutdown_connections 2024-04-26T01:42:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:42.307+0000 7fe25cc00700 1 -- 172.21.15.52:0/1981301239 wait complete. 2024-04-26T01:42:43.155 INFO:teuthology.orchestra.run.smithi052.stdout:150323855438 2024-04-26T01:42:43.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T01:42:43.475 INFO:teuthology.orchestra.run.smithi052.stdout:77309411418 2024-04-26T01:42:43.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T01:42:43.531 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:43 smithi052 ceph-mon[27851]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:42:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:43 smithi191 ceph-mon[32745]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:42:43.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:43 smithi183 ceph-mon[32752]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:42:43.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 -- 172.21.15.52:0/1062407016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4076f70 msgr2=0x7f7fa4077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/1062407016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4076f70 0x7f7fa4077350 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009990 tx=0x7f7f94009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 -- 172.21.15.52:0/1062407016 shutdown_connections 2024-04-26T01:42:43.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/1062407016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 0x7f7fa4115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/1062407016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4076f70 0x7f7fa4077350 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/1062407016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4077890 0x7f7fa41132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 -- 172.21.15.52:0/1062407016 >> 172.21.15.52:0/1062407016 conn(0x7f7fa4100660 msgr2=0x7f7fa4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:43.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.975+0000 7f7fa3e00700 1 -- 172.21.15.52:0/1062407016 shutdown_connections 2024-04-26T01:42:43.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.976+0000 7f7fa3e00700 1 -- 172.21.15.52:0/1062407016 wait complete. 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 Processor -- start 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 -- start start 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4076f70 0x7f7fa419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4077890 0x7f7fa419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 0x7f7fa41a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4119180 con 0x7f7fa4077890 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4119000 con 0x7f7fa4113810 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.977+0000 7f7fa3e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa4119300 con 0x7f7fa4076f70 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4077890 0x7f7fa419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4077890 0x7f7fa419fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55814/0 (socket says 172.21.15.52:55814) 2024-04-26T01:42:43.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2000700 1 -- 172.21.15.52:0/272131561 learned_addr learned my addr 172.21.15.52:0/272131561 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:43.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4076f70 0x7f7fa419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa3400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 0x7f7fa41a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 -- 172.21.15.52:0/272131561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 msgr2=0x7f7fa41a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 0x7f7fa41a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 -- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4077890 msgr2=0x7f7fa419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7fa4077890 0x7f7fa419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa2a00700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa41a4690 con 0x7f7fa4076f70 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.978+0000 7f7fa3400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7fa4113810 0x7f7fa41a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7fa2a00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4076f70 0x7f7fa419f6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f940059d0 tx=0x7f7f940047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94027070 con 0x7f7fa4076f70 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa41a4920 con 0x7f7fa4076f70 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa41ac200 con 0x7f7fa4076f70 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f94022470 con 0x7f7fa4076f70 2024-04-26T01:42:43.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.979+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94017650 con 0x7f7fa4076f70 2024-04-26T01:42:43.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.980+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7fa4063110 con 0x7f7fa4076f70 2024-04-26T01:42:43.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.982+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7f9402a070 con 0x7f7fa4076f70 2024-04-26T01:42:43.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7f9b400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7f880a3d50 0x7f7f880a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f7f940d19c0 con 0x7f7fa4076f70 2024-04-26T01:42:43.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7fa2000700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7f880a3d50 0x7f7f880a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7f9b400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] conn(0x7f7f880ad890 0x7f7f880afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:43.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f880b0400 con 0x7f7f880ad890 2024-04-26T01:42:43.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.984+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f7f940d1d70 con 0x7f7fa4076f70 2024-04-26T01:42:43.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.985+0000 7f7fa3400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] conn(0x7f7f880ad890 0x7f7f880afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:43.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.985+0000 7f7fa2000700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7f880a3d50 0x7f7f880a6210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4079a70 tx=0x7f7f90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.985+0000 7f7fa3400700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] conn(0x7f7f880ad890 0x7f7f880afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:43.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:43.986+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== osd.11 v2:172.21.15.191:6824/1249163110 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f880b0400 con 0x7f7f880ad890 2024-04-26T01:42:44.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.005+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 --> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7fa4063590 con 0x7f7f880ad890 2024-04-26T01:42:44.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7f9b400700 1 -- 172.21.15.52:0/272131561 <== osd.11 v2:172.21.15.191:6824/1249163110 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7fa4063590 con 0x7f7f880ad890 2024-04-26T01:42:44.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] conn(0x7f7f880ad890 msgr2=0x7f7f880afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] conn(0x7f7f880ad890 0x7f7f880afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7f880a3d50 msgr2=0x7f7f880a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f7f880a3d50 0x7f7f880a6210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4079a70 tx=0x7f7f90009260 comp rx=0 tx=0).stop 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4076f70 msgr2=0x7f7fa419f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 --2- 172.21.15.52:0/272131561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fa4076f70 0x7f7fa419f6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f940059d0 tx=0x7f7f940047b0 comp rx=0 tx=0).stop 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3400700 1 -- 172.21.15.52:0/272131561 reap_dead start 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 shutdown_connections 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 >> 172.21.15.52:0/272131561 conn(0x7f7fa4100660 msgr2=0x7f7fa4102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 shutdown_connections 2024-04-26T01:42:44.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:44.006+0000 7f7fa3e00700 1 -- 172.21.15.52:0/272131561 wait complete. 2024-04-26T01:42:45.018 INFO:teuthology.orchestra.run.smithi052.stdout:326417514540 2024-04-26T01:42:45.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T01:42:45.633 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:45 smithi052 ceph-mon[27851]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:45.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:45 smithi191 ceph-mon[32745]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:45 smithi183 ceph-mon[32752]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:45.708 INFO:teuthology.orchestra.run.smithi052.stdout:210453397568 2024-04-26T01:42:45.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 -- 172.21.15.52:0/214491761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c074c70 msgr2=0x7fa04c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 --2- 172.21.15.52:0/214491761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c074c70 0x7fa04c075050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa040009990 tx=0x7fa040009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 -- 172.21.15.52:0/214491761 shutdown_connections 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 --2- 172.21.15.52:0/214491761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04c10f440 0x7fa04c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 --2- 172.21.15.52:0/214491761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c074c70 0x7fa04c075050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 --2- 172.21.15.52:0/214491761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.203+0000 7fa054600700 1 -- 172.21.15.52:0/214491761 >> 172.21.15.52:0/214491761 conn(0x7fa04c079ba0 msgr2=0x7fa04c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:46.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.205+0000 7fa054600700 1 -- 172.21.15.52:0/214491761 shutdown_connections 2024-04-26T01:42:46.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.205+0000 7fa054600700 1 -- 172.21.15.52:0/214491761 wait complete. 2024-04-26T01:42:46.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.206+0000 7fa054600700 1 Processor -- start 2024-04-26T01:42:46.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.206+0000 7fa054600700 1 -- start start 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04c074c70 0x7fa04c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 0x7fa04c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c114e70 con 0x7fa04c10f440 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c114cf0 con 0x7fa04c075590 2024-04-26T01:42:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa054600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c114ff0 con 0x7fa04c074c70 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa051200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.207+0000 7fa052600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 0x7fa04c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c19b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44556/0 (socket says 172.21.15.52:44556) 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa052600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 0x7fa04c19fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40372/0 (socket says 172.21.15.52:40372) 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 -- 172.21.15.52:0/1385026304 learned_addr learned my addr 172.21.15.52:0/1385026304 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:46.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051c00700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04c074c70 0x7fa04c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 -- 172.21.15.52:0/1385026304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04c074c70 msgr2=0x7fa04c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04c074c70 0x7fa04c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 -- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 msgr2=0x7fa04c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 0x7fa04c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c1a02f0 con 0x7fa04c075590 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa052600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04c10f440 0x7fa04c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.208+0000 7fa051200700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c19b820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e970 tx=0x7fa03800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.209+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038004220 con 0x7fa04c075590 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.209+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0380043c0 con 0x7fa04c075590 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.209+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c1a05e0 con 0x7fa04c075590 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.209+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038003ce0 con 0x7fa04c075590 2024-04-26T01:42:46.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.209+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c1a7e60 con 0x7fa04c075590 2024-04-26T01:42:46.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.210+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa04c063190 con 0x7fa04c075590 2024-04-26T01:42:46.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.212+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa03800c7e0 con 0x7fa04c075590 2024-04-26T01:42:46.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.214+0000 7fa04a000700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa0340a3d50 0x7fa0340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fa0380d0770 con 0x7fa04c075590 2024-04-26T01:42:46.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa051c00700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa0340a3d50 0x7fa0340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa04a000700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] conn(0x7fa0340ad890 0x7fa0340afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:46.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa03800be50 con 0x7fa0340ad890 2024-04-26T01:42:46.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fa0380d7050 con 0x7fa04c075590 2024-04-26T01:42:46.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.215+0000 7fa052600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] conn(0x7fa0340ad890 0x7fa0340afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:46.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.216+0000 7fa051c00700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa0340a3d50 0x7fa0340a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa040006010 tx=0x7fa04000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.216+0000 7fa052600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] conn(0x7fa0340ad890 0x7fa0340afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:46.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.217+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== osd.0 v2:172.21.15.52:6802/4212634049 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa03800be50 con 0x7fa0340ad890 2024-04-26T01:42:46.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.236+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 --> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa04c063590 con 0x7fa0340ad890 2024-04-26T01:42:46.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.236+0000 7fa04a000700 1 -- 172.21.15.52:0/1385026304 <== osd.0 v2:172.21.15.52:6802/4212634049 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa04c063590 con 0x7fa0340ad890 2024-04-26T01:42:46.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] conn(0x7fa0340ad890 msgr2=0x7fa0340afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] conn(0x7fa0340ad890 0x7fa0340afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:46.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa0340a3d50 msgr2=0x7fa0340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa0340a3d50 0x7fa0340a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa040006010 tx=0x7fa04000b260 comp rx=0 tx=0).stop 2024-04-26T01:42:46.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 msgr2=0x7fa04c19b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 --2- 172.21.15.52:0/1385026304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa04c075590 0x7fa04c19b820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e970 tx=0x7fa03800ec80 comp rx=0 tx=0).stop 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa052600700 1 -- 172.21.15.52:0/1385026304 reap_dead start 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 shutdown_connections 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 >> 172.21.15.52:0/1385026304 conn(0x7fa04c079ba0 msgr2=0x7fa04c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 shutdown_connections 2024-04-26T01:42:46.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:46.237+0000 7fa054600700 1 -- 172.21.15.52:0/1385026304 wait complete. 2024-04-26T01:42:46.408 INFO:teuthology.orchestra.run.smithi052.stdout:115964117072 2024-04-26T01:42:46.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T01:42:46.945 INFO:teuthology.orchestra.run.smithi052.stdout:55834574943 2024-04-26T01:42:46.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T01:42:47.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:47 smithi052 ceph-mon[27851]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:47.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:47 smithi191 ceph-mon[32745]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:47.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:47 smithi183 ceph-mon[32752]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:47.966 INFO:teuthology.orchestra.run.smithi052.stdout:34359738467 2024-04-26T01:42:47.967 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T01:42:48.726 INFO:teuthology.orchestra.run.smithi052.stdout:360777252903 2024-04-26T01:42:48.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T01:42:48.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 -- 172.21.15.52:0/2529416414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 msgr2=0x7fe44010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 --2- 172.21.15.52:0/2529416414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 0x7fe44010f900 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe43800b0a0 tx=0x7fe43800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 -- 172.21.15.52:0/2529416414 shutdown_connections 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 --2- 172.21.15.52:0/2529416414 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 0x7fe44010f900 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 --2- 172.21.15.52:0/2529416414 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe440108a30 0x7fe440108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 --2- 172.21.15.52:0/2529416414 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 -- 172.21.15.52:0/2529416414 >> 172.21.15.52:0/2529416414 conn(0x7fe4400788f0 msgr2=0x7fe440078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 -- 172.21.15.52:0/2529416414 shutdown_connections 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.729+0000 7fe445c00700 1 -- 172.21.15.52:0/2529416414 wait complete. 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.730+0000 7fe445c00700 1 Processor -- start 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.730+0000 7fe445c00700 1 -- start start 2024-04-26T01:42:48.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 0x7fe44019b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe440108a30 0x7fe44019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe440112bd0 con 0x7fe440108a30 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe440112a50 con 0x7fe44006b0f0 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe445c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe440112d50 con 0x7fe44006a750 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44019b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56112/0 (socket says 172.21.15.52:56112) 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe43ea00700 1 -- 172.21.15.52:0/4053162880 learned_addr learned my addr 172.21.15.52:0/4053162880 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.731+0000 7fe43f400700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe440108a30 0x7fe44019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 -- 172.21.15.52:0/4053162880 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 msgr2=0x7fe44019b820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe44006b0f0 0x7fe44019b820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 -- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe440108a30 msgr2=0x7fe44019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe440108a30 0x7fe44019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4401a02f0 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe43ea00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44019b2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe43000e9a0 tx=0x7fe43000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.732+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43000cbd0 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.733+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe43000cd70 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.733+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430018680 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.733+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4401a05e0 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.734+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4401a7e60 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.735+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe440063110 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.739+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe430010430 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.740+0000 7fe437400700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe4240a3d50 0x7fe4240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.741+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fe4300d0b90 con 0x7fe44006a750 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.741+0000 7fe43e000700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe4240a3d50 0x7fe4240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.741+0000 7fe437400700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] conn(0x7fe4240ad890 0x7fe4240afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.741+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4240b0400 con 0x7fe4240ad890 2024-04-26T01:42:48.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.741+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fe4300d0f40 con 0x7fe44006a750 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.742+0000 7fe43f400700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] conn(0x7fe4240ad890 0x7fe4240afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.743+0000 7fe43e000700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe4240a3d50 0x7fe4240a6210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe440078f80 tx=0x7fe42c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.743+0000 7fe43f400700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] conn(0x7fe4240ad890 0x7fe4240afd30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.745+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== osd.9 v2:172.21.15.52:6826/405746213 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe4240b0400 con 0x7fe4240ad890 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.768+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 --> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe4400635e0 con 0x7fe4240ad890 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.768+0000 7fe437400700 1 -- 172.21.15.52:0/4053162880 <== osd.9 v2:172.21.15.52:6826/405746213 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe4400635e0 con 0x7fe4240ad890 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] conn(0x7fe4240ad890 msgr2=0x7fe4240afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] conn(0x7fe4240ad890 0x7fe4240afd30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe4240a3d50 msgr2=0x7fe4240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe4240a3d50 0x7fe4240a6210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe440078f80 tx=0x7fe42c005c90 comp rx=0 tx=0).stop 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 msgr2=0x7fe44019b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 --2- 172.21.15.52:0/4053162880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe44006a750 0x7fe44019b2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe43000e9a0 tx=0x7fe43000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe43f400700 1 -- 172.21.15.52:0/4053162880 reap_dead start 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 shutdown_connections 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.769+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 >> 172.21.15.52:0/4053162880 conn(0x7fe4400788f0 msgr2=0x7fe44010df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.770+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 shutdown_connections 2024-04-26T01:42:48.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:48.770+0000 7fe445c00700 1 -- 172.21.15.52:0/4053162880 wait complete. 2024-04-26T01:42:49.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:49 smithi191 ceph-mon[32745]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:42:49.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:49 smithi183 ceph-mon[32752]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:42:49.684 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:49 smithi052 ceph-mon[27851]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:42:49.859 INFO:teuthology.orchestra.run.smithi052.stdout:176093659208 2024-04-26T01:42:49.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T01:42:50.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 -- 172.21.15.52:0/988661485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0102320 msgr2=0x7fa5d010e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 --2- 172.21.15.52:0/988661485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0102320 0x7fa5d010e830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b0a0 tx=0x7fa5c800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 -- 172.21.15.52:0/988661485 shutdown_connections 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 --2- 172.21.15.52:0/988661485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0102320 0x7fa5d010e830 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 --2- 172.21.15.52:0/988661485 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0107960 0x7fa5d0107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 --2- 172.21.15.52:0/988661485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 -- 172.21.15.52:0/988661485 >> 172.21.15.52:0/988661485 conn(0x7fa5d00789e0 msgr2=0x7fa5d0078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 -- 172.21.15.52:0/988661485 shutdown_connections 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.296+0000 7fa5d8400700 1 -- 172.21.15.52:0/988661485 wait complete. 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.297+0000 7fa5d8400700 1 Processor -- start 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.297+0000 7fa5d8400700 1 -- start start 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0102320 0x7fa5d019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0107960 0x7fa5d019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d0111bf0 con 0x7fa5d0107960 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d0111a70 con 0x7fa5d0102320 2024-04-26T01:42:50.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d8400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d0111d70 con 0x7fa5d0101960 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d5a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d5000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0102320 0x7fa5d019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d5a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56144/0 (socket says 172.21.15.52:56144) 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d5a00700 1 -- 172.21.15.52:0/2918842149 learned_addr learned my addr 172.21.15.52:0/2918842149 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.298+0000 7fa5d6400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0107960 0x7fa5d019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 -- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0102320 msgr2=0x7fa5d019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0102320 0x7fa5d019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 -- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0107960 msgr2=0x7fa5d019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5d0107960 0x7fa5d019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d01a0350 con 0x7fa5d0101960 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5000700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5d0102320 0x7fa5d019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.299+0000 7fa5d5a00700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d019b390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400e9a0 tx=0x7fa5c400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.300+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400ca10 con 0x7fa5d0101960 2024-04-26T01:42:50.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.300+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d01a0640 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.300+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c400cbb0 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.300+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4018680 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.300+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d01a7ec0 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.301+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5d0105550 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.303+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa5c4010430 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.304+0000 7fa5ce000700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa5b80a3cd0 0x7fa5b80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.304+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fa5c40d0ca0 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5d5000700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa5b80a3cd0 0x7fa5b80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5ce000700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] conn(0x7fa5b80ad810 0x7fa5b80afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5c4016ec0 con 0x7fa5b80ad810 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fa5c40d1050 con 0x7fa5d0101960 2024-04-26T01:42:50.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5d6400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] conn(0x7fa5b80ad810 0x7fa5b80afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.305+0000 7fa5d5000700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa5b80a3cd0 0x7fa5b80a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5d010ec30 tx=0x7fa5bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.306+0000 7fa5d6400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] conn(0x7fa5b80ad810 0x7fa5b80afcb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.307+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== osd.7 v2:172.21.15.183:6816/1224783051 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5c4016ec0 con 0x7fa5b80ad810 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.325+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 --> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa5d01030a0 con 0x7fa5b80ad810 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.325+0000 7fa5ce000700 1 -- 172.21.15.52:0/2918842149 <== osd.7 v2:172.21.15.183:6816/1224783051 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa5d01030a0 con 0x7fa5b80ad810 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] conn(0x7fa5b80ad810 msgr2=0x7fa5b80afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] conn(0x7fa5b80ad810 0x7fa5b80afcb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa5b80a3cd0 msgr2=0x7fa5b80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa5b80a3cd0 0x7fa5b80a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5d010ec30 tx=0x7fa5bc009260 comp rx=0 tx=0).stop 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 msgr2=0x7fa5d019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 --2- 172.21.15.52:0/2918842149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d0101960 0x7fa5d019b390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400e9a0 tx=0x7fa5c400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d6400700 1 -- 172.21.15.52:0/2918842149 reap_dead start 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 shutdown_connections 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 >> 172.21.15.52:0/2918842149 conn(0x7fa5d00789e0 msgr2=0x7fa5d006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 shutdown_connections 2024-04-26T01:42:50.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:50.326+0000 7fa5d8400700 1 -- 172.21.15.52:0/2918842149 wait complete. 2024-04-26T01:42:51.241 INFO:teuthology.orchestra.run.smithi052.stdout:231928234045 2024-04-26T01:42:51.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T01:42:51.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:51 smithi191 ceph-mon[32745]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T01:42:51.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:51 smithi183 ceph-mon[32752]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T01:42:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:51 smithi052 ceph-mon[27851]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T01:42:51.799 INFO:teuthology.orchestra.run.smithi052.stdout:300647710771 2024-04-26T01:42:51.800 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T01:42:51.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.028+0000 7f72ee800700 1 -- 172.21.15.52:0/467042650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e8075790 msgr2=0x7f72e810ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.028+0000 7f72ee800700 1 --2- 172.21.15.52:0/467042650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e8075790 0x7f72e810ec60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009990 tx=0x7f72d4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 -- 172.21.15.52:0/467042650 shutdown_connections 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 --2- 172.21.15.52:0/467042650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e810f1a0 0x7f72e8111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 --2- 172.21.15.52:0/467042650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e8074e70 0x7f72e8075250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 --2- 172.21.15.52:0/467042650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e8075790 0x7f72e810ec60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 -- 172.21.15.52:0/467042650 >> 172.21.15.52:0/467042650 conn(0x7f72e80fe490 msgr2=0x7f72e81008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 -- 172.21.15.52:0/467042650 shutdown_connections 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.029+0000 7f72ee800700 1 -- 172.21.15.52:0/467042650 wait complete. 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.030+0000 7f72ee800700 1 Processor -- start 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.030+0000 7f72ee800700 1 -- start start 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 0x7f72e819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e8075790 0x7f72e819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e810f1a0 0x7f72e819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8114d90 con 0x7f72e810f1a0 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8114c10 con 0x7f72e8074e70 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ee800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8114f10 con 0x7f72e8075790 2024-04-26T01:42:51.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 0x7f72e819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 0x7f72e819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44612/0 (socket says 172.21.15.52:44612) 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 -- 172.21.15.52:0/974093491 learned_addr learned my addr 172.21.15.52:0/974093491 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ede00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e810f1a0 0x7f72e819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72eca00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e8075790 0x7f72e819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 -- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e8075790 msgr2=0x7f72e819b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e8075790 0x7f72e819b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 -- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e810f1a0 msgr2=0x7f72e819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e810f1a0 0x7f72e819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.031+0000 7f72ed400700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e81a0300 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72ed400700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 0x7f72e819b340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00e9c0 tx=0x7f72dc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00cbd0 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc00ee60 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0055c0 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e81a05f0 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.032+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e81a7e70 con 0x7f72e8074e70 2024-04-26T01:42:51.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.033+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72e8063150 con 0x7f72e8074e70 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.036+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72dc004750 con 0x7f72e8074e70 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72e6000700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f72d00a3cd0 0x7f72d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f72dc0d0c40 con 0x7f72e8074e70 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72eca00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f72d00a3cd0 0x7f72d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72e6000700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] conn(0x7f72d00ad810 0x7f72d00afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72dc01ee50 con 0x7f72d00ad810 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.037+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f72dc0d0ff0 con 0x7f72e8074e70 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.038+0000 7f72ede00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] conn(0x7f72d00ad810 0x7f72d00afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.038+0000 7f72eca00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f72d00a3cd0 0x7f72d00a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f72e81105d0 tx=0x7f72d400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.038+0000 7f72ede00700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] conn(0x7f72d00ad810 0x7f72d00afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.039+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== osd.8 v2:172.21.15.191:6816/3371568470 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72dc01ee50 con 0x7f72d00ad810 2024-04-26T01:42:51.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.057+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 --> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f72e80635d0 con 0x7f72d00ad810 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.058+0000 7f72e6000700 1 -- 172.21.15.52:0/974093491 <== osd.8 v2:172.21.15.191:6816/3371568470 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f72e80635d0 con 0x7f72d00ad810 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] conn(0x7f72d00ad810 msgr2=0x7f72d00afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] conn(0x7f72d00ad810 0x7f72d00afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f72d00a3cd0 msgr2=0x7f72d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f72d00a3cd0 0x7f72d00a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f72e81105d0 tx=0x7f72d400b260 comp rx=0 tx=0).stop 2024-04-26T01:42:51.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 msgr2=0x7f72e819b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 --2- 172.21.15.52:0/974093491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e8074e70 0x7f72e819b340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00e9c0 tx=0x7f72dc00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ede00700 1 -- 172.21.15.52:0/974093491 reap_dead start 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 shutdown_connections 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 >> 172.21.15.52:0/974093491 conn(0x7f72e80fe490 msgr2=0x7f72e80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.059+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 shutdown_connections 2024-04-26T01:42:51.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:51.060+0000 7f72ee800700 1 -- 172.21.15.52:0/974093491 wait complete. 2024-04-26T01:42:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:52.247 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:52.483 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:52.657 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:52.718 INFO:teuthology.orchestra.run.smithi052.stdout:266287972409 2024-04-26T01:42:52.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T01:42:53.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:53 smithi052 ceph-mon[27851]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:42:53.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:53 smithi191 ceph-mon[32745]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:42:53.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:53 smithi183 ceph-mon[32752]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:42:53.748 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:55.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:55 smithi191 ceph-mon[32745]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:42:55.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:55 smithi183 ceph-mon[32752]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:42:55.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:55 smithi052 ceph-mon[27851]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:42:56.937 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:56.941 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:42:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:57 smithi052 ceph-mon[27851]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:57.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:57 smithi191 ceph-mon[32745]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:57.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:57 smithi183 ceph-mon[32752]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 -- 172.21.15.52:0/740675935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790406b0f0 msgr2=0x7f790410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 --2- 172.21.15.52:0/740675935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790406b0f0 0x7f790410f900 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00b0a0 tx=0x7f78fc00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 -- 172.21.15.52:0/740675935 shutdown_connections 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 --2- 172.21.15.52:0/740675935 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f790406b0f0 0x7f790410f900 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 --2- 172.21.15.52:0/740675935 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7904108a30 0x7f7904108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 --2- 172.21.15.52:0/740675935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406a750 0x7f790406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 -- 172.21.15.52:0/740675935 >> 172.21.15.52:0/740675935 conn(0x7f79040788f0 msgr2=0x7f7904078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 -- 172.21.15.52:0/740675935 shutdown_connections 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.050+0000 7f790b400700 1 -- 172.21.15.52:0/740675935 wait complete. 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.051+0000 7f790b400700 1 Processor -- start 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.051+0000 7f790b400700 1 -- start start 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 0x7f790419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406b0f0 0x7f790419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7904112c60 con 0x7f7904108a30 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7904112ae0 con 0x7f790406a750 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f790b400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7904112de0 con 0x7f790406b0f0 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f7909400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:59.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f7908a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 0x7f790419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:59.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f78fbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406b0f0 0x7f790419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:59.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f7909400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48694/0 (socket says 172.21.15.52:48694) 2024-04-26T01:42:59.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.052+0000 7f7909400700 1 -- 172.21.15.52:0/4191306690 learned_addr learned my addr 172.21.15.52:0/4191306690 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:42:59.503 INFO:teuthology.orchestra.run.smithi052.stdout:150323855442 2024-04-26T01:42:59.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 -- 172.21.15.52:0/4191306690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406b0f0 msgr2=0x7f790419b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406b0f0 0x7f790419b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 -- 172.21.15.52:0/4191306690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 msgr2=0x7f790419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 0x7f790419b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 -- 172.21.15.52:0/4191306690 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79041a0360 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7908a00700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 0x7f790419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7909400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00b070 tx=0x7f78fc009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc021070 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78fc004e40 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc004630 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.053+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79041a05f0 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.054+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79041a7ed0 con 0x7f7904108a30 2024-04-26T01:42:59.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.055+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7904069730 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.057+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f78fc007980 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.058+0000 7f7902000700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 0x7f78e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.058+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f78fc0d1390 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.058+0000 7f7908a00700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 0x7f78e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.059+0000 7f7908a00700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 0x7f78e40a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009fd0 tx=0x7f78ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.063+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78fc09dc50 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.299+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f790402df80 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.300+0000 7f7902000700 1 -- 172.21.15.52:0/4191306690 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f78fc09d550 con 0x7f7904108a30 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 msgr2=0x7f78e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 0x7f78e40a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009fd0 tx=0x7f78ec009260 comp rx=0 tx=0).stop 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 msgr2=0x7f790419fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00b070 tx=0x7f78fc009c90 comp rx=0 tx=0).stop 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 shutdown_connections 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f78e40a3d50 0x7f78e40a6210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7904108a30 0x7f790419fc20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f790406a750 0x7f790419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 --2- 172.21.15.52:0/4191306690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f790406b0f0 0x7f790419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:42:59.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 >> 172.21.15.52:0/4191306690 conn(0x7f79040788f0 msgr2=0x7f790410df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:42:59.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 shutdown_connections 2024-04-26T01:42:59.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:42:59.303+0000 7f790b400700 1 -- 172.21.15.52:0/4191306690 wait complete. 2024-04-26T01:42:59.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:59 smithi183 ceph-mon[32752]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:59.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:42:59 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4191306690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:42:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:59 smithi052 ceph-mon[27851]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:42:59 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4191306690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:42:59.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:59 smithi191 ceph-mon[32745]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:42:59.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:42:59 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4191306690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T01:43:00.048 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:00.048 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:00.393 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:00.394 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:01.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 -- 172.21.15.52:0/501486039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490074c70 msgr2=0x7f1490075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 --2- 172.21.15.52:0/501486039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490074c70 0x7f1490075050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1484009990 tx=0x7f1484009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 -- 172.21.15.52:0/501486039 shutdown_connections 2024-04-26T01:43:01.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 --2- 172.21.15.52:0/501486039 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 0x7f1490111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 --2- 172.21.15.52:0/501486039 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490074c70 0x7f1490075050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 --2- 172.21.15.52:0/501486039 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490075590 0x7f149010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 -- 172.21.15.52:0/501486039 >> 172.21.15.52:0/501486039 conn(0x7f1490079ba0 msgr2=0x7f1490079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 -- 172.21.15.52:0/501486039 shutdown_connections 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.055+0000 7f1498600700 1 -- 172.21.15.52:0/501486039 wait complete. 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 Processor -- start 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 -- start start 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490074c70 0x7f149019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 0x7f149019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1490114e40 con 0x7f1490075590 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1490114cc0 con 0x7f1490074c70 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1498600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1490114fc0 con 0x7f149010f440 2024-04-26T01:43:01.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1495200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.057+0000 7f1495200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48708/0 (socket says 172.21.15.52:48708) 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 -- 172.21.15.52:0/685692983 learned_addr learned my addr 172.21.15.52:0/685692983 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495c00700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490074c70 0x7f149019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1496600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 0x7f149019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 -- 172.21.15.52:0/685692983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 msgr2=0x7f149019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 0x7f149019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 -- 172.21.15.52:0/685692983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490074c70 msgr2=0x7f149019b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490074c70 0x7f149019b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 -- 172.21.15.52:0/685692983 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14901a0280 con 0x7f1490075590 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.058+0000 7f1495200700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f147c00c990 tx=0x7f147c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.059+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c007940 con 0x7f1490075590 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.059+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f147c00f450 con 0x7f1490075590 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.059+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c018660 con 0x7f1490075590 2024-04-26T01:43:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.059+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14901a0570 con 0x7f1490075590 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.059+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a7df0 con 0x7f1490075590 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.060+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1490069730 con 0x7f1490075590 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.062+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f147c007ae0 con 0x7f1490075590 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.064+0000 7f148e000700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 0x7f14780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.065+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f147c0d03d0 con 0x7f1490075590 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.065+0000 7f1495c00700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 0x7f14780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.065+0000 7f1495c00700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 0x7f14780a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1484000c00 tx=0x7f148400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.069+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f147c014030 con 0x7f1490075590 2024-04-26T01:43:01.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.301+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f14901a7ff0 con 0x7f1490075590 2024-04-26T01:43:01.300 INFO:teuthology.orchestra.run.smithi052.stdout:77309411422 2024-04-26T01:43:01.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.302+0000 7f148e000700 1 -- 172.21.15.52:0/685692983 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f147c09cc00 con 0x7f1490075590 2024-04-26T01:43:01.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.305+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 msgr2=0x7f14780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.305+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 0x7f14780a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1484000c00 tx=0x7f148400b540 comp rx=0 tx=0).stop 2024-04-26T01:43:01.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.305+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 msgr2=0x7f149019b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:01.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.305+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f147c00c990 tx=0x7f147c00cca0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 shutdown_connections 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14780a3d50 0x7f14780a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f149010f440 0x7f149019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1490074c70 0x7f149019b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 --2- 172.21.15.52:0/685692983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1490075590 0x7f149019b7b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 >> 172.21.15.52:0/685692983 conn(0x7f1490079ba0 msgr2=0x7f14900ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 shutdown_connections 2024-04-26T01:43:01.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.306+0000 7f1498600700 1 -- 172.21.15.52:0/685692983 wait complete. 2024-04-26T01:43:01.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:01 smithi052 ceph-mon[27851]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:01.652 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:01 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/685692983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:43:01.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:01 smithi183 ceph-mon[32752]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:01.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:01 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/685692983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:43:01.703 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:01 smithi191 ceph-mon[32745]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:01.703 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:01 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/685692983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T01:43:02.625 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:02 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3845548132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:43:02.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:02 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3845548132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:43:02.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:02 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3845548132' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T01:43:03.167 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855438 got 150323855442 for osd.4 2024-04-26T01:43:03.167 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:03.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 -- 172.21.15.52:0/386991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c10f440 msgr2=0x7f219c111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:03.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 --2- 172.21.15.52:0/386991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c10f440 0x7f219c111830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f219400b0a0 tx=0x7f219400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 -- 172.21.15.52:0/386991483 shutdown_connections 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 --2- 172.21.15.52:0/386991483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c10f440 0x7f219c111830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 --2- 172.21.15.52:0/386991483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c074c70 0x7f219c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 --2- 172.21.15.52:0/386991483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c075590 0x7f219c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 -- 172.21.15.52:0/386991483 >> 172.21.15.52:0/386991483 conn(0x7f219c079ba0 msgr2=0x7f219c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 -- 172.21.15.52:0/386991483 shutdown_connections 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.636+0000 7f21a2e00700 1 -- 172.21.15.52:0/386991483 wait complete. 2024-04-26T01:43:03.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.637+0000 7f21a2e00700 1 Processor -- start 2024-04-26T01:43:03.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 -- start start 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 0x7f219c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c10f440 0x7f219c19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f219c114dd0 con 0x7f219c10f440 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f219c114c50 con 0x7f219c074c70 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f21a2e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f219c114f50 con 0x7f219c075590 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f219b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f219b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44170/0 (socket says 172.21.15.52:44170) 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f219be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 0x7f219c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.638+0000 7f219b400700 1 -- 172.21.15.52:0/3099556355 learned_addr learned my addr 172.21.15.52:0/3099556355 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:03.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f21a0e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c10f440 0x7f219c19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 -- 172.21.15.52:0/3099556355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 msgr2=0x7f219c19b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 0x7f219c19b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 -- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c10f440 msgr2=0x7f219c19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c10f440 0x7f219c19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 -- 172.21.15.52:0/3099556355 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219c1a0220 con 0x7f219c075590 2024-04-26T01:43:03.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219be00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 0x7f219c19b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.639+0000 7f219b400700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f218800e970 tx=0x7f218800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.640+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188004220 con 0x7f219c075590 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.640+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21880043c0 con 0x7f219c075590 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.640+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f219c1a0510 con 0x7f219c075590 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.640+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188003ce0 con 0x7f219c075590 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.640+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219c1a7d90 con 0x7f219c075590 2024-04-26T01:43:03.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.641+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219c069730 con 0x7f219c075590 2024-04-26T01:43:03.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.643+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f218800c7e0 con 0x7f219c075590 2024-04-26T01:43:03.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.644+0000 7f2198c00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 0x7f21800a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:03.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.645+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f21880d07a0 con 0x7f219c075590 2024-04-26T01:43:03.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.645+0000 7f219be00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 0x7f21800a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:03.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.645+0000 7f219be00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 0x7f21800a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f218c009680 tx=0x7f218c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:03.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.649+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f218809cfd0 con 0x7f219c075590 2024-04-26T01:43:03.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:03 smithi183 ceph-mon[32752]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:03.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:03 smithi052 ceph-mon[27851]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:03.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:03 smithi191 ceph-mon[32745]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:03.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.908+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f219c02df80 con 0x7f219c075590 2024-04-26T01:43:03.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.909+0000 7f2198c00700 1 -- 172.21.15.52:0/3099556355 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f218809c8d0 con 0x7f219c075590 2024-04-26T01:43:03.907 INFO:teuthology.orchestra.run.smithi052.stdout:210453397573 2024-04-26T01:43:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 msgr2=0x7f21800a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 0x7f21800a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f218c009680 tx=0x7f218c009260 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 msgr2=0x7f219c19b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f218800e970 tx=0x7f218800ec80 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 shutdown_connections 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f21800a3d50 0x7f21800a6210 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f219c10f440 0x7f219c19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f219c074c70 0x7f219c19b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 --2- 172.21.15.52:0/3099556355 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f219c075590 0x7f219c19b7a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 >> 172.21.15.52:0/3099556355 conn(0x7f219c079ba0 msgr2=0x7f219c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 shutdown_connections 2024-04-26T01:43:03.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:03.912+0000 7f21a2e00700 1 -- 172.21.15.52:0/3099556355 wait complete. 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 -- 172.21.15.52:0/3117283251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e4108a30 msgr2=0x7f60e4108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 --2- 172.21.15.52:0/3117283251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e4108a30 0x7f60e4108e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098f0 tx=0x7f60d8009c00 comp rx=0 tx=0).stop 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 -- 172.21.15.52:0/3117283251 shutdown_connections 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 --2- 172.21.15.52:0/3117283251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e406b0f0 0x7f60e410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 --2- 172.21.15.52:0/3117283251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e4108a30 0x7f60e4108e10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 --2- 172.21.15.52:0/3117283251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406a750 0x7f60e406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 -- 172.21.15.52:0/3117283251 >> 172.21.15.52:0/3117283251 conn(0x7f60e40788f0 msgr2=0x7f60e4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:04.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 -- 172.21.15.52:0/3117283251 shutdown_connections 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.138+0000 7f60ec000700 1 -- 172.21.15.52:0/3117283251 wait complete. 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.139+0000 7f60ec000700 1 Processor -- start 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.139+0000 7f60ec000700 1 -- start start 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 0x7f60e419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 0x7f60e419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4102c60 con 0x7f60e406b0f0 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4102ae0 con 0x7f60e4108a30 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ec000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4102de0 con 0x7f60e406a750 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60e9600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60e8c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 0x7f60e419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60e9600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44192/0 (socket says 172.21.15.52:44192) 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60e9600700 1 -- 172.21.15.52:0/3578912763 learned_addr learned my addr 172.21.15.52:0/3578912763 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.140+0000 7f60ea000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 0x7f60e419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:04.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 -- 172.21.15.52:0/3578912763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 msgr2=0x7f60e419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 0x7f60e419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 -- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 msgr2=0x7f60e419b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 0x7f60e419b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 -- 172.21.15.52:0/3578912763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e41a0420 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60ea000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 0x7f60e419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e8c00700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 0x7f60e419b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e9600700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009230 tx=0x7f60d800fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8005140 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60d800b560 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.141+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e41a06b0 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.142+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8003ae0 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.142+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e41a7f90 con 0x7f60e406a750 2024-04-26T01:43:04.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.143+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e4104170 con 0x7f60e406a750 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.144+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f60d8003c80 con 0x7f60e406a750 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.146+0000 7f60e2000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 0x7f60cc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.146+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f60d80d1500 con 0x7f60e406a750 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.146+0000 7f60e8c00700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 0x7f60cc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.147+0000 7f60e8c00700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 0x7f60cc0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60e4079110 tx=0x7f60d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.150+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60d8013030 con 0x7f60e406a750 2024-04-26T01:43:04.634 INFO:teuthology.orchestra.run.smithi052.stdout:115964117077 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.399+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f60e41a8160 con 0x7f60e406a750 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.400+0000 7f60e2000700 1 -- 172.21.15.52:0/3578912763 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f60d809ddc0 con 0x7f60e406a750 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.403+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 msgr2=0x7f60cc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.403+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 0x7f60cc0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60e4079110 tx=0x7f60d0009260 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.403+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 msgr2=0x7f60e419b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.403+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009230 tx=0x7f60d800fa70 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 shutdown_connections 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f60cc0a3cd0 0x7f60cc0a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f60e4108a30 0x7f60e419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60e406a750 0x7f60e419b460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 --2- 172.21.15.52:0/3578912763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f60e406b0f0 0x7f60e419b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:04.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 >> 172.21.15.52:0/3578912763 conn(0x7f60e40788f0 msgr2=0x7f60e4106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:04.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 shutdown_connections 2024-04-26T01:43:04.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:04.404+0000 7f60ec000700 1 -- 172.21.15.52:0/3578912763 wait complete. 2024-04-26T01:43:04.673 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:04 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:04.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:04 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3099556355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:43:04.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:04 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3578912763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:43:04.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:04 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:04.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:04 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3099556355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:43:04.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:04 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3578912763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:43:04.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:04 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:43:04.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:04 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3099556355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T01:43:04.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:04 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3578912763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T01:43:05.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411418 got 77309411422 for osd.2 2024-04-26T01:43:05.316 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:05.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.530+0000 7f1426200700 1 -- 172.21.15.52:0/3070977162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 msgr2=0x7f14201052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.530+0000 7f1426200700 1 --2- 172.21.15.52:0/3070977162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f14201052a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1410009920 tx=0x7f1410009c30 comp rx=0 tx=0).stop 2024-04-26T01:43:05.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 -- 172.21.15.52:0/3070977162 shutdown_connections 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 --2- 172.21.15.52:0/3070977162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f1420111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 --2- 172.21.15.52:0/3070977162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f14201052a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 --2- 172.21.15.52:0/3070977162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 0x7f14201098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 -- 172.21.15.52:0/3070977162 >> 172.21.15.52:0/3070977162 conn(0x7f1420100750 msgr2=0x7f1420102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 -- 172.21.15.52:0/3070977162 shutdown_connections 2024-04-26T01:43:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.531+0000 7f1426200700 1 -- 172.21.15.52:0/3070977162 wait complete. 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 Processor -- start 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 -- start start 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 0x7f142019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f142019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1420114e80 con 0x7f1420105870 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1420114d00 con 0x7f1420109fa0 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.532+0000 7f1426200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1420115000 con 0x7f1420104ec0 2024-04-26T01:43:05.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41910/0 (socket says 172.21.15.52:41910) 2024-04-26T01:43:05.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 -- 172.21.15.52:0/2483139525 learned_addr learned my addr 172.21.15.52:0/2483139525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:05.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141fe00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f142019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f1416a00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 0x7f142019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 -- 172.21.15.52:0/2483139525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 msgr2=0x7f142019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f142019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 -- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 msgr2=0x7f142019b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 0x7f142019b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141f400700 1 -- 172.21.15.52:0/2483139525 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14201a0400 con 0x7f1420104ec0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.533+0000 7f141fe00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f142019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f141f400700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f141000f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410027070 con 0x7f1420104ec0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14100054f0 con 0x7f1420104ec0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14201a0690 con 0x7f1420104ec0 2024-04-26T01:43:05.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410004930 con 0x7f1420104ec0 2024-04-26T01:43:05.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.534+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14201a7f70 con 0x7f1420104ec0 2024-04-26T01:43:05.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.535+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1420069730 con 0x7f1420104ec0 2024-04-26T01:43:05.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.537+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f141002a070 con 0x7f1420104ec0 2024-04-26T01:43:05.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.540+0000 7f141cc00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 0x7f14040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.540+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f14100d1250 con 0x7f1420104ec0 2024-04-26T01:43:05.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.540+0000 7f1416a00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 0x7f14040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.540+0000 7f1416a00700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 0x7f14040a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f142010b550 tx=0x7f1408009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.541+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1410031020 con 0x7f1420104ec0 2024-04-26T01:43:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 -- 172.21.15.52:0/3721390110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 msgr2=0x7fa6a810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3721390110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a810f8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0009f90 tx=0x7fa6a000b330 comp rx=0 tx=0).stop 2024-04-26T01:43:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 -- 172.21.15.52:0/3721390110 shutdown_connections 2024-04-26T01:43:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3721390110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a810f8b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3721390110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8108a50 0x7fa6a8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3721390110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8102a90 0x7fa6a8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 -- 172.21.15.52:0/3721390110 >> 172.21.15.52:0/3721390110 conn(0x7fa6a8079ba0 msgr2=0x7fa6a8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 -- 172.21.15.52:0/3721390110 shutdown_connections 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.832+0000 7fa6ae400700 1 -- 172.21.15.52:0/3721390110 wait complete. 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.833+0000 7fa6ae400700 1 Processor -- start 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.833+0000 7fa6ae400700 1 -- start start 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 0x7fa6a819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8108a50 0x7fa6a819fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8112c20 con 0x7fa6a8102a90 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8112aa0 con 0x7fa6a8108a50 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6ae400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8112da0 con 0x7fa6a8103450 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6a7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 0x7fa6a819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6a6a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6a6a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44154/0 (socket says 172.21.15.52:44154) 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6a6a00700 1 -- 172.21.15.52:0/3845548132 learned_addr learned my addr 172.21.15.52:0/3845548132 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.834+0000 7fa6a7e00700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8108a50 0x7fa6a819fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 -- 172.21.15.52:0/3845548132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 msgr2=0x7fa6a819b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 -- 172.21.15.52:0/3845548132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8108a50 msgr2=0x7fa6a819fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8108a50 0x7fa6a819fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 -- 172.21.15.52:0/3845548132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a81a02b0 con 0x7fa6a8102a90 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a6a00700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6a7400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 0x7fa6a819b2a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa69800eb90 tx=0x7fa69800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69800cc90 con 0x7fa6a8102a90 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa69800ce30 con 0x7fa6a8102a90 2024-04-26T01:43:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698010430 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.835+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a81a05a0 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.836+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a81a7e20 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.837+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a8069730 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.839+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa698003ef0 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.841+0000 7fa69fe00700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 0x7fa68c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.841+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fa6980d0870 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.841+0000 7fa6a6a00700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 0x7fa68c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.842+0000 7fa6a6a00700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 0x7fa68c0a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa6a807a430 tx=0x7fa694009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:01.843+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa69809d130 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.082+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa6a802df80 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.082+0000 7fa69fe00700 1 -- 172.21.15.52:0/3845548132 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa69809ca30 con 0x7fa6a8102a90 2024-04-26T01:43:05.583 INFO:teuthology.orchestra.run.smithi052.stdout:34359738470 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 msgr2=0x7fa68c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 0x7fa68c0a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa6a807a430 tx=0x7fa694009260 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 msgr2=0x7fa6a819b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 0x7fa6a819b2a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa69800eb90 tx=0x7fa69800eea0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 shutdown_connections 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa68c0a3d50 0x7fa68c0a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6a8108a50 0x7fa6a819fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa6a8102a90 0x7fa6a819b2a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 --2- 172.21.15.52:0/3845548132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6a8103450 0x7fa6a819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 >> 172.21.15.52:0/3845548132 conn(0x7fa6a8079ba0 msgr2=0x7fa6a810df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 shutdown_connections 2024-04-26T01:43:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:02.085+0000 7fa6ae400700 1 -- 172.21.15.52:0/3845548132 wait complete. 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:05 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.777+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f142002df80 con 0x7f1420104ec0 2024-04-26T01:43:05.775 INFO:teuthology.orchestra.run.smithi052.stdout:300647710774 2024-04-26T01:43:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.777+0000 7f141cc00700 1 -- 172.21.15.52:0/2483139525 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1410013030 con 0x7f1420104ec0 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 msgr2=0x7f14040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 0x7f14040a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f142010b550 tx=0x7f1408009260 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 msgr2=0x7f142019b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f141000f990 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 shutdown_connections 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14040a3d50 0x7f14040a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1420109fa0 0x7f142019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420104ec0 0x7f142019b3f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 --2- 172.21.15.52:0/2483139525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1420105870 0x7f142019b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:05.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.780+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 >> 172.21.15.52:0/2483139525 conn(0x7f1420100750 msgr2=0x7f1420101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.781+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 shutdown_connections 2024-04-26T01:43:05.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:05.781+0000 7f1426200700 1 -- 172.21.15.52:0/2483139525 wait complete. 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:05 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:05 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:06.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:06 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2483139525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:43:06.839 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:06.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:06 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2483139525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:43:06.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:06 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2483139525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T01:43:07.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:07 smithi052 ceph-mon[27851]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:07.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710771 got 300647710774 for osd.9 2024-04-26T01:43:07.885 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:07.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:07 smithi191 ceph-mon[32745]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:07.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:07 smithi183 ceph-mon[32752]: pgmap v394: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:08.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738467 got 34359738470 for osd.0 2024-04-26T01:43:08.530 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:09.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:09 smithi052 ceph-mon[27851]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:09.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:09 smithi191 ceph-mon[32745]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:09.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:09 smithi183 ceph-mon[32752]: pgmap v395: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:11.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 -- 172.21.15.52:0/1285032044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78102a90 msgr2=0x7f0a78102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:11.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1285032044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78102a90 0x7f0a78102f10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009990 tx=0x7f0a64009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 -- 172.21.15.52:0/1285032044 shutdown_connections 2024-04-26T01:43:11.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1285032044 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a7810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1285032044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78108a50 0x7f0a78108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1285032044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78102a90 0x7f0a78102f10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 -- 172.21.15.52:0/1285032044 >> 172.21.15.52:0/1285032044 conn(0x7f0a78079ba0 msgr2=0x7f0a78079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 -- 172.21.15.52:0/1285032044 shutdown_connections 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.308+0000 7f0a7d400700 1 -- 172.21.15.52:0/1285032044 wait complete. 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.309+0000 7f0a7d400700 1 Processor -- start 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 -- start start 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78102a90 0x7f0a78074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78108a50 0x7f0a780757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78112cd0 con 0x7f0a78103450 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78112b50 con 0x7f0a78102a90 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a7d400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78112e50 con 0x7f0a78108a50 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a75e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:11.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a75e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35326/0 (socket says 172.21.15.52:35326) 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.310+0000 7f0a75e00700 1 -- 172.21.15.52:0/1589826627 learned_addr learned my addr 172.21.15.52:0/1589826627 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a76800700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78102a90 0x7f0a78074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 -- 172.21.15.52:0/1589826627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78108a50 msgr2=0x7f0a780757c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78108a50 0x7f0a780757c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 -- 172.21.15.52:0/1589826627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78102a90 msgr2=0x7f0a78074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78102a90 0x7f0a78074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 -- 172.21.15.52:0/1589826627 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a781a7790 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.311+0000 7f0a75e00700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400b2e0 tx=0x7f0a640057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.312+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64026070 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.312+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a64021410 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.312+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781a7a20 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.312+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64017650 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.312+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a781a7f50 con 0x7f0a78103450 2024-04-26T01:43:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.313+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a78104180 con 0x7f0a78103450 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.316+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0a64029070 con 0x7f0a78103450 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.318+0000 7f0a6f400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 0x7f0a5c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.318+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f0a640d15d0 con 0x7f0a78103450 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.318+0000 7f0a76800700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 0x7f0a5c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.319+0000 7f0a76800700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 0x7f0a5c0a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006fd0 tx=0x7f0a68009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.320+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a64030080 con 0x7f0a78103450 2024-04-26T01:43:11.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.578+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f0a7802e140 con 0x7f0a78103450 2024-04-26T01:43:11.576 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:11 smithi052 ceph-mon[27851]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:11.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.579+0000 7f0a6f400700 1 -- 172.21.15.52:0/1589826627 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f0a64013070 con 0x7f0a78103450 2024-04-26T01:43:11.577 INFO:teuthology.orchestra.run.smithi052.stdout:326417514546 2024-04-26T01:43:11.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 msgr2=0x7f0a5c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:11.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 0x7f0a5c0a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006fd0 tx=0x7f0a68009190 comp rx=0 tx=0).stop 2024-04-26T01:43:11.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 msgr2=0x7f0a78075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:11.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400b2e0 tx=0x7f0a640057c0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 shutdown_connections 2024-04-26T01:43:11.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f0a5c0a3d50 0x7f0a5c0a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a78108a50 0x7f0a780757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a78102a90 0x7f0a78074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 --2- 172.21.15.52:0/1589826627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0a78103450 0x7f0a78075280 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:11.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 >> 172.21.15.52:0/1589826627 conn(0x7f0a78079ba0 msgr2=0x7f0a78101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:11.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.582+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 shutdown_connections 2024-04-26T01:43:11.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:11.583+0000 7f0a7d400700 1 -- 172.21.15.52:0/1589826627 wait complete. 2024-04-26T01:43:11.726 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:11 smithi183 ceph-mon[32752]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:11.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:11 smithi191 ceph-mon[32745]: pgmap v396: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:12.057 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397568 got 210453397573 for osd.6 2024-04-26T01:43:12.057 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:12.633 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:12 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1589826627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:43:12.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:12 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1589826627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:43:12.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:12 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1589826627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T01:43:13.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:13 smithi052 ceph-mon[27851]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:13.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:13 smithi191 ceph-mon[32745]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:13.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:13 smithi183 ceph-mon[32752]: pgmap v397: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:14.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.001+0000 7fc6ed000700 1 -- 172.21.15.52:0/2328937395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 msgr2=0x7fc6e81096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.001+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2328937395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81096b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009920 tx=0x7fc6d8009c30 comp rx=0 tx=0).stop 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 -- 172.21.15.52:0/2328937395 shutdown_connections 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2328937395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e8115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2328937395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81096b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2328937395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 -- 172.21.15.52:0/2328937395 >> 172.21.15.52:0/2328937395 conn(0x7fc6e8076210 msgr2=0x7fc6e8107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 -- 172.21.15.52:0/2328937395 shutdown_connections 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.002+0000 7fc6ed000700 1 -- 172.21.15.52:0/2328937395 wait complete. 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.003+0000 7fc6ed000700 1 Processor -- start 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.003+0000 7fc6ed000700 1 -- start start 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e81a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e8119290 con 0x7fc6e8109c80 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e8119110 con 0x7fc6e810e3b0 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6ed000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e8119410 con 0x7fc6e81092d0 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6dfe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e81a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6e5e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6e6800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6dfe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e81a5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35336/0 (socket says 172.21.15.52:35336) 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.004+0000 7fc6dfe00700 1 -- 172.21.15.52:0/2251712560 learned_addr learned my addr 172.21.15.52:0/2251712560 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 -- 172.21.15.52:0/2251712560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 msgr2=0x7fc6e819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 -- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 msgr2=0x7fc6e81a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e81a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 -- 172.21.15.52:0/2251712560 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e819fd70 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e6800700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6e5e00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81a5450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d800f950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8027070 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6d80054f0 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e81a0000 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8004930 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.005+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e81ac320 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.006+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e810a7b0 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.009+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc6d802a070 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.010+0000 7fc6dea00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 0x7fc6cc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.010+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fc6d80d11e0 con 0x7fc6e81092d0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.011+0000 7fc6dfe00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 0x7fc6cc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.011+0000 7fc6dfe00700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 0x7fc6cc0a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6e810fdf0 tx=0x7fc6d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:14.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.012+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6d8031080 con 0x7fc6e81092d0 2024-04-26T01:43:14.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.248+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc6e802df80 con 0x7fc6e81092d0 2024-04-26T01:43:14.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.249+0000 7fc6dea00700 1 -- 172.21.15.52:0/2251712560 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc6d8013030 con 0x7fc6e81092d0 2024-04-26T01:43:14.247 INFO:teuthology.orchestra.run.smithi052.stdout:55834574949 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 msgr2=0x7fc6cc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 0x7fc6cc0a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6e810fdf0 tx=0x7fc6d0009260 comp rx=0 tx=0).stop 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 msgr2=0x7fc6e81a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81a5450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d800f950 comp rx=0 tx=0).stop 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 shutdown_connections 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.252+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc6cc0a3d50 0x7fc6cc0a6210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc6e810e3b0 0x7fc6e819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6e81092d0 0x7fc6e81a5450 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 --2- 172.21.15.52:0/2251712560 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6e8109c80 0x7fc6e81a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 >> 172.21.15.52:0/2251712560 conn(0x7fc6e8076210 msgr2=0x7fc6e8077e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:14.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 shutdown_connections 2024-04-26T01:43:14.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.253+0000 7fc6ed000700 1 -- 172.21.15.52:0/2251712560 wait complete. 2024-04-26T01:43:14.292 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514540 got 326417514546 for osd.10 2024-04-26T01:43:14.292 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:14.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:14 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2251712560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 -- 172.21.15.52:0/3131516032 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 msgr2=0x7f45a0102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 --2- 172.21.15.52:0/3131516032 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0102de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f458c009990 tx=0x7f458c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 -- 172.21.15.52:0/3131516032 shutdown_connections 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 --2- 172.21.15.52:0/3131516032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0103320 0x7f45a010f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 --2- 172.21.15.52:0/3131516032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0108960 0x7f45a0108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 --2- 172.21.15.52:0/3131516032 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0102de0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.797+0000 7f45a6800700 1 -- 172.21.15.52:0/3131516032 >> 172.21.15.52:0/3131516032 conn(0x7f45a00fe490 msgr2=0x7f45a01008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:14.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.798+0000 7f45a6800700 1 -- 172.21.15.52:0/3131516032 shutdown_connections 2024-04-26T01:43:14.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.798+0000 7f45a6800700 1 -- 172.21.15.52:0/3131516032 wait complete. 2024-04-26T01:43:14.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 Processor -- start 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 -- start start 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 0x7f45a0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0108960 0x7f45a00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112bf0 con 0x7f45a0108960 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112a70 con 0x7f45a0102960 2024-04-26T01:43:14.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.799+0000 7f45a6800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0112d70 con 0x7f45a0103320 2024-04-26T01:43:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0108960 0x7f45a00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a4a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 0x7f45a0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57412/0 (socket says 172.21.15.52:57412) 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 -- 172.21.15.52:0/2161685902 learned_addr learned my addr 172.21.15.52:0/2161685902 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 -- 172.21.15.52:0/2161685902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 msgr2=0x7f45a0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 0x7f45a0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 -- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0108960 msgr2=0x7f45a00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0108960 0x7f45a00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a5400700 1 -- 172.21.15.52:0/2161685902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a01a75e0 con 0x7f45a0102960 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.800+0000 7f45a4a00700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 0x7f45a0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f45a5400700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f459400e9a0 tx=0x7f459400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459400ca10 con 0x7f45a0102960 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459400cbb0 con 0x7f45a0102960 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01a78d0 con 0x7f45a0102960 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594018680 con 0x7f45a0102960 2024-04-26T01:43:14.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.801+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a7e10 con 0x7f45a0102960 2024-04-26T01:43:14.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.802+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a0069730 con 0x7f45a0102960 2024-04-26T01:43:14.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.804+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4594010430 con 0x7f45a0102960 2024-04-26T01:43:14.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.806+0000 7f459e000700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 0x7f45880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:14.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.806+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f45940d06e0 con 0x7f45a0102960 2024-04-26T01:43:14.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.806+0000 7f45a4a00700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 0x7f45880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:14.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.807+0000 7f45a4a00700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 0x7f45880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45a0100b70 tx=0x7f458c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:14.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:14.811+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f459409cf10 con 0x7f45a0102960 2024-04-26T01:43:14.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:14 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2251712560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:43:14.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:14 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2251712560' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T01:43:15.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.062+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f45a002df80 con 0x7f45a0102960 2024-04-26T01:43:15.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.062+0000 7f459e000700 1 -- 172.21.15.52:0/2161685902 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f459409c810 con 0x7f45a0102960 2024-04-26T01:43:15.060 INFO:teuthology.orchestra.run.smithi052.stdout:360777252909 2024-04-26T01:43:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.065+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 msgr2=0x7f45880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.065+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 0x7f45880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45a0100b70 tx=0x7f458c01a040 comp rx=0 tx=0).stop 2024-04-26T01:43:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.065+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 msgr2=0x7f45a0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.065+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f459400e9a0 tx=0x7f459400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 shutdown_connections 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f45880a3cd0 0x7f45880a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f45a0108960 0x7f45a00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45a0102960 0x7f45a0074d40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 --2- 172.21.15.52:0/2161685902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45a0103320 0x7f45a0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 >> 172.21.15.52:0/2161685902 conn(0x7f45a00fe490 msgr2=0x7f45a00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 shutdown_connections 2024-04-26T01:43:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:15.066+0000 7f45a6800700 1 -- 172.21.15.52:0/2161685902 wait complete. 2024-04-26T01:43:15.814 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:15 smithi052 ceph-mon[27851]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:15.814 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:15 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2161685902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:43:15.887 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252903 got 360777252909 for osd.11 2024-04-26T01:43:15.888 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:15.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:15 smithi191 ceph-mon[32745]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:15.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:15 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2161685902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:43:15.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:15 smithi183 ceph-mon[32752]: pgmap v398: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:15.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:15 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2161685902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T01:43:16.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117072 got 115964117077 for osd.3 2024-04-26T01:43:16.161 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:16.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574943 got 55834574949 for osd.1 2024-04-26T01:43:16.992 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.792+0000 7fa776400700 1 -- 172.21.15.52:0/3881371963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770109940 msgr2=0x7fa770115de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.792+0000 7fa776400700 1 --2- 172.21.15.52:0/3881371963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770109940 0x7fa770115de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa768009f90 tx=0x7fa76800b330 comp rx=0 tx=0).stop 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 -- 172.21.15.52:0/3881371963 shutdown_connections 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 --2- 172.21.15.52:0/3881371963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770109940 0x7fa770115de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 --2- 172.21.15.52:0/3881371963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa77010f030 0x7fa77010f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 --2- 172.21.15.52:0/3881371963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa770108f80 0x7fa770109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 -- 172.21.15.52:0/3881371963 >> 172.21.15.52:0/3881371963 conn(0x7fa7700787d0 msgr2=0x7fa770107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 -- 172.21.15.52:0/3881371963 shutdown_connections 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.793+0000 7fa776400700 1 -- 172.21.15.52:0/3881371963 wait complete. 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 Processor -- start 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 -- start start 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 0x7fa77019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 0x7fa77019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa77010f030 0x7fa7701a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa770119140 con 0x7fa770108f80 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa770118fc0 con 0x7fa770109940 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.794+0000 7fa776400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa7701192c0 con 0x7fa77010f030 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa775000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 0x7fa77019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 0x7fa77019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa775000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 0x7fa77019f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45254/0 (socket says 172.21.15.52:45254) 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa775000700 1 -- 172.21.15.52:0/1324011747 learned_addr learned my addr 172.21.15.52:0/1324011747 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa775a00700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa77010f030 0x7fa7701a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 -- 172.21.15.52:0/1324011747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa77010f030 msgr2=0x7fa7701a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa77010f030 0x7fa7701a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 -- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 msgr2=0x7fa77019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 0x7fa77019f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 -- 172.21.15.52:0/1324011747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7701a4560 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.795+0000 7fa76fe00700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 0x7fa77019fa90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e9f0 tx=0x7fa75c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.796+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00cbd0 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.796+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7701a4850 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.796+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa75c00cd70 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.796+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c018680 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.796+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7701ac0d0 con 0x7fa770109940 2024-04-26T01:43:17.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.797+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa770069730 con 0x7fa770109940 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.799+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa75c010430 con 0x7fa770109940 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.800+0000 7fa76d600700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 0x7fa7580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.801+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fa75c0d0650 con 0x7fa770109940 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.801+0000 7fa775000700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 0x7fa7580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.801+0000 7fa775000700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 0x7fa7580a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa764005950 tx=0x7fa764009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:16.805+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa75c09cfe0 con 0x7fa770109940 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stdout:266287972414 2024-04-26T01:43:17.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.051+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa77002df80 con 0x7fa770109940 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.052+0000 7fa76d600700 1 -- 172.21.15.52:0/1324011747 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa75c09c8e0 con 0x7fa770109940 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 msgr2=0x7fa7580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 0x7fa7580a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa764005950 tx=0x7fa764009310 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 msgr2=0x7fa77019fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 0x7fa77019fa90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e9f0 tx=0x7fa75c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 shutdown_connections 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa7580a3c40 0x7fa7580a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa77010f030 0x7fa7701a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa770108f80 0x7fa77019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 --2- 172.21.15.52:0/1324011747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa770109940 0x7fa77019fa90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:17.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 >> 172.21.15.52:0/1324011747 conn(0x7fa7700787d0 msgr2=0x7fa770079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:17.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 shutdown_connections 2024-04-26T01:43:17.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:17.055+0000 7fa776400700 1 -- 172.21.15.52:0/1324011747 wait complete. 2024-04-26T01:43:17.755 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:17 smithi052 ceph-mon[27851]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:17.755 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:17 smithi052 ceph-mon[27851]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:43:17.755 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:17 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1324011747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:43:17.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:17 smithi191 ceph-mon[32745]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:17.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:17 smithi191 ceph-mon[32745]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:43:17.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:17 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1324011747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:43:17.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:17 smithi183 ceph-mon[32752]: pgmap v399: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:17.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:17 smithi183 ceph-mon[32752]: mgrmap e18: x(active, since 10m), standbys: y, z 2024-04-26T01:43:17.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:17 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1324011747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T01:43:18.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972409 got 266287972414 for osd.8 2024-04-26T01:43:18.851 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:19.631 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:19 smithi052 ceph-mon[27851]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 -- 172.21.15.52:0/1004907033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4102240 msgr2=0x7f4cb4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/1004907033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4102240 0x7f4cb4106710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009990 tx=0x7f4ca0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 -- 172.21.15.52:0/1004907033 shutdown_connections 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/1004907033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4106d30 0x7f4cb41071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/1004907033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4101920 0x7f4cb4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/1004907033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4102240 0x7f4cb4106710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.633+0000 7f4cb9a00700 1 -- 172.21.15.52:0/1004907033 >> 172.21.15.52:0/1004907033 conn(0x7f4cb40fd360 msgr2=0x7f4cb40ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.634+0000 7f4cb9a00700 1 -- 172.21.15.52:0/1004907033 shutdown_connections 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.634+0000 7f4cb9a00700 1 -- 172.21.15.52:0/1004907033 wait complete. 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.635+0000 7f4cb9a00700 1 Processor -- start 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.635+0000 7f4cb9a00700 1 -- start start 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4101920 0x7f4cb419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4106d30 0x7f4cb41a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41191b0 con 0x7f4cb4101920 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb4119030 con 0x7f4cb4106d30 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb4119330 con 0x7f4cb4102240 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40786/0 (socket says 172.21.15.52:40786) 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 -- 172.21.15.52:0/384837503 learned_addr learned my addr 172.21.15.52:0/384837503 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:19.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb9000700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4106d30 0x7f4cb41a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3e00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4101920 0x7f4cb419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 -- 172.21.15.52:0/384837503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4106d30 msgr2=0x7f4cb41a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4106d30 0x7f4cb41a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 -- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4101920 msgr2=0x7f4cb419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4101920 0x7f4cb419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.636+0000 7f4cb3400700 1 -- 172.21.15.52:0/384837503 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb41a4710 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb3400700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000c00 tx=0x7f4ca00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0026070 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb41a49a0 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb41ac280 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca0021410 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.637+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0017650 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.639+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c94005360 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.641+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4ca0029070 con 0x7f4cb4102240 2024-04-26T01:43:19.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.642+0000 7f4cb0c00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.642+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f4ca00d1160 con 0x7f4cb4102240 2024-04-26T01:43:19.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.642+0000 7f4cb3e00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.643+0000 7f4cb3e00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005950 tx=0x7f4ca40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:19.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.646+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ca005dff0 con 0x7f4cb4102240 2024-04-26T01:43:19.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.883+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4c94005650 con 0x7f4cb4102240 2024-04-26T01:43:19.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.884+0000 7f4cb0c00700 1 -- 172.21.15.52:0/384837503 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4ca0013070 con 0x7f4cb4102240 2024-04-26T01:43:19.881 INFO:teuthology.orchestra.run.smithi052.stdout:176093659215 2024-04-26T01:43:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 msgr2=0x7f4c980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4005950 tx=0x7f4ca40058e0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 msgr2=0x7f4cb419fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000c00 tx=0x7f4ca00047b0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 shutdown_connections 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4cb4106d30 0x7f4cb41a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cb4101920 0x7f4cb419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 --2- 172.21.15.52:0/384837503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb4102240 0x7f4cb419fc40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 >> 172.21.15.52:0/384837503 conn(0x7f4cb40fd360 msgr2=0x7f4cb41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 shutdown_connections 2024-04-26T01:43:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.887+0000 7f4cb9a00700 1 -- 172.21.15.52:0/384837503 wait complete. 2024-04-26T01:43:19.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:19 smithi191 ceph-mon[32745]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:43:19.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:19 smithi183 ceph-mon[32752]: pgmap v400: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.532+0000 7f4382200700 1 -- 172.21.15.52:0/1683450029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c113c10 msgr2=0x7f437c116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.532+0000 7f4382200700 1 --2- 172.21.15.52:0/1683450029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c113c10 0x7f437c116000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f437400b0a0 tx=0x7f437400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 -- 172.21.15.52:0/1683450029 shutdown_connections 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 --2- 172.21.15.52:0/1683450029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c113c10 0x7f437c116000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 --2- 172.21.15.52:0/1683450029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 --2- 172.21.15.52:0/1683450029 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c075590 0x7f437c1136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 -- 172.21.15.52:0/1683450029 >> 172.21.15.52:0/1683450029 conn(0x7f437c079ba0 msgr2=0x7f437c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 -- 172.21.15.52:0/1683450029 shutdown_connections 2024-04-26T01:43:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.533+0000 7f4382200700 1 -- 172.21.15.52:0/1683450029 wait complete. 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.534+0000 7f4382200700 1 Processor -- start 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.534+0000 7f4382200700 1 -- start start 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 0x7f437c1a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f437c119240 con 0x7f437c075590 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1190c0 con 0x7f437c113c10 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f4382200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f437c1193c0 con 0x7f437c074c70 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f437aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f437b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f437b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40758/0 (socket says 172.21.15.52:40758) 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f437aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45276/0 (socket says 172.21.15.52:45276) 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.535+0000 7f437b400700 1 -- 172.21.15.52:0/2111638980 learned_addr learned my addr 172.21.15.52:0/2111638980 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 0x7f437c1a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 -- 172.21.15.52:0/2111638980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 msgr2=0x7f437c19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 -- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 msgr2=0x7f437c19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 -- 172.21.15.52:0/2111638980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c1a4740 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437b400700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437aa00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.536+0000 7f437be00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 0x7f437c1a4000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f437400b070 tx=0x7f4374004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.537+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374021070 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.537+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4374007e30 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.537+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c1a49d0 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.537+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43740045f0 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.537+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c1ac2b0 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.538+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437c069730 con 0x7f437c113c10 2024-04-26T01:43:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.540+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4374007980 con 0x7f437c113c10 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.542+0000 7f4373e00700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 0x7f43600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.542+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f437400f020 con 0x7f437c113c10 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.542+0000 7f437b400700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 0x7f43600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.543+0000 7f437b400700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 0x7f43600a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f436c011590 tx=0x7f436c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.545+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f437402a020 con 0x7f437c113c10 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stdout:231928234050 2024-04-26T01:43:19.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.792+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f437c02df80 con 0x7f437c113c10 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.793+0000 7f4373e00700 1 -- 172.21.15.52:0/2111638980 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f437409e060 con 0x7f437c113c10 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 msgr2=0x7f43600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 0x7f43600a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f436c011590 tx=0x7f436c009260 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 msgr2=0x7f437c1a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 0x7f437c1a4000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f437400b070 tx=0x7f4374004c80 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 shutdown_connections 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f43600a3cc0 0x7f43600a6180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f437c113c10 0x7f437c1a4000 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c074c70 0x7f437c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 --2- 172.21.15.52:0/2111638980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f437c075590 0x7f437c19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 >> 172.21.15.52:0/2111638980 conn(0x7f437c079ba0 msgr2=0x7f437c110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 shutdown_connections 2024-04-26T01:43:19.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:19.797+0000 7f4382200700 1 -- 172.21.15.52:0/2111638980 wait complete. 2024-04-26T01:43:20.663 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:20 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2111638980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:43:20.664 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:20 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/384837503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:43:20.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:20 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2111638980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:43:20.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:20 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/384837503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:43:20.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:20 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2111638980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T01:43:20.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:20 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/384837503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T01:43:21.061 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234045 got 231928234050 for osd.7 2024-04-26T01:43:21.061 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:21.532 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659208 got 176093659215 for osd.5 2024-04-26T01:43:21.532 DEBUG:teuthology.parallel:result is None 2024-04-26T01:43:21.532 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:43:21.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:43:21.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:21 smithi052 ceph-mon[27851]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:43:21.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:21 smithi191 ceph-mon[32745]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:43:21.923 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:21 smithi183 ceph-mon[32752]: pgmap v401: 97 pgs: 97 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:43:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:22.806 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:22 smithi052 ceph-mon[27851]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:22.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:22 smithi191 ceph-mon[32745]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:22.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:22 smithi183 ceph-mon[32752]: pgmap v402: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 -- 172.21.15.52:0/42334928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 msgr2=0x7fdbbc075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/42334928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc075050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009990 tx=0x7fdbb0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 -- 172.21.15.52:0/42334928 shutdown_connections 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/42334928 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 0x7fdbbc111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/42334928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc075050 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/42334928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 0x7fdbbc10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 -- 172.21.15.52:0/42334928 >> 172.21.15.52:0/42334928 conn(0x7fdbbc079bf0 msgr2=0x7fdbbc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:24.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 -- 172.21.15.52:0/42334928 shutdown_connections 2024-04-26T01:43:24.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.702+0000 7fdbc3a00700 1 -- 172.21.15.52:0/42334928 wait complete. 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 Processor -- start 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 -- start start 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 0x7fdbbc19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 0x7fdbbc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc114e90 con 0x7fdbbc10f440 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc114d10 con 0x7fdbbc074c70 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.704+0000 7fdbc3a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc115010 con 0x7fdbbc075590 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:24.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49844/0 (socket says 172.21.15.52:49844) 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 0x7fdbbc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 -- 172.21.15.52:0/2810907311 learned_addr learned my addr 172.21.15.52:0/2810907311 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbbbe00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 0x7fdbbc19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 -- 172.21.15.52:0/2810907311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 msgr2=0x7fdbbc19b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 0x7fdbbc19b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 -- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 msgr2=0x7fdbbc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 0x7fdbbc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.705+0000 7fdbc1000700 1 -- 172.21.15.52:0/2810907311 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc1a03b0 con 0x7fdbbc074c70 2024-04-26T01:43:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbc1000700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0027070 con 0x7fdbbc074c70 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbb0022470 con 0x7fdbbc074c70 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbbc1a0640 con 0x7fdbbc074c70 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0017650 con 0x7fdbbc074c70 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.706+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbbc1a7f20 con 0x7fdbbc074c70 2024-04-26T01:43:24.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.707+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbbc069730 con 0x7fdbbc074c70 2024-04-26T01:43:24.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.709+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdbb002a070 con 0x7fdbbc074c70 2024-04-26T01:43:24.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.711+0000 7fdbb9600700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 0x7fdba40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:24.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.711+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fdbb00d0ff0 con 0x7fdbbc074c70 2024-04-26T01:43:24.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.711+0000 7fdbbbe00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 0x7fdba40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:24.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.712+0000 7fdbbbe00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 0x7fdba40a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009680 tx=0x7fdba8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:24.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.713+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbb0013070 con 0x7fdbbc074c70 2024-04-26T01:43:24.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.953+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdbbc19c700 con 0x7fdba40a3cc0 2024-04-26T01:43:24.951 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:24 smithi052 ceph-mon[27851]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:24.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.962+0000 7fdbb9600700 1 -- 172.21.15.52:0/2810907311 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258911 (secure 0 0 0) 0x7fdbbc19c700 con 0x7fdba40a3cc0 2024-04-26T01:43:24.965 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 msgr2=0x7fdba40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 0x7fdba40a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009680 tx=0x7fdba8009260 comp rx=0 tx=0).stop 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 msgr2=0x7fdbbc19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb00057c0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 shutdown_connections 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fdba40a3cc0 0x7fdba40a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdbbc10f440 0x7fdbbc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdbbc074c70 0x7fdbbc19b3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 --2- 172.21.15.52:0/2810907311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbbc075590 0x7fdbbc19b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:24.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 >> 172.21.15.52:0/2810907311 conn(0x7fdbbc079bf0 msgr2=0x7fdbbc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:24.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 shutdown_connections 2024-04-26T01:43:24.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:24.969+0000 7fdbc3a00700 1 -- 172.21.15.52:0/2810907311 wait complete. 2024-04-26T01:43:24.969 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:43:25.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:24 smithi191 ceph-mon[32745]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:25.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:24 smithi183 ceph-mon[32752]: pgmap v403: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:25.857 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-04-26T01:43:23.719331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659717,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":853,"num_read_kb":550,"num_write":358,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392360,"kb_used_data":14088,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549464,"statfs":{"total":1151940427776,"available":1151538651136,"internally_reserved":0,"allocated":14426112,"data_stored":7944891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":42,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004866"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689522+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:43:15.689522+0000","last_peered":"2024-04-26T01:43:15.689522+0000","last_clean":"2024-04-26T01:43:15.689522+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:43:15.689522+0000","last_undegraded":"2024-04-26T01:43:15.689522+0000","last_fullsized":"2024-04-26T01:43:15.689522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671617+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:43:15.671617+0000","last_peered":"2024-04-26T01:43:15.671617+0000","last_clean":"2024-04-26T01:43:15.671617+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:43:15.671617+0000","last_undegraded":"2024-04-26T01:43:15.671617+0000","last_fullsized":"2024-04-26T01:43:15.671617+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684407+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:43:15.684407+0000","last_peered":"2024-04-26T01:43:15.684407+0000","last_clean":"2024-04-26T01:43:15.684407+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:43:15.684407+0000","last_undegraded":"2024-04-26T01:43:15.684407+0000","last_fullsized":"2024-04-26T01:43:15.684407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687731+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:43:15.687731+0000","last_peered":"2024-04-26T01:43:15.687731+0000","last_clean":"2024-04-26T01:43:15.687731+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:43:15.687731+0000","last_undegraded":"2024-04-26T01:43:15.687731+0000","last_fullsized":"2024-04-26T01:43:15.687731+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.669756+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:43:15.669756+0000","last_peered":"2024-04-26T01:43:15.669756+0000","last_clean":"2024-04-26T01:43:15.669756+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:43:15.669756+0000","last_undegraded":"2024-04-26T01:43:15.669756+0000","last_fullsized":"2024-04-26T01:43:15.669756+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685463+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:43:15.685463+0000","last_peered":"2024-04-26T01:43:15.685463+0000","last_clean":"2024-04-26T01:43:15.685463+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:43:15.685463+0000","last_undegraded":"2024-04-26T01:43:15.685463+0000","last_fullsized":"2024-04-26T01:43:15.685463+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.681594+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:43:15.681594+0000","last_peered":"2024-04-26T01:43:15.681594+0000","last_clean":"2024-04-26T01:43:15.681594+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:43:15.681594+0000","last_undegraded":"2024-04-26T01:43:15.681594+0000","last_fullsized":"2024-04-26T01:43:15.681594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683271+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:43:15.683271+0000","last_peered":"2024-04-26T01:43:15.683271+0000","last_clean":"2024-04-26T01:43:15.683271+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:43:15.683271+0000","last_undegraded":"2024-04-26T01:43:15.683271+0000","last_fullsized":"2024-04-26T01:43:15.683271+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.679007+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:43:15.679007+0000","last_peered":"2024-04-26T01:43:15.679007+0000","last_clean":"2024-04-26T01:43:15.679007+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:43:15.679007+0000","last_undegraded":"2024-04-26T01:43:15.679007+0000","last_fullsized":"2024-04-26T01:43:15.679007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.676262+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:43:15.676262+0000","last_peered":"2024-04-26T01:43:15.676262+0000","last_clean":"2024-04-26T01:43:15.676262+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:43:15.676262+0000","last_undegraded":"2024-04-26T01:43:15.676262+0000","last_fullsized":"2024-04-26T01:43:15.676262+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.666703+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:43:15.666703+0000","last_peered":"2024-04-26T01:43:15.666703+0000","last_clean":"2024-04-26T01:43:15.666703+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:43:15.666703+0000","last_undegraded":"2024-04-26T01:43:15.666703+0000","last_fullsized":"2024-04-26T01:43:15.666703+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.663780+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:43:15.663780+0000","last_peered":"2024-04-26T01:43:15.663780+0000","last_clean":"2024-04-26T01:43:15.663780+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:43:15.663780+0000","last_undegraded":"2024-04-26T01:43:15.663780+0000","last_fullsized":"2024-04-26T01:43:15.663780+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689087+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:43:15.689087+0000","last_peered":"2024-04-26T01:43:15.689087+0000","last_clean":"2024-04-26T01:43:15.689087+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:43:15.689087+0000","last_undegraded":"2024-04-26T01:43:15.689087+0000","last_fullsized":"2024-04-26T01:43:15.689087+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.672017+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:43:15.672017+0000","last_peered":"2024-04-26T01:43:15.672017+0000","last_clean":"2024-04-26T01:43:15.672017+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:43:15.672017+0000","last_undegraded":"2024-04-26T01:43:15.672017+0000","last_fullsized":"2024-04-26T01:43:15.672017+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683459+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:43:15.683459+0000","last_peered":"2024-04-26T01:43:15.683459+0000","last_clean":"2024-04-26T01:43:15.683459+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:43:15.683459+0000","last_undegraded":"2024-04-26T01:43:15.683459+0000","last_fullsized":"2024-04-26T01:43:15.683459+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688308+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:43:15.688308+0000","last_peered":"2024-04-26T01:43:15.688308+0000","last_clean":"2024-04-26T01:43:15.688308+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:43:15.688308+0000","last_undegraded":"2024-04-26T01:43:15.688308+0000","last_fullsized":"2024-04-26T01:43:15.688308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689860+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:43:15.689860+0000","last_peered":"2024-04-26T01:43:15.689860+0000","last_clean":"2024-04-26T01:43:15.689860+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:43:15.689860+0000","last_undegraded":"2024-04-26T01:43:15.689860+0000","last_fullsized":"2024-04-26T01:43:15.689860+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.681869+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:43:15.681869+0000","last_peered":"2024-04-26T01:43:15.681869+0000","last_clean":"2024-04-26T01:43:15.681869+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:43:15.681869+0000","last_undegraded":"2024-04-26T01:43:15.681869+0000","last_fullsized":"2024-04-26T01:43:15.681869+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671856+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:43:15.671856+0000","last_peered":"2024-04-26T01:43:15.671856+0000","last_clean":"2024-04-26T01:43:15.671856+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:43:15.671856+0000","last_undegraded":"2024-04-26T01:43:15.671856+0000","last_fullsized":"2024-04-26T01:43:15.671856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498749+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:41:47.498749+0000","last_peered":"2024-04-26T01:41:47.498749+0000","last_clean":"2024-04-26T01:41:47.498749+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:41:47.498749+0000","last_undegraded":"2024-04-26T01:41:47.498749+0000","last_fullsized":"2024-04-26T01:41:47.498749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684057+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:43:15.684057+0000","last_peered":"2024-04-26T01:43:15.684057+0000","last_clean":"2024-04-26T01:43:15.684057+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:43:15.684057+0000","last_undegraded":"2024-04-26T01:43:15.684057+0000","last_fullsized":"2024-04-26T01:43:15.684057+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510500+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:41:47.510500+0000","last_peered":"2024-04-26T01:41:47.510500+0000","last_clean":"2024-04-26T01:41:47.510500+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:41:47.510500+0000","last_undegraded":"2024-04-26T01:41:47.510500+0000","last_fullsized":"2024-04-26T01:41:47.510500+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.691590+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:43:15.691590+0000","last_peered":"2024-04-26T01:43:15.691590+0000","last_clean":"2024-04-26T01:43:15.691590+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:43:15.691590+0000","last_undegraded":"2024-04-26T01:43:15.691590+0000","last_fullsized":"2024-04-26T01:43:15.691590+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.506427+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:41:47.506427+0000","last_peered":"2024-04-26T01:41:47.506427+0000","last_clean":"2024-04-26T01:41:47.506427+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:41:47.506427+0000","last_undegraded":"2024-04-26T01:41:47.506427+0000","last_fullsized":"2024-04-26T01:41:47.506427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687010+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:43:15.687010+0000","last_peered":"2024-04-26T01:43:15.687010+0000","last_clean":"2024-04-26T01:43:15.687010+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:43:15.687010+0000","last_undegraded":"2024-04-26T01:43:15.687010+0000","last_fullsized":"2024-04-26T01:43:15.687010+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.500059+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:41:47.500059+0000","last_peered":"2024-04-26T01:41:47.500059+0000","last_clean":"2024-04-26T01:41:47.500059+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:41:47.500059+0000","last_undegraded":"2024-04-26T01:41:47.500059+0000","last_fullsized":"2024-04-26T01:41:47.500059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.649115+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:43:15.649115+0000","last_peered":"2024-04-26T01:43:15.649115+0000","last_clean":"2024-04-26T01:43:15.649115+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:43:15.649115+0000","last_undegraded":"2024-04-26T01:43:15.649115+0000","last_fullsized":"2024-04-26T01:43:15.649115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":147,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:19.564074+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:43:19.564074+0000","last_peered":"2024-04-26T01:43:19.564074+0000","last_clean":"2024-04-26T01:43:19.564074+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:43:19.564074+0000","last_undegraded":"2024-04-26T01:43:19.564074+0000","last_fullsized":"2024-04-26T01:43:19.564074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688332+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:43:15.688332+0000","last_peered":"2024-04-26T01:43:15.688332+0000","last_clean":"2024-04-26T01:43:15.688332+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:43:15.688332+0000","last_undegraded":"2024-04-26T01:43:15.688332+0000","last_fullsized":"2024-04-26T01:43:15.688332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508594+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:41:47.508594+0000","last_peered":"2024-04-26T01:41:47.508594+0000","last_clean":"2024-04-26T01:41:47.508594+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:41:47.508594+0000","last_undegraded":"2024-04-26T01:41:47.508594+0000","last_fullsized":"2024-04-26T01:41:47.508594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671152+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:43:15.671152+0000","last_peered":"2024-04-26T01:43:15.671152+0000","last_clean":"2024-04-26T01:43:15.671152+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:43:15.671152+0000","last_undegraded":"2024-04-26T01:43:15.671152+0000","last_fullsized":"2024-04-26T01:43:15.671152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494651+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:41:47.494651+0000","last_peered":"2024-04-26T01:41:47.494651+0000","last_clean":"2024-04-26T01:41:47.494651+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:41:47.494651+0000","last_undegraded":"2024-04-26T01:41:47.494651+0000","last_fullsized":"2024-04-26T01:41:47.494651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690931+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:43:15.690931+0000","last_peered":"2024-04-26T01:43:15.690931+0000","last_clean":"2024-04-26T01:43:15.690931+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:43:15.690931+0000","last_undegraded":"2024-04-26T01:43:15.690931+0000","last_fullsized":"2024-04-26T01:43:15.690931+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508108+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:41:47.508108+0000","last_peered":"2024-04-26T01:41:47.508108+0000","last_clean":"2024-04-26T01:41:47.508108+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:41:47.508108+0000","last_undegraded":"2024-04-26T01:41:47.508108+0000","last_fullsized":"2024-04-26T01:41:47.508108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685894+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:43:15.685894+0000","last_peered":"2024-04-26T01:43:15.685894+0000","last_clean":"2024-04-26T01:43:15.685894+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:43:15.685894+0000","last_undegraded":"2024-04-26T01:43:15.685894+0000","last_fullsized":"2024-04-26T01:43:15.685894+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":145,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:21.566254+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:43:21.566254+0000","last_peered":"2024-04-26T01:43:21.566254+0000","last_clean":"2024-04-26T01:43:21.566254+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:43:21.566254+0000","last_undegraded":"2024-04-26T01:43:21.566254+0000","last_fullsized":"2024-04-26T01:43:21.566254+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688947+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:43:15.688947+0000","last_peered":"2024-04-26T01:43:15.688947+0000","last_clean":"2024-04-26T01:43:15.688947+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:43:15.688947+0000","last_undegraded":"2024-04-26T01:43:15.688947+0000","last_fullsized":"2024-04-26T01:43:15.688947+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499680+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:41:47.499680+0000","last_peered":"2024-04-26T01:41:47.499680+0000","last_clean":"2024-04-26T01:41:47.499680+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:41:47.499680+0000","last_undegraded":"2024-04-26T01:41:47.499680+0000","last_fullsized":"2024-04-26T01:41:47.499680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687839+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:43:15.687839+0000","last_peered":"2024-04-26T01:43:15.687839+0000","last_clean":"2024-04-26T01:43:15.687839+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:43:15.687839+0000","last_undegraded":"2024-04-26T01:43:15.687839+0000","last_fullsized":"2024-04-26T01:43:15.687839+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508753+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:41:47.508753+0000","last_peered":"2024-04-26T01:41:47.508753+0000","last_clean":"2024-04-26T01:41:47.508753+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:41:47.508753+0000","last_undegraded":"2024-04-26T01:41:47.508753+0000","last_fullsized":"2024-04-26T01:41:47.508753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.679923+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:43:15.679923+0000","last_peered":"2024-04-26T01:43:15.679923+0000","last_clean":"2024-04-26T01:43:15.679923+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:43:15.679923+0000","last_undegraded":"2024-04-26T01:43:15.679923+0000","last_fullsized":"2024-04-26T01:43:15.679923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'35","reported_seq":304,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:22.567914+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:43:22.567914+0000","last_peered":"2024-04-26T01:43:22.567914+0000","last_clean":"2024-04-26T01:43:22.567914+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:43:22.567914+0000","last_undegraded":"2024-04-26T01:43:22.567914+0000","last_fullsized":"2024-04-26T01:43:22.567914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64973,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":242,"num_read_kb":184,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.651382+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:43:15.651382+0000","last_peered":"2024-04-26T01:43:15.651382+0000","last_clean":"2024-04-26T01:43:15.651382+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:43:15.651382+0000","last_undegraded":"2024-04-26T01:43:15.651382+0000","last_fullsized":"2024-04-26T01:43:15.651382+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494906+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:41:47.494906+0000","last_peered":"2024-04-26T01:41:47.494906+0000","last_clean":"2024-04-26T01:41:47.494906+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:41:47.494906+0000","last_undegraded":"2024-04-26T01:41:47.494906+0000","last_fullsized":"2024-04-26T01:41:47.494906+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690289+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:43:15.690289+0000","last_peered":"2024-04-26T01:43:15.690289+0000","last_clean":"2024-04-26T01:43:15.690289+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:43:15.690289+0000","last_undegraded":"2024-04-26T01:43:15.690289+0000","last_fullsized":"2024-04-26T01:43:15.690289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508390+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:41:47.508390+0000","last_peered":"2024-04-26T01:41:47.508390+0000","last_clean":"2024-04-26T01:41:47.508390+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:41:47.508390+0000","last_undegraded":"2024-04-26T01:41:47.508390+0000","last_fullsized":"2024-04-26T01:41:47.508390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687656+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:43:15.687656+0000","last_peered":"2024-04-26T01:43:15.687656+0000","last_clean":"2024-04-26T01:43:15.687656+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:43:15.687656+0000","last_undegraded":"2024-04-26T01:43:15.687656+0000","last_fullsized":"2024-04-26T01:43:15.687656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'5","reported_seq":150,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:20.703663+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:43:20.703663+0000","last_peered":"2024-04-26T01:43:20.703663+0000","last_clean":"2024-04-26T01:43:20.703663+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:43:20.703663+0000","last_undegraded":"2024-04-26T01:43:20.703663+0000","last_fullsized":"2024-04-26T01:43:20.703663+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670756+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:43:15.670756+0000","last_peered":"2024-04-26T01:43:15.670756+0000","last_clean":"2024-04-26T01:43:15.670756+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:43:15.670756+0000","last_undegraded":"2024-04-26T01:43:15.670756+0000","last_fullsized":"2024-04-26T01:43:15.670756+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499052+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:41:47.499052+0000","last_peered":"2024-04-26T01:41:47.499052+0000","last_clean":"2024-04-26T01:41:47.499052+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:41:47.499052+0000","last_undegraded":"2024-04-26T01:41:47.499052+0000","last_fullsized":"2024-04-26T01:41:47.499052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684019+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:43:15.684019+0000","last_peered":"2024-04-26T01:43:15.684019+0000","last_clean":"2024-04-26T01:43:15.684019+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:43:15.684019+0000","last_undegraded":"2024-04-26T01:43:15.684019+0000","last_fullsized":"2024-04-26T01:43:15.684019+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499658+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:41:47.499658+0000","last_peered":"2024-04-26T01:41:47.499658+0000","last_clean":"2024-04-26T01:41:47.499658+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:41:47.499658+0000","last_undegraded":"2024-04-26T01:41:47.499658+0000","last_fullsized":"2024-04-26T01:41:47.499658+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670784+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:43:15.670784+0000","last_peered":"2024-04-26T01:43:15.670784+0000","last_clean":"2024-04-26T01:43:15.670784+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:43:15.670784+0000","last_undegraded":"2024-04-26T01:43:15.670784+0000","last_fullsized":"2024-04-26T01:43:15.670784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509508+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:41:47.509508+0000","last_peered":"2024-04-26T01:41:47.509508+0000","last_clean":"2024-04-26T01:41:47.509508+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:41:47.509508+0000","last_undegraded":"2024-04-26T01:41:47.509508+0000","last_fullsized":"2024-04-26T01:41:47.509508+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687462+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:43:15.687462+0000","last_peered":"2024-04-26T01:43:15.687462+0000","last_clean":"2024-04-26T01:43:15.687462+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:43:15.687462+0000","last_undegraded":"2024-04-26T01:43:15.687462+0000","last_fullsized":"2024-04-26T01:43:15.687462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499492+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:41:47.499492+0000","last_peered":"2024-04-26T01:41:47.499492+0000","last_clean":"2024-04-26T01:41:47.499492+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:41:47.499492+0000","last_undegraded":"2024-04-26T01:41:47.499492+0000","last_fullsized":"2024-04-26T01:41:47.499492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687115+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:43:15.687115+0000","last_peered":"2024-04-26T01:43:15.687115+0000","last_clean":"2024-04-26T01:43:15.687115+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:43:15.687115+0000","last_undegraded":"2024-04-26T01:43:15.687115+0000","last_fullsized":"2024-04-26T01:43:15.687115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.650937+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:43:15.650937+0000","last_peered":"2024-04-26T01:43:15.650937+0000","last_clean":"2024-04-26T01:43:15.650937+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:43:15.650937+0000","last_undegraded":"2024-04-26T01:43:15.650937+0000","last_fullsized":"2024-04-26T01:43:15.650937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":144,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:17.695910+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:43:17.695910+0000","last_peered":"2024-04-26T01:43:17.695910+0000","last_clean":"2024-04-26T01:43:17.695910+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:43:17.695910+0000","last_undegraded":"2024-04-26T01:43:17.695910+0000","last_fullsized":"2024-04-26T01:43:17.695910+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690660+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:43:15.690660+0000","last_peered":"2024-04-26T01:43:15.690660+0000","last_clean":"2024-04-26T01:43:15.690660+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:43:15.690660+0000","last_undegraded":"2024-04-26T01:43:15.690660+0000","last_fullsized":"2024-04-26T01:43:15.690660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497492+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:41:47.497492+0000","last_peered":"2024-04-26T01:41:47.497492+0000","last_clean":"2024-04-26T01:41:47.497492+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:41:47.497492+0000","last_undegraded":"2024-04-26T01:41:47.497492+0000","last_fullsized":"2024-04-26T01:41:47.497492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499373+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:41:47.499373+0000","last_peered":"2024-04-26T01:41:47.499373+0000","last_clean":"2024-04-26T01:41:47.499373+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:41:47.499373+0000","last_undegraded":"2024-04-26T01:41:47.499373+0000","last_fullsized":"2024-04-26T01:41:47.499373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.664392+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:43:15.664392+0000","last_peered":"2024-04-26T01:43:15.664392+0000","last_clean":"2024-04-26T01:43:15.664392+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:43:15.664392+0000","last_undegraded":"2024-04-26T01:43:15.664392+0000","last_fullsized":"2024-04-26T01:43:15.664392+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499589+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:41:47.499589+0000","last_peered":"2024-04-26T01:41:47.499589+0000","last_clean":"2024-04-26T01:41:47.499589+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:41:47.499589+0000","last_undegraded":"2024-04-26T01:41:47.499589+0000","last_fullsized":"2024-04-26T01:41:47.499589+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499443+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:41:47.499443+0000","last_peered":"2024-04-26T01:41:47.499443+0000","last_clean":"2024-04-26T01:41:47.499443+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:41:47.499443+0000","last_undegraded":"2024-04-26T01:41:47.499443+0000","last_fullsized":"2024-04-26T01:41:47.499443+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689438+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:43:15.689438+0000","last_peered":"2024-04-26T01:43:15.689438+0000","last_clean":"2024-04-26T01:43:15.689438+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:43:15.689438+0000","last_undegraded":"2024-04-26T01:43:15.689438+0000","last_fullsized":"2024-04-26T01:43:15.689438+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'13","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:10.292229+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:43:10.292229+0000","last_peered":"2024-04-26T01:43:10.292229+0000","last_clean":"2024-04-26T01:43:10.292229+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:43:10.292229+0000","last_undegraded":"2024-04-26T01:43:10.292229+0000","last_fullsized":"2024-04-26T01:43:10.292229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.675567+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:43:15.675567+0000","last_peered":"2024-04-26T01:43:15.675567+0000","last_clean":"2024-04-26T01:43:15.675567+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:43:15.675567+0000","last_undegraded":"2024-04-26T01:43:15.675567+0000","last_fullsized":"2024-04-26T01:43:15.675567+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497136+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:41:47.497136+0000","last_peered":"2024-04-26T01:41:47.497136+0000","last_clean":"2024-04-26T01:41:47.497136+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:41:47.497136+0000","last_undegraded":"2024-04-26T01:41:47.497136+0000","last_fullsized":"2024-04-26T01:41:47.497136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688744+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:43:15.688744+0000","last_peered":"2024-04-26T01:43:15.688744+0000","last_clean":"2024-04-26T01:43:15.688744+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:43:15.688744+0000","last_undegraded":"2024-04-26T01:43:15.688744+0000","last_fullsized":"2024-04-26T01:43:15.688744+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508465+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:41:47.508465+0000","last_peered":"2024-04-26T01:41:47.508465+0000","last_clean":"2024-04-26T01:41:47.508465+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:41:47.508465+0000","last_undegraded":"2024-04-26T01:41:47.508465+0000","last_fullsized":"2024-04-26T01:41:47.508465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.650431+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:43:15.650431+0000","last_peered":"2024-04-26T01:43:15.650431+0000","last_clean":"2024-04-26T01:43:15.650431+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:43:15.650431+0000","last_undegraded":"2024-04-26T01:43:15.650431+0000","last_fullsized":"2024-04-26T01:43:15.650431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509420+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:41:47.509420+0000","last_peered":"2024-04-26T01:41:47.509420+0000","last_clean":"2024-04-26T01:41:47.509420+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:41:47.509420+0000","last_undegraded":"2024-04-26T01:41:47.509420+0000","last_fullsized":"2024-04-26T01:41:47.509420+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685890+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:43:15.685890+0000","last_peered":"2024-04-26T01:43:15.685890+0000","last_clean":"2024-04-26T01:43:15.685890+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:43:15.685890+0000","last_undegraded":"2024-04-26T01:43:15.685890+0000","last_fullsized":"2024-04-26T01:43:15.685890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494699+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:41:47.494699+0000","last_peered":"2024-04-26T01:41:47.494699+0000","last_clean":"2024-04-26T01:41:47.494699+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:41:47.494699+0000","last_undegraded":"2024-04-26T01:41:47.494699+0000","last_fullsized":"2024-04-26T01:41:47.494699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690698+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:43:15.690698+0000","last_peered":"2024-04-26T01:43:15.690698+0000","last_clean":"2024-04-26T01:43:15.690698+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:43:15.690698+0000","last_undegraded":"2024-04-26T01:43:15.690698+0000","last_fullsized":"2024-04-26T01:43:15.690698+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499377+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:41:47.499377+0000","last_peered":"2024-04-26T01:41:47.499377+0000","last_clean":"2024-04-26T01:41:47.499377+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:41:47.499377+0000","last_undegraded":"2024-04-26T01:41:47.499377+0000","last_fullsized":"2024-04-26T01:41:47.499377+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687729+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:43:15.687729+0000","last_peered":"2024-04-26T01:43:15.687729+0000","last_clean":"2024-04-26T01:43:15.687729+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:43:15.687729+0000","last_undegraded":"2024-04-26T01:43:15.687729+0000","last_fullsized":"2024-04-26T01:43:15.687729+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.513560+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:41:47.513560+0000","last_peered":"2024-04-26T01:41:47.513560+0000","last_clean":"2024-04-26T01:41:47.513560+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:41:47.513560+0000","last_undegraded":"2024-04-26T01:41:47.513560+0000","last_fullsized":"2024-04-26T01:41:47.513560+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.678454+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:43:15.678454+0000","last_peered":"2024-04-26T01:43:15.678454+0000","last_clean":"2024-04-26T01:43:15.678454+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:43:15.678454+0000","last_undegraded":"2024-04-26T01:43:15.678454+0000","last_fullsized":"2024-04-26T01:43:15.678454+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508572+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:41:47.508572+0000","last_peered":"2024-04-26T01:41:47.508572+0000","last_clean":"2024-04-26T01:41:47.508572+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:41:47.508572+0000","last_undegraded":"2024-04-26T01:41:47.508572+0000","last_fullsized":"2024-04-26T01:41:47.508572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670241+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:43:15.670241+0000","last_peered":"2024-04-26T01:43:15.670241+0000","last_clean":"2024-04-26T01:43:15.670241+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:43:15.670241+0000","last_undegraded":"2024-04-26T01:43:15.670241+0000","last_fullsized":"2024-04-26T01:43:15.670241+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495277+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:41:47.495277+0000","last_peered":"2024-04-26T01:41:47.495277+0000","last_clean":"2024-04-26T01:41:47.495277+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:41:47.495277+0000","last_undegraded":"2024-04-26T01:41:47.495277+0000","last_fullsized":"2024-04-26T01:41:47.495277+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.672763+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:43:15.672763+0000","last_peered":"2024-04-26T01:43:15.672763+0000","last_clean":"2024-04-26T01:43:15.672763+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:43:15.672763+0000","last_undegraded":"2024-04-26T01:43:15.672763+0000","last_fullsized":"2024-04-26T01:43:15.672763+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.674332+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:43:15.674332+0000","last_peered":"2024-04-26T01:43:15.674332+0000","last_clean":"2024-04-26T01:43:15.674332+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:43:15.674332+0000","last_undegraded":"2024-04-26T01:43:15.674332+0000","last_fullsized":"2024-04-26T01:43:15.674332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.652494+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:43:15.652494+0000","last_peered":"2024-04-26T01:43:15.652494+0000","last_clean":"2024-04-26T01:43:15.652494+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:43:15.652494+0000","last_undegraded":"2024-04-26T01:43:15.652494+0000","last_fullsized":"2024-04-26T01:43:15.652494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.677737+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:43:15.677737+0000","last_peered":"2024-04-26T01:43:15.677737+0000","last_clean":"2024-04-26T01:43:15.677737+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:43:15.677737+0000","last_undegraded":"2024-04-26T01:43:15.677737+0000","last_fullsized":"2024-04-26T01:43:15.677737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.673990+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:43:15.673990+0000","last_peered":"2024-04-26T01:43:15.673990+0000","last_clean":"2024-04-26T01:43:15.673990+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:43:15.673990+0000","last_undegraded":"2024-04-26T01:43:15.673990+0000","last_fullsized":"2024-04-26T01:43:15.673990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688707+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:43:15.688707+0000","last_peered":"2024-04-26T01:43:15.688707+0000","last_clean":"2024-04-26T01:43:15.688707+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:43:15.688707+0000","last_undegraded":"2024-04-26T01:43:15.688707+0000","last_fullsized":"2024-04-26T01:43:15.688707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671956+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:43:15.671956+0000","last_peered":"2024-04-26T01:43:15.671956+0000","last_clean":"2024-04-26T01:43:15.671956+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:43:15.671956+0000","last_undegraded":"2024-04-26T01:43:15.671956+0000","last_fullsized":"2024-04-26T01:43:15.671956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.668316+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:43:15.668316+0000","last_peered":"2024-04-26T01:43:15.668316+0000","last_clean":"2024-04-26T01:43:15.668316+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:43:15.668316+0000","last_undegraded":"2024-04-26T01:43:15.668316+0000","last_fullsized":"2024-04-26T01:43:15.668316+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.680458+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:43:15.680458+0000","last_peered":"2024-04-26T01:43:15.680458+0000","last_clean":"2024-04-26T01:43:15.680458+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:43:15.680458+0000","last_undegraded":"2024-04-26T01:43:15.680458+0000","last_fullsized":"2024-04-26T01:43:15.680458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.682134+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:43:15.682134+0000","last_peered":"2024-04-26T01:43:15.682134+0000","last_clean":"2024-04-26T01:43:15.682134+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:43:15.682134+0000","last_undegraded":"2024-04-26T01:43:15.682134+0000","last_fullsized":"2024-04-26T01:43:15.682134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683714+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:43:15.683714+0000","last_peered":"2024-04-26T01:43:15.683714+0000","last_clean":"2024-04-26T01:43:15.683714+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:43:15.683714+0000","last_undegraded":"2024-04-26T01:43:15.683714+0000","last_fullsized":"2024-04-26T01:43:15.683714+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.692029+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:43:15.692029+0000","last_peered":"2024-04-26T01:43:15.692029+0000","last_clean":"2024-04-26T01:43:15.692029+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:43:15.692029+0000","last_undegraded":"2024-04-26T01:43:15.692029+0000","last_fullsized":"2024-04-26T01:43:15.692029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688924+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:43:15.688924+0000","last_peered":"2024-04-26T01:43:15.688924+0000","last_clean":"2024-04-26T01:43:15.688924+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:43:15.688924+0000","last_undegraded":"2024-04-26T01:43:15.688924+0000","last_fullsized":"2024-04-26T01:43:15.688924+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688602+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:43:15.688602+0000","last_peered":"2024-04-26T01:43:15.688602+0000","last_clean":"2024-04-26T01:43:15.688602+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:43:15.688602+0000","last_undegraded":"2024-04-26T01:43:15.688602+0000","last_fullsized":"2024-04-26T01:43:15.688602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69115,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":717,"num_read_kb":431,"num_write":175,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":497727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514549,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32856,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712296,"statfs":{"total":95995035648,"available":95961391104,"internally_reserved":0,"allocated":1720320,"data_stored":1154713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710778,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":497963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972415,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":497883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234052,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1089954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397577,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":497352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":562118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":498623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":562886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":497663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855447,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":498903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659216,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32780,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712372,"statfs":{"total":95995035648,"available":95961468928,"internally_reserved":0,"allocated":1642496,"data_stored":1089106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:43:25.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:43:26.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.151 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:25 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:25 smithi191 ceph-mon[32745]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:26.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:25 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:25 smithi183 ceph-mon[32752]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:26.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:25 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:26.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:25 smithi052 ceph-mon[27851]: from='client.24806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:26.353 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:26 smithi183 ceph-mon[32752]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:27.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:26 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:26 smithi052 ceph-mon[27851]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:26 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:27.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:26 smithi191 ceph-mon[32745]: pgmap v404: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T01:43:27.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:43:27.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:43:27.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:26 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:43:28.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.646+0000 7f0380200700 1 -- 172.21.15.52:0/3209409498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 msgr2=0x7f03781096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.646+0000 7f0380200700 1 --2- 172.21.15.52:0/3209409498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781096b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f036c009990 tx=0x7f036c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 -- 172.21.15.52:0/3209409498 shutdown_connections 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 --2- 172.21.15.52:0/3209409498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f037810e3b0 0x7f0378115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 --2- 172.21.15.52:0/3209409498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781096b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 --2- 172.21.15.52:0/3209409498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0378109c80 0x7f037810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 -- 172.21.15.52:0/3209409498 >> 172.21.15.52:0/3209409498 conn(0x7f0378076210 msgr2=0x7f0378107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 -- 172.21.15.52:0/3209409498 shutdown_connections 2024-04-26T01:43:28.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.647+0000 7f0380200700 1 -- 172.21.15.52:0/3209409498 wait complete. 2024-04-26T01:43:28.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 Processor -- start 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 -- start start 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 0x7f03781a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f037810e3b0 0x7f037819f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f03781192c0 con 0x7f037810e3b0 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0378119140 con 0x7f03781092d0 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.648+0000 7f0380200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0378119440 con 0x7f0378109c80 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037e200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f037810e3b0 0x7f037819f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037d800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 0x7f03781a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037d800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49868/0 (socket says 172.21.15.52:49868) 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037d800700 1 -- 172.21.15.52:0/3535111268 learned_addr learned my addr 172.21.15.52:0/3535111268 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 -- 172.21.15.52:0/3535111268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 msgr2=0x7f03781a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 -- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f037810e3b0 msgr2=0x7f037819f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f037810e3b0 0x7f037819f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 -- 172.21.15.52:0/3535111268 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037819fd20 con 0x7f0378109c80 2024-04-26T01:43:28.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.649+0000 7f037ce00700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 0x7f03781a5940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f036400e9c0 tx=0x7f036400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:28.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.650+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036400c9c0 con 0x7f0378109c80 2024-04-26T01:43:28.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.650+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03781a0010 con 0x7f0378109c80 2024-04-26T01:43:28.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.650+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f036400cb60 con 0x7f0378109c80 2024-04-26T01:43:28.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.650+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364018680 con 0x7f0378109c80 2024-04-26T01:43:28.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.650+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03781ac2d0 con 0x7f0378109c80 2024-04-26T01:43:28.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.651+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0378069730 con 0x7f0378109c80 2024-04-26T01:43:28.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.653+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0364010430 con 0x7f0378109c80 2024-04-26T01:43:28.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.655+0000 7f0376000700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 0x7f03600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:28.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.655+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f03640d0890 con 0x7f0378109c80 2024-04-26T01:43:28.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.655+0000 7f037d800700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 0x7f03600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:28.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.656+0000 7f037d800700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 0x7f03600a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f036c00b790 tx=0x7f036c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:28.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.657+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f036409d220 con 0x7f0378109c80 2024-04-26T01:43:28.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.884+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0378062e80 con 0x7f03600a3cc0 2024-04-26T01:43:28.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.893+0000 7f0376000700 1 -- 172.21.15.52:0/3535111268 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258851 (secure 0 0 0) 0x7f036c067340 con 0x7f03600a3cc0 2024-04-26T01:43:28.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:28.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.900+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 msgr2=0x7f03600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.900+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 0x7f03600a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f036c00b790 tx=0x7f036c00b260 comp rx=0 tx=0).stop 2024-04-26T01:43:28.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.900+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 msgr2=0x7f03781a5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:28.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.900+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 0x7f03781a5940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f036400e9c0 tx=0x7f036400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 shutdown_connections 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f03600a3cc0 0x7f03600a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f037810e3b0 0x7f037819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03781092d0 0x7f03781a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 --2- 172.21.15.52:0/3535111268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0378109c80 0x7f03781a5940 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 >> 172.21.15.52:0/3535111268 conn(0x7f0378076210 msgr2=0x7f03781086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 shutdown_connections 2024-04-26T01:43:28.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:28.901+0000 7f0380200700 1 -- 172.21.15.52:0/3535111268 wait complete. 2024-04-26T01:43:28.900 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:43:29.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:28 smithi052 ceph-mon[27851]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:43:29.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:28 smithi191 ceph-mon[32745]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:43:29.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:28 smithi183 ceph-mon[32752]: pgmap v405: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:43:29.696 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-04-26T01:43:27.721197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659717,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":883,"num_read_kb":565,"num_write":358,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":788,"ondisk_log_size":788,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392360,"kb_used_data":14088,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549464,"statfs":{"total":1151940427776,"available":1151538651136,"internally_reserved":0,"allocated":14426112,"data_stored":7944891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":42,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004735"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689522+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:43:15.689522+0000","last_peered":"2024-04-26T01:43:15.689522+0000","last_clean":"2024-04-26T01:43:15.689522+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:43:15.689522+0000","last_undegraded":"2024-04-26T01:43:15.689522+0000","last_fullsized":"2024-04-26T01:43:15.689522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671617+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:43:15.671617+0000","last_peered":"2024-04-26T01:43:15.671617+0000","last_clean":"2024-04-26T01:43:15.671617+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:43:15.671617+0000","last_undegraded":"2024-04-26T01:43:15.671617+0000","last_fullsized":"2024-04-26T01:43:15.671617+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684407+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:43:15.684407+0000","last_peered":"2024-04-26T01:43:15.684407+0000","last_clean":"2024-04-26T01:43:15.684407+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:43:15.684407+0000","last_undegraded":"2024-04-26T01:43:15.684407+0000","last_fullsized":"2024-04-26T01:43:15.684407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687731+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:43:15.687731+0000","last_peered":"2024-04-26T01:43:15.687731+0000","last_clean":"2024-04-26T01:43:15.687731+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:43:15.687731+0000","last_undegraded":"2024-04-26T01:43:15.687731+0000","last_fullsized":"2024-04-26T01:43:15.687731+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.669756+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:43:15.669756+0000","last_peered":"2024-04-26T01:43:15.669756+0000","last_clean":"2024-04-26T01:43:15.669756+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:43:15.669756+0000","last_undegraded":"2024-04-26T01:43:15.669756+0000","last_fullsized":"2024-04-26T01:43:15.669756+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685463+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:43:15.685463+0000","last_peered":"2024-04-26T01:43:15.685463+0000","last_clean":"2024-04-26T01:43:15.685463+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:43:15.685463+0000","last_undegraded":"2024-04-26T01:43:15.685463+0000","last_fullsized":"2024-04-26T01:43:15.685463+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.681594+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:43:15.681594+0000","last_peered":"2024-04-26T01:43:15.681594+0000","last_clean":"2024-04-26T01:43:15.681594+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:43:15.681594+0000","last_undegraded":"2024-04-26T01:43:15.681594+0000","last_fullsized":"2024-04-26T01:43:15.681594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683271+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:43:15.683271+0000","last_peered":"2024-04-26T01:43:15.683271+0000","last_clean":"2024-04-26T01:43:15.683271+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:43:15.683271+0000","last_undegraded":"2024-04-26T01:43:15.683271+0000","last_fullsized":"2024-04-26T01:43:15.683271+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.679007+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:43:15.679007+0000","last_peered":"2024-04-26T01:43:15.679007+0000","last_clean":"2024-04-26T01:43:15.679007+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:43:15.679007+0000","last_undegraded":"2024-04-26T01:43:15.679007+0000","last_fullsized":"2024-04-26T01:43:15.679007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.676498+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:43:15.676498+0000","last_peered":"2024-04-26T01:43:15.676498+0000","last_clean":"2024-04-26T01:43:15.676498+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:43:15.676498+0000","last_undegraded":"2024-04-26T01:43:15.676498+0000","last_fullsized":"2024-04-26T01:43:15.676498+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.666703+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:43:15.666703+0000","last_peered":"2024-04-26T01:43:15.666703+0000","last_clean":"2024-04-26T01:43:15.666703+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:43:15.666703+0000","last_undegraded":"2024-04-26T01:43:15.666703+0000","last_fullsized":"2024-04-26T01:43:15.666703+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.663780+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:43:15.663780+0000","last_peered":"2024-04-26T01:43:15.663780+0000","last_clean":"2024-04-26T01:43:15.663780+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:43:15.663780+0000","last_undegraded":"2024-04-26T01:43:15.663780+0000","last_fullsized":"2024-04-26T01:43:15.663780+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689087+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:43:15.689087+0000","last_peered":"2024-04-26T01:43:15.689087+0000","last_clean":"2024-04-26T01:43:15.689087+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:43:15.689087+0000","last_undegraded":"2024-04-26T01:43:15.689087+0000","last_fullsized":"2024-04-26T01:43:15.689087+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.675543+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:43:15.675543+0000","last_peered":"2024-04-26T01:43:15.675543+0000","last_clean":"2024-04-26T01:43:15.675543+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:43:15.675543+0000","last_undegraded":"2024-04-26T01:43:15.675543+0000","last_fullsized":"2024-04-26T01:43:15.675543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683650+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:43:15.683650+0000","last_peered":"2024-04-26T01:43:15.683650+0000","last_clean":"2024-04-26T01:43:15.683650+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:43:15.683650+0000","last_undegraded":"2024-04-26T01:43:15.683650+0000","last_fullsized":"2024-04-26T01:43:15.683650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688308+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:43:15.688308+0000","last_peered":"2024-04-26T01:43:15.688308+0000","last_clean":"2024-04-26T01:43:15.688308+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:43:15.688308+0000","last_undegraded":"2024-04-26T01:43:15.688308+0000","last_fullsized":"2024-04-26T01:43:15.688308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689860+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:43:15.689860+0000","last_peered":"2024-04-26T01:43:15.689860+0000","last_clean":"2024-04-26T01:43:15.689860+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:43:15.689860+0000","last_undegraded":"2024-04-26T01:43:15.689860+0000","last_fullsized":"2024-04-26T01:43:15.689860+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.686170+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:43:15.686170+0000","last_peered":"2024-04-26T01:43:15.686170+0000","last_clean":"2024-04-26T01:43:15.686170+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:43:15.686170+0000","last_undegraded":"2024-04-26T01:43:15.686170+0000","last_fullsized":"2024-04-26T01:43:15.686170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671856+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:43:15.671856+0000","last_peered":"2024-04-26T01:43:15.671856+0000","last_clean":"2024-04-26T01:43:15.671856+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:43:15.671856+0000","last_undegraded":"2024-04-26T01:43:15.671856+0000","last_fullsized":"2024-04-26T01:43:15.671856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.498749+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:41:47.498749+0000","last_peered":"2024-04-26T01:41:47.498749+0000","last_clean":"2024-04-26T01:41:47.498749+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:41:47.498749+0000","last_undegraded":"2024-04-26T01:41:47.498749+0000","last_fullsized":"2024-04-26T01:41:47.498749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684057+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:43:15.684057+0000","last_peered":"2024-04-26T01:43:15.684057+0000","last_clean":"2024-04-26T01:43:15.684057+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:43:15.684057+0000","last_undegraded":"2024-04-26T01:43:15.684057+0000","last_fullsized":"2024-04-26T01:43:15.684057+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.510500+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:41:47.510500+0000","last_peered":"2024-04-26T01:41:47.510500+0000","last_clean":"2024-04-26T01:41:47.510500+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:41:47.510500+0000","last_undegraded":"2024-04-26T01:41:47.510500+0000","last_fullsized":"2024-04-26T01:41:47.510500+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.691590+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:43:15.691590+0000","last_peered":"2024-04-26T01:43:15.691590+0000","last_clean":"2024-04-26T01:43:15.691590+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:43:15.691590+0000","last_undegraded":"2024-04-26T01:43:15.691590+0000","last_fullsized":"2024-04-26T01:43:15.691590+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.506427+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:41:47.506427+0000","last_peered":"2024-04-26T01:41:47.506427+0000","last_clean":"2024-04-26T01:41:47.506427+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:41:47.506427+0000","last_undegraded":"2024-04-26T01:41:47.506427+0000","last_fullsized":"2024-04-26T01:41:47.506427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687010+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:43:15.687010+0000","last_peered":"2024-04-26T01:43:15.687010+0000","last_clean":"2024-04-26T01:43:15.687010+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:43:15.687010+0000","last_undegraded":"2024-04-26T01:43:15.687010+0000","last_fullsized":"2024-04-26T01:43:15.687010+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.500059+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:41:47.500059+0000","last_peered":"2024-04-26T01:41:47.500059+0000","last_clean":"2024-04-26T01:41:47.500059+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:41:47.500059+0000","last_undegraded":"2024-04-26T01:41:47.500059+0000","last_fullsized":"2024-04-26T01:41:47.500059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.649115+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:43:15.649115+0000","last_peered":"2024-04-26T01:43:15.649115+0000","last_clean":"2024-04-26T01:43:15.649115+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:43:15.649115+0000","last_undegraded":"2024-04-26T01:43:15.649115+0000","last_fullsized":"2024-04-26T01:43:15.649115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:24.571098+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:43:24.571098+0000","last_peered":"2024-04-26T01:43:24.571098+0000","last_clean":"2024-04-26T01:43:24.571098+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:43:24.571098+0000","last_undegraded":"2024-04-26T01:43:24.571098+0000","last_fullsized":"2024-04-26T01:43:24.571098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688332+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:43:15.688332+0000","last_peered":"2024-04-26T01:43:15.688332+0000","last_clean":"2024-04-26T01:43:15.688332+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:43:15.688332+0000","last_undegraded":"2024-04-26T01:43:15.688332+0000","last_fullsized":"2024-04-26T01:43:15.688332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508594+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:41:47.508594+0000","last_peered":"2024-04-26T01:41:47.508594+0000","last_clean":"2024-04-26T01:41:47.508594+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:41:47.508594+0000","last_undegraded":"2024-04-26T01:41:47.508594+0000","last_fullsized":"2024-04-26T01:41:47.508594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671152+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:43:15.671152+0000","last_peered":"2024-04-26T01:43:15.671152+0000","last_clean":"2024-04-26T01:43:15.671152+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:43:15.671152+0000","last_undegraded":"2024-04-26T01:43:15.671152+0000","last_fullsized":"2024-04-26T01:43:15.671152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494699+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:41:47.494699+0000","last_peered":"2024-04-26T01:41:47.494699+0000","last_clean":"2024-04-26T01:41:47.494699+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:41:47.494699+0000","last_undegraded":"2024-04-26T01:41:47.494699+0000","last_fullsized":"2024-04-26T01:41:47.494699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690931+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:43:15.690931+0000","last_peered":"2024-04-26T01:43:15.690931+0000","last_clean":"2024-04-26T01:43:15.690931+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:43:15.690931+0000","last_undegraded":"2024-04-26T01:43:15.690931+0000","last_fullsized":"2024-04-26T01:43:15.690931+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508174+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:41:47.508174+0000","last_peered":"2024-04-26T01:41:47.508174+0000","last_clean":"2024-04-26T01:41:47.508174+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:41:47.508174+0000","last_undegraded":"2024-04-26T01:41:47.508174+0000","last_fullsized":"2024-04-26T01:41:47.508174+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685894+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:43:15.685894+0000","last_peered":"2024-04-26T01:43:15.685894+0000","last_clean":"2024-04-26T01:43:15.685894+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:43:15.685894+0000","last_undegraded":"2024-04-26T01:43:15.685894+0000","last_fullsized":"2024-04-26T01:43:15.685894+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:27.574922+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:43:27.574922+0000","last_peered":"2024-04-26T01:43:27.574922+0000","last_clean":"2024-04-26T01:43:27.574922+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:43:27.574922+0000","last_undegraded":"2024-04-26T01:43:27.574922+0000","last_fullsized":"2024-04-26T01:43:27.574922+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688947+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:43:15.688947+0000","last_peered":"2024-04-26T01:43:15.688947+0000","last_clean":"2024-04-26T01:43:15.688947+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:43:15.688947+0000","last_undegraded":"2024-04-26T01:43:15.688947+0000","last_fullsized":"2024-04-26T01:43:15.688947+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499680+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:41:47.499680+0000","last_peered":"2024-04-26T01:41:47.499680+0000","last_clean":"2024-04-26T01:41:47.499680+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:41:47.499680+0000","last_undegraded":"2024-04-26T01:41:47.499680+0000","last_fullsized":"2024-04-26T01:41:47.499680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687839+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:43:15.687839+0000","last_peered":"2024-04-26T01:43:15.687839+0000","last_clean":"2024-04-26T01:43:15.687839+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:43:15.687839+0000","last_undegraded":"2024-04-26T01:43:15.687839+0000","last_fullsized":"2024-04-26T01:43:15.687839+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508795+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:41:47.508795+0000","last_peered":"2024-04-26T01:41:47.508795+0000","last_clean":"2024-04-26T01:41:47.508795+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:41:47.508795+0000","last_undegraded":"2024-04-26T01:41:47.508795+0000","last_fullsized":"2024-04-26T01:41:47.508795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.679923+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:43:15.679923+0000","last_peered":"2024-04-26T01:43:15.679923+0000","last_clean":"2024-04-26T01:43:15.679923+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:43:15.679923+0000","last_undegraded":"2024-04-26T01:43:15.679923+0000","last_fullsized":"2024-04-26T01:43:15.679923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'35","reported_seq":314,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:27.574879+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:43:27.574879+0000","last_peered":"2024-04-26T01:43:27.574879+0000","last_clean":"2024-04-26T01:43:27.574879+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:43:27.574879+0000","last_undegraded":"2024-04-26T01:43:27.574879+0000","last_fullsized":"2024-04-26T01:43:27.574879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64973,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":252,"num_read_kb":189,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.651382+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:43:15.651382+0000","last_peered":"2024-04-26T01:43:15.651382+0000","last_clean":"2024-04-26T01:43:15.651382+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:43:15.651382+0000","last_undegraded":"2024-04-26T01:43:15.651382+0000","last_fullsized":"2024-04-26T01:43:15.651382+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494906+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:41:47.494906+0000","last_peered":"2024-04-26T01:41:47.494906+0000","last_clean":"2024-04-26T01:41:47.494906+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:41:47.494906+0000","last_undegraded":"2024-04-26T01:41:47.494906+0000","last_fullsized":"2024-04-26T01:41:47.494906+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690459+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:43:15.690459+0000","last_peered":"2024-04-26T01:43:15.690459+0000","last_clean":"2024-04-26T01:43:15.690459+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:43:15.690459+0000","last_undegraded":"2024-04-26T01:43:15.690459+0000","last_fullsized":"2024-04-26T01:43:15.690459+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508390+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:41:47.508390+0000","last_peered":"2024-04-26T01:41:47.508390+0000","last_clean":"2024-04-26T01:41:47.508390+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:41:47.508390+0000","last_undegraded":"2024-04-26T01:41:47.508390+0000","last_fullsized":"2024-04-26T01:41:47.508390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687656+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:43:15.687656+0000","last_peered":"2024-04-26T01:43:15.687656+0000","last_clean":"2024-04-26T01:43:15.687656+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:43:15.687656+0000","last_undegraded":"2024-04-26T01:43:15.687656+0000","last_fullsized":"2024-04-26T01:43:15.687656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'5","reported_seq":155,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:24.712780+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:43:24.712780+0000","last_peered":"2024-04-26T01:43:24.712780+0000","last_clean":"2024-04-26T01:43:24.712780+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:43:24.712780+0000","last_undegraded":"2024-04-26T01:43:24.712780+0000","last_fullsized":"2024-04-26T01:43:24.712780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670756+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:43:15.670756+0000","last_peered":"2024-04-26T01:43:15.670756+0000","last_clean":"2024-04-26T01:43:15.670756+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:43:15.670756+0000","last_undegraded":"2024-04-26T01:43:15.670756+0000","last_fullsized":"2024-04-26T01:43:15.670756+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499052+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:41:47.499052+0000","last_peered":"2024-04-26T01:41:47.499052+0000","last_clean":"2024-04-26T01:41:47.499052+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:41:47.499052+0000","last_undegraded":"2024-04-26T01:41:47.499052+0000","last_fullsized":"2024-04-26T01:41:47.499052+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.684019+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:43:15.684019+0000","last_peered":"2024-04-26T01:43:15.684019+0000","last_clean":"2024-04-26T01:43:15.684019+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:43:15.684019+0000","last_undegraded":"2024-04-26T01:43:15.684019+0000","last_fullsized":"2024-04-26T01:43:15.684019+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499658+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:41:47.499658+0000","last_peered":"2024-04-26T01:41:47.499658+0000","last_clean":"2024-04-26T01:41:47.499658+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:41:47.499658+0000","last_undegraded":"2024-04-26T01:41:47.499658+0000","last_fullsized":"2024-04-26T01:41:47.499658+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670784+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:43:15.670784+0000","last_peered":"2024-04-26T01:43:15.670784+0000","last_clean":"2024-04-26T01:43:15.670784+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:43:15.670784+0000","last_undegraded":"2024-04-26T01:43:15.670784+0000","last_fullsized":"2024-04-26T01:43:15.670784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509508+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:41:47.509508+0000","last_peered":"2024-04-26T01:41:47.509508+0000","last_clean":"2024-04-26T01:41:47.509508+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:41:47.509508+0000","last_undegraded":"2024-04-26T01:41:47.509508+0000","last_fullsized":"2024-04-26T01:41:47.509508+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687462+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:43:15.687462+0000","last_peered":"2024-04-26T01:43:15.687462+0000","last_clean":"2024-04-26T01:43:15.687462+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:43:15.687462+0000","last_undegraded":"2024-04-26T01:43:15.687462+0000","last_fullsized":"2024-04-26T01:43:15.687462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499492+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:41:47.499492+0000","last_peered":"2024-04-26T01:41:47.499492+0000","last_clean":"2024-04-26T01:41:47.499492+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:41:47.499492+0000","last_undegraded":"2024-04-26T01:41:47.499492+0000","last_fullsized":"2024-04-26T01:41:47.499492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687115+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:43:15.687115+0000","last_peered":"2024-04-26T01:43:15.687115+0000","last_clean":"2024-04-26T01:43:15.687115+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:43:15.687115+0000","last_undegraded":"2024-04-26T01:43:15.687115+0000","last_fullsized":"2024-04-26T01:43:15.687115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.650937+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:43:15.650937+0000","last_peered":"2024-04-26T01:43:15.650937+0000","last_clean":"2024-04-26T01:43:15.650937+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:43:15.650937+0000","last_undegraded":"2024-04-26T01:43:15.650937+0000","last_fullsized":"2024-04-26T01:43:15.650937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":149,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:22.707866+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:43:22.707866+0000","last_peered":"2024-04-26T01:43:22.707866+0000","last_clean":"2024-04-26T01:43:22.707866+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:43:22.707866+0000","last_undegraded":"2024-04-26T01:43:22.707866+0000","last_fullsized":"2024-04-26T01:43:22.707866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690660+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:43:15.690660+0000","last_peered":"2024-04-26T01:43:15.690660+0000","last_clean":"2024-04-26T01:43:15.690660+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:43:15.690660+0000","last_undegraded":"2024-04-26T01:43:15.690660+0000","last_fullsized":"2024-04-26T01:43:15.690660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497492+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:41:47.497492+0000","last_peered":"2024-04-26T01:41:47.497492+0000","last_clean":"2024-04-26T01:41:47.497492+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:41:47.497492+0000","last_undegraded":"2024-04-26T01:41:47.497492+0000","last_fullsized":"2024-04-26T01:41:47.497492+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499373+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:41:47.499373+0000","last_peered":"2024-04-26T01:41:47.499373+0000","last_clean":"2024-04-26T01:41:47.499373+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:41:47.499373+0000","last_undegraded":"2024-04-26T01:41:47.499373+0000","last_fullsized":"2024-04-26T01:41:47.499373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.664392+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:43:15.664392+0000","last_peered":"2024-04-26T01:43:15.664392+0000","last_clean":"2024-04-26T01:43:15.664392+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:43:15.664392+0000","last_undegraded":"2024-04-26T01:43:15.664392+0000","last_fullsized":"2024-04-26T01:43:15.664392+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499589+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:41:47.499589+0000","last_peered":"2024-04-26T01:41:47.499589+0000","last_clean":"2024-04-26T01:41:47.499589+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:41:47.499589+0000","last_undegraded":"2024-04-26T01:41:47.499589+0000","last_fullsized":"2024-04-26T01:41:47.499589+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499443+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:41:47.499443+0000","last_peered":"2024-04-26T01:41:47.499443+0000","last_clean":"2024-04-26T01:41:47.499443+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:41:47.499443+0000","last_undegraded":"2024-04-26T01:41:47.499443+0000","last_fullsized":"2024-04-26T01:41:47.499443+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.689438+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:43:15.689438+0000","last_peered":"2024-04-26T01:43:15.689438+0000","last_clean":"2024-04-26T01:43:15.689438+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:43:15.689438+0000","last_undegraded":"2024-04-26T01:43:15.689438+0000","last_fullsized":"2024-04-26T01:43:15.689438+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'13","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:10.292229+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:43:10.292229+0000","last_peered":"2024-04-26T01:43:10.292229+0000","last_clean":"2024-04-26T01:43:10.292229+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:43:10.292229+0000","last_undegraded":"2024-04-26T01:43:10.292229+0000","last_fullsized":"2024-04-26T01:43:10.292229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.675567+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:43:15.675567+0000","last_peered":"2024-04-26T01:43:15.675567+0000","last_clean":"2024-04-26T01:43:15.675567+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:43:15.675567+0000","last_undegraded":"2024-04-26T01:43:15.675567+0000","last_fullsized":"2024-04-26T01:43:15.675567+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.497136+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:41:47.497136+0000","last_peered":"2024-04-26T01:41:47.497136+0000","last_clean":"2024-04-26T01:41:47.497136+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:41:47.497136+0000","last_undegraded":"2024-04-26T01:41:47.497136+0000","last_fullsized":"2024-04-26T01:41:47.497136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688744+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:43:15.688744+0000","last_peered":"2024-04-26T01:43:15.688744+0000","last_clean":"2024-04-26T01:43:15.688744+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:43:15.688744+0000","last_undegraded":"2024-04-26T01:43:15.688744+0000","last_fullsized":"2024-04-26T01:43:15.688744+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508465+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:41:47.508465+0000","last_peered":"2024-04-26T01:41:47.508465+0000","last_clean":"2024-04-26T01:41:47.508465+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:41:47.508465+0000","last_undegraded":"2024-04-26T01:41:47.508465+0000","last_fullsized":"2024-04-26T01:41:47.508465+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.650431+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:43:15.650431+0000","last_peered":"2024-04-26T01:43:15.650431+0000","last_clean":"2024-04-26T01:43:15.650431+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:43:15.650431+0000","last_undegraded":"2024-04-26T01:43:15.650431+0000","last_fullsized":"2024-04-26T01:43:15.650431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.509420+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:41:47.509420+0000","last_peered":"2024-04-26T01:41:47.509420+0000","last_clean":"2024-04-26T01:41:47.509420+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:41:47.509420+0000","last_undegraded":"2024-04-26T01:41:47.509420+0000","last_fullsized":"2024-04-26T01:41:47.509420+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.685890+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:43:15.685890+0000","last_peered":"2024-04-26T01:43:15.685890+0000","last_clean":"2024-04-26T01:43:15.685890+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:43:15.685890+0000","last_undegraded":"2024-04-26T01:43:15.685890+0000","last_fullsized":"2024-04-26T01:43:15.685890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.494699+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:41:47.494699+0000","last_peered":"2024-04-26T01:41:47.494699+0000","last_clean":"2024-04-26T01:41:47.494699+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:41:47.494699+0000","last_undegraded":"2024-04-26T01:41:47.494699+0000","last_fullsized":"2024-04-26T01:41:47.494699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.690698+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:43:15.690698+0000","last_peered":"2024-04-26T01:43:15.690698+0000","last_clean":"2024-04-26T01:43:15.690698+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:43:15.690698+0000","last_undegraded":"2024-04-26T01:43:15.690698+0000","last_fullsized":"2024-04-26T01:43:15.690698+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.499377+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:41:47.499377+0000","last_peered":"2024-04-26T01:41:47.499377+0000","last_clean":"2024-04-26T01:41:47.499377+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:41:47.499377+0000","last_undegraded":"2024-04-26T01:41:47.499377+0000","last_fullsized":"2024-04-26T01:41:47.499377+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.687729+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:43:15.687729+0000","last_peered":"2024-04-26T01:43:15.687729+0000","last_clean":"2024-04-26T01:43:15.687729+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:43:15.687729+0000","last_undegraded":"2024-04-26T01:43:15.687729+0000","last_fullsized":"2024-04-26T01:43:15.687729+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.513560+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:41:47.513560+0000","last_peered":"2024-04-26T01:41:47.513560+0000","last_clean":"2024-04-26T01:41:47.513560+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:41:47.513560+0000","last_undegraded":"2024-04-26T01:41:47.513560+0000","last_fullsized":"2024-04-26T01:41:47.513560+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.678454+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:43:15.678454+0000","last_peered":"2024-04-26T01:43:15.678454+0000","last_clean":"2024-04-26T01:43:15.678454+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:43:15.678454+0000","last_undegraded":"2024-04-26T01:43:15.678454+0000","last_fullsized":"2024-04-26T01:43:15.678454+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.508572+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:41:47.508572+0000","last_peered":"2024-04-26T01:41:47.508572+0000","last_clean":"2024-04-26T01:41:47.508572+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:41:47.508572+0000","last_undegraded":"2024-04-26T01:41:47.508572+0000","last_fullsized":"2024-04-26T01:41:47.508572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.670241+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:43:15.670241+0000","last_peered":"2024-04-26T01:43:15.670241+0000","last_clean":"2024-04-26T01:43:15.670241+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:43:15.670241+0000","last_undegraded":"2024-04-26T01:43:15.670241+0000","last_fullsized":"2024-04-26T01:43:15.670241+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:41:47.495277+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:41:47.495277+0000","last_peered":"2024-04-26T01:41:47.495277+0000","last_clean":"2024-04-26T01:41:47.495277+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:41:47.495277+0000","last_undegraded":"2024-04-26T01:41:47.495277+0000","last_fullsized":"2024-04-26T01:41:47.495277+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.672763+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:43:15.672763+0000","last_peered":"2024-04-26T01:43:15.672763+0000","last_clean":"2024-04-26T01:43:15.672763+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:43:15.672763+0000","last_undegraded":"2024-04-26T01:43:15.672763+0000","last_fullsized":"2024-04-26T01:43:15.672763+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.674332+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:43:15.674332+0000","last_peered":"2024-04-26T01:43:15.674332+0000","last_clean":"2024-04-26T01:43:15.674332+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:43:15.674332+0000","last_undegraded":"2024-04-26T01:43:15.674332+0000","last_fullsized":"2024-04-26T01:43:15.674332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.652494+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:43:15.652494+0000","last_peered":"2024-04-26T01:43:15.652494+0000","last_clean":"2024-04-26T01:43:15.652494+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:43:15.652494+0000","last_undegraded":"2024-04-26T01:43:15.652494+0000","last_fullsized":"2024-04-26T01:43:15.652494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.677737+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:43:15.677737+0000","last_peered":"2024-04-26T01:43:15.677737+0000","last_clean":"2024-04-26T01:43:15.677737+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:43:15.677737+0000","last_undegraded":"2024-04-26T01:43:15.677737+0000","last_fullsized":"2024-04-26T01:43:15.677737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.673990+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:43:15.673990+0000","last_peered":"2024-04-26T01:43:15.673990+0000","last_clean":"2024-04-26T01:43:15.673990+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:43:15.673990+0000","last_undegraded":"2024-04-26T01:43:15.673990+0000","last_fullsized":"2024-04-26T01:43:15.673990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688707+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:43:15.688707+0000","last_peered":"2024-04-26T01:43:15.688707+0000","last_clean":"2024-04-26T01:43:15.688707+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:43:15.688707+0000","last_undegraded":"2024-04-26T01:43:15.688707+0000","last_fullsized":"2024-04-26T01:43:15.688707+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.671956+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:43:15.671956+0000","last_peered":"2024-04-26T01:43:15.671956+0000","last_clean":"2024-04-26T01:43:15.671956+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:43:15.671956+0000","last_undegraded":"2024-04-26T01:43:15.671956+0000","last_fullsized":"2024-04-26T01:43:15.671956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.668316+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:43:15.668316+0000","last_peered":"2024-04-26T01:43:15.668316+0000","last_clean":"2024-04-26T01:43:15.668316+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:43:15.668316+0000","last_undegraded":"2024-04-26T01:43:15.668316+0000","last_fullsized":"2024-04-26T01:43:15.668316+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.680458+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:43:15.680458+0000","last_peered":"2024-04-26T01:43:15.680458+0000","last_clean":"2024-04-26T01:43:15.680458+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:43:15.680458+0000","last_undegraded":"2024-04-26T01:43:15.680458+0000","last_fullsized":"2024-04-26T01:43:15.680458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.682134+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:43:15.682134+0000","last_peered":"2024-04-26T01:43:15.682134+0000","last_clean":"2024-04-26T01:43:15.682134+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:43:15.682134+0000","last_undegraded":"2024-04-26T01:43:15.682134+0000","last_fullsized":"2024-04-26T01:43:15.682134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.683714+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:43:15.683714+0000","last_peered":"2024-04-26T01:43:15.683714+0000","last_clean":"2024-04-26T01:43:15.683714+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:43:15.683714+0000","last_undegraded":"2024-04-26T01:43:15.683714+0000","last_fullsized":"2024-04-26T01:43:15.683714+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.692029+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:43:15.692029+0000","last_peered":"2024-04-26T01:43:15.692029+0000","last_clean":"2024-04-26T01:43:15.692029+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:43:15.692029+0000","last_undegraded":"2024-04-26T01:43:15.692029+0000","last_fullsized":"2024-04-26T01:43:15.692029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688924+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:43:15.688924+0000","last_peered":"2024-04-26T01:43:15.688924+0000","last_clean":"2024-04-26T01:43:15.688924+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:43:15.688924+0000","last_undegraded":"2024-04-26T01:43:15.688924+0000","last_fullsized":"2024-04-26T01:43:15.688924+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T01:43:15.688602+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:43:15.688602+0000","last_peered":"2024-04-26T01:43:15.688602+0000","last_clean":"2024-04-26T01:43:15.688602+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:43:15.688602+0000","last_undegraded":"2024-04-26T01:43:15.688602+0000","last_fullsized":"2024-04-26T01:43:15.688602+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69115,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":747,"num_read_kb":446,"num_write":175,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":497727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514550,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32856,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712296,"statfs":{"total":95995035648,"available":95961391104,"internally_reserved":0,"allocated":1720320,"data_stored":1154713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710779,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":497963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972416,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":497883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234052,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1089954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397578,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":497352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":562118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":498623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":562886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":497663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855448,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":498903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659217,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32780,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712372,"statfs":{"total":95995035648,"available":95961468928,"internally_reserved":0,"allocated":1642496,"data_stored":1089106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:43:29.700 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:43:29.700 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T01:43:29.700 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T01:43:29.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T01:43:30.199 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:30.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:29 smithi052 ceph-mon[27851]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:30.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:29 smithi191 ceph-mon[32745]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:30.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:29 smithi183 ceph-mon[32752]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:43:31.303 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:30 smithi191 ceph-mon[32745]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:43:31.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:30 smithi052 ceph-mon[27851]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:43:31.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:30 smithi183 ceph-mon[32752]: pgmap v406: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:43:32.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 -- 172.21.15.52:0/1156144510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109310 msgr2=0x7fd3041096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:32.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 --2- 172.21.15.52:0/1156144510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109310 0x7fd3041096f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40099c0 tx=0x7fd2f4009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 -- 172.21.15.52:0/1156144510 shutdown_connections 2024-04-26T01:43:32.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 --2- 172.21.15.52:0/1156144510 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30410e3a0 0x7fd304115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 --2- 172.21.15.52:0/1156144510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109310 0x7fd3041096f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 --2- 172.21.15.52:0/1156144510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd304109cc0 0x7fd30410dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 -- 172.21.15.52:0/1156144510 >> 172.21.15.52:0/1156144510 conn(0x7fd304076210 msgr2=0x7fd304078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 -- 172.21.15.52:0/1156144510 shutdown_connections 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.281+0000 7fd30b400700 1 -- 172.21.15.52:0/1156144510 wait complete. 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.282+0000 7fd30b400700 1 Processor -- start 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 -- start start 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:32.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 0x7fd3041a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:32.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:32.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd304119250 con 0x7fd30419f830 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041190d0 con 0x7fd30410e3a0 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.283+0000 7fd30b400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041193d0 con 0x7fd304109cc0 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd309400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd308a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd309400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57120/0 (socket says 172.21.15.52:57120) 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd308a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:49726/0 (socket says 172.21.15.52:49726) 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd309400700 1 -- 172.21.15.52:0/3159717613 learned_addr learned my addr 172.21.15.52:0/3159717613 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 0x7fd3041a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 -- 172.21.15.52:0/3159717613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 msgr2=0x7fd3041a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 -- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 msgr2=0x7fd30419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:32.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd2fbe00700 1 -- 172.21.15.52:0/3159717613 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3041a03f0 con 0x7fd30410e3a0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd308a00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.284+0000 7fd309400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd2fbe00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 0x7fd3041a5ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00e9c0 tx=0x7fd2ec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec004220 con 0x7fd30410e3a0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ec0043c0 con 0x7fd30410e3a0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3041ac4c0 con 0x7fd30410e3a0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec005020 con 0x7fd30410e3a0 2024-04-26T01:43:32.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.285+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041ac860 con 0x7fd30410e3a0 2024-04-26T01:43:32.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.286+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30410a7a0 con 0x7fd30410e3a0 2024-04-26T01:43:32.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.288+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd2ec004750 con 0x7fd30410e3a0 2024-04-26T01:43:32.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.290+0000 7fd302000700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 0x7fd2e80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:32.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.290+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fd2ec0d0300 con 0x7fd30410e3a0 2024-04-26T01:43:32.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.290+0000 7fd308a00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 0x7fd2e80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:32.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.291+0000 7fd308a00700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 0x7fd2e80a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b580 tx=0x7fd2f4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:32.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.294+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2ec09cc90 con 0x7fd30410e3a0 2024-04-26T01:43:32.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.569+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd3041a0d10 con 0x7fd30410e3a0 2024-04-26T01:43:32.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.570+0000 7fd302000700 1 -- 172.21.15.52:0/3159717613 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd2ec09c590 con 0x7fd30410e3a0 2024-04-26T01:43:32.567 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:32.568 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 msgr2=0x7fd2e80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 0x7fd2e80a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b580 tx=0x7fd2f4005fb0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 msgr2=0x7fd3041a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 0x7fd3041a5ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00e9c0 tx=0x7fd2ec00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 shutdown_connections 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fd2e80a3cc0 0x7fd2e80a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd30419f830 0x7fd30419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd304109cc0 0x7fd3041a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 --2- 172.21.15.52:0/3159717613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd30410e3a0 0x7fd3041a5ba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.573+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 >> 172.21.15.52:0/3159717613 conn(0x7fd304076210 msgr2=0x7fd30410d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.574+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 shutdown_connections 2024-04-26T01:43:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:32.574+0000 7fd30b400700 1 -- 172.21.15.52:0/3159717613 wait complete. 2024-04-26T01:43:33.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:32 smithi052 ceph-mon[27851]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:43:33.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:32 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3159717613' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:43:33.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:32 smithi191 ceph-mon[32745]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:43:33.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:32 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3159717613' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:43:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:32 smithi183 ceph-mon[32752]: pgmap v407: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T01:43:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:32 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3159717613' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T01:43:33.479 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T01:43:33.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:43:33.959 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:35.393 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:34 smithi052 ceph-mon[27851]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:35.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:34 smithi191 ceph-mon[32745]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:35.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:34 smithi183 ceph-mon[32752]: pgmap v408: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 -- 172.21.15.52:0/3540335147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018109fa0 msgr2=0x7f9018111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 --2- 172.21.15.52:0/3540335147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018109fa0 0x7f9018111b70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f901000b0d0 tx=0x7f901000b3e0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 -- 172.21.15.52:0/3540335147 shutdown_connections 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 --2- 172.21.15.52:0/3540335147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018109fa0 0x7f9018111b70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 --2- 172.21.15.52:0/3540335147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f90181052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 --2- 172.21.15.52:0/3540335147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018105870 0x7f90181098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 -- 172.21.15.52:0/3540335147 >> 172.21.15.52:0/3540335147 conn(0x7f9018100750 msgr2=0x7f9018102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 -- 172.21.15.52:0/3540335147 shutdown_connections 2024-04-26T01:43:36.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.180+0000 7f901e000700 1 -- 172.21.15.52:0/3540335147 wait complete. 2024-04-26T01:43:36.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.181+0000 7f901e000700 1 Processor -- start 2024-04-26T01:43:36.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.181+0000 7f901e000700 1 -- start start 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 0x7f901819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018109fa0 0x7f901819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9018114e30 con 0x7f9018105870 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9018114cb0 con 0x7f9018109fa0 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f901e000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9018114fb0 con 0x7f9018104ec0 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f9017400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f900ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 0x7f901819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f9017400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52992/0 (socket says 172.21.15.52:52992) 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f9017400700 1 -- 172.21.15.52:0/2444046995 learned_addr learned my addr 172.21.15.52:0/2444046995 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:36.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.182+0000 7f9017e00700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018109fa0 0x7f901819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 -- 172.21.15.52:0/2444046995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 msgr2=0x7f901819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 -- 172.21.15.52:0/2444046995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018109fa0 msgr2=0x7f901819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018109fa0 0x7f901819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 -- 172.21.15.52:0/2444046995 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90181a0280 con 0x7f9018105870 2024-04-26T01:43:36.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f9017400700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f900ea00700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 0x7f901819b800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f900000d720 tx=0x7f900000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000014030 con 0x7f9018105870 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90181a0570 con 0x7f9018105870 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90000044d0 con 0x7f9018105870 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000005400 con 0x7f9018105870 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.183+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90181a7df0 con 0x7f9018105870 2024-04-26T01:43:36.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.185+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9018069730 con 0x7f9018105870 2024-04-26T01:43:36.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.187+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f90000055a0 con 0x7f9018105870 2024-04-26T01:43:36.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.188+0000 7f9014c00700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 0x7f8ffc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:36.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.188+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f90000cfed0 con 0x7f9018105870 2024-04-26T01:43:36.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.188+0000 7f9017400700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 0x7f8ffc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:36.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.189+0000 7f9017400700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 0x7f8ffc0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9008011550 tx=0x7f9008009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:36.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.193+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f900009c860 con 0x7f9018105870 2024-04-26T01:43:36.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.474+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f901802df80 con 0x7f9018105870 2024-04-26T01:43:36.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.476+0000 7f9014c00700 1 -- 172.21.15.52:0/2444046995 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f900009c160 con 0x7f9018105870 2024-04-26T01:43:36.474 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:36.474 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 msgr2=0x7f8ffc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 0x7f8ffc0a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9008011550 tx=0x7f9008009260 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 msgr2=0x7f901819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 0x7f901819b800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f900000d720 tx=0x7f900000da30 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 shutdown_connections 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.479+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8ffc0a3cc0 0x7f8ffc0a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9018109fa0 0x7f901819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9018104ec0 0x7f901819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 --2- 172.21.15.52:0/2444046995 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9018105870 0x7f901819b800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:36.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 >> 172.21.15.52:0/2444046995 conn(0x7f9018100750 msgr2=0x7f90181012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:36.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 shutdown_connections 2024-04-26T01:43:36.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:36.480+0000 7f901e000700 1 -- 172.21.15.52:0/2444046995 wait complete. 2024-04-26T01:43:36.479 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:43:37.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:36 smithi052 ceph-mon[27851]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:37.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:36 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2444046995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:37.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:43:37.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:36 smithi191 ceph-mon[32745]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:37.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:36 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2444046995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:37.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:36 smithi183 ceph-mon[32752]: pgmap v409: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:37.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:36 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2444046995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:37.683 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:39.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:38 smithi191 ceph-mon[32745]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:39.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:38 smithi183 ceph-mon[32752]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:39.451 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:38 smithi052 ceph-mon[27851]: pgmap v410: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 -- 172.21.15.52:0/2436286875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58104ec0 msgr2=0x7ffb581052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 --2- 172.21.15.52:0/2436286875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58104ec0 0x7ffb581052a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffb480099c0 tx=0x7ffb48009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 -- 172.21.15.52:0/2436286875 shutdown_connections 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 --2- 172.21.15.52:0/2436286875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 0x7ffb58111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 --2- 172.21.15.52:0/2436286875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58104ec0 0x7ffb581052a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 --2- 172.21.15.52:0/2436286875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58105870 0x7ffb581098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 -- 172.21.15.52:0/2436286875 >> 172.21.15.52:0/2436286875 conn(0x7ffb58100750 msgr2=0x7ffb58102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 -- 172.21.15.52:0/2436286875 shutdown_connections 2024-04-26T01:43:39.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.998+0000 7ffb5de00700 1 -- 172.21.15.52:0/2436286875 wait complete. 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.999+0000 7ffb5de00700 1 Processor -- start 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:39.999+0000 7ffb5de00700 1 -- start start 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 0x7ffb5819b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 0x7ffb5819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ffb58114e30 con 0x7ffb58105870 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffb58114cb0 con 0x7ffb58104ec0 2024-04-26T01:43:39.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb5de00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffb58114fb0 con 0x7ffb58109fa0 2024-04-26T01:43:39.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb56000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 0x7ffb5819b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb56a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb56a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49618/0 (socket says 172.21.15.52:49618) 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb56a00700 1 -- 172.21.15.52:0/3982870042 learned_addr learned my addr 172.21.15.52:0/3982870042 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.000+0000 7ffb57400700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 0x7ffb5819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 -- 172.21.15.52:0/3982870042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 msgr2=0x7ffb5819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 0x7ffb5819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 -- 172.21.15.52:0/3982870042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 msgr2=0x7ffb5819b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 -- 172.21.15.52:0/3982870042 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb581a03a0 con 0x7ffb58105870 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56a00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb56000700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 0x7ffb5819b920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400e9c0 tx=0x7ffb4400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400c9c0 con 0x7ffb58105870 2024-04-26T01:43:39.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb4400cb60 con 0x7ffb58105870 2024-04-26T01:43:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb581a0690 con 0x7ffb58105870 2024-04-26T01:43:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.001+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44018680 con 0x7ffb58105870 2024-04-26T01:43:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.002+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb581a7f10 con 0x7ffb58105870 2024-04-26T01:43:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.003+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb58069730 con 0x7ffb58105870 2024-04-26T01:43:40.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.005+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffb44010430 con 0x7ffb58105870 2024-04-26T01:43:40.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.007+0000 7ffb5ca00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:40.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.007+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7ffb440d0900 con 0x7ffb58105870 2024-04-26T01:43:40.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.008+0000 7ffb56a00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:40.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.008+0000 7ffb56a00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009990 tx=0x7ffb48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:40.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.011+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb4409d290 con 0x7ffb58105870 2024-04-26T01:43:40.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.287+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb5802df80 con 0x7ffb58105870 2024-04-26T01:43:40.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.288+0000 7ffb5ca00700 1 -- 172.21.15.52:0/3982870042 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7ffb4409cb90 con 0x7ffb58105870 2024-04-26T01:43:40.286 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:40.287 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:43:40.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.292+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 msgr2=0x7ffb3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.292+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009990 tx=0x7ffb48005960 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.292+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 msgr2=0x7ffb5819b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.292+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 0x7ffb5819b920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400e9c0 tx=0x7ffb4400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.292+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 shutdown_connections 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb58109fa0 0x7ffb5819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffb58104ec0 0x7ffb5819b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 --2- 172.21.15.52:0/3982870042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffb58105870 0x7ffb5819b920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 >> 172.21.15.52:0/3982870042 conn(0x7ffb58100750 msgr2=0x7ffb58102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:40.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 shutdown_connections 2024-04-26T01:43:40.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:40.293+0000 7ffb5de00700 1 -- 172.21.15.52:0/3982870042 wait complete. 2024-04-26T01:43:40.292 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:43:41.167 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:40 smithi052 ceph-mon[27851]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:41.167 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:40 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3982870042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:41.229 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:43:41.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:40 smithi191 ceph-mon[32745]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:41.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:40 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3982870042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:40 smithi183 ceph-mon[32752]: pgmap v411: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:40 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3982870042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:41.713 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:43 smithi052 ceph-mon[27851]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:43.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:43 smithi191 ceph-mon[32745]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:43 smithi183 ceph-mon[32752]: pgmap v412: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 -- 172.21.15.52:0/219216381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 msgr2=0x7f6cdc1052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 --2- 172.21.15.52:0/219216381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc1052c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009990 tx=0x7f6ccc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 -- 172.21.15.52:0/219216381 shutdown_connections 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 --2- 172.21.15.52:0/219216381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc109fc0 0x7f6cdc111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 --2- 172.21.15.52:0/219216381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc1052c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 --2- 172.21.15.52:0/219216381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc105890 0x7f6cdc1098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 -- 172.21.15.52:0/219216381 >> 172.21.15.52:0/219216381 conn(0x7f6cdc100770 msgr2=0x7f6cdc102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 -- 172.21.15.52:0/219216381 shutdown_connections 2024-04-26T01:43:44.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.071+0000 7f6ce3200700 1 -- 172.21.15.52:0/219216381 wait complete. 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.072+0000 7f6ce3200700 1 Processor -- start 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.072+0000 7f6ce3200700 1 -- start start 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc109fc0 0x7f6cdc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc114ed0 con 0x7f6cdc105890 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc114d50 con 0x7f6cdc104ee0 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce3200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc115050 con 0x7f6cdc109fc0 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6cdb400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6cdb400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50458/0 (socket says 172.21.15.52:50458) 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6cdb400700 1 -- 172.21.15.52:0/3617916204 learned_addr learned my addr 172.21.15.52:0/3617916204 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:44.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6cdbe00700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.073+0000 7f6ce1200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc109fc0 0x7f6cdc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 -- 172.21.15.52:0/3617916204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc109fc0 msgr2=0x7f6cdc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc109fc0 0x7f6cdc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 -- 172.21.15.52:0/3617916204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 msgr2=0x7f6cdc19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 -- 172.21.15.52:0/3617916204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc1a03b0 con 0x7f6cdc105890 2024-04-26T01:43:44.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdbe00700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cdb400700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800e9f0 tx=0x7f6cc800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8004a30 con 0x7f6cdc105890 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cc800cc70 con 0x7f6cdc105890 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8005530 con 0x7f6cdc105890 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.074+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc1a06a0 con 0x7f6cdc105890 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.075+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc1a7f20 con 0x7f6cdc105890 2024-04-26T01:43:44.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.076+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc069730 con 0x7f6cdc105890 2024-04-26T01:43:44.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.078+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6cc8004bd0 con 0x7f6cdc105890 2024-04-26T01:43:44.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.079+0000 7f6cd8c00700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 0x7f6cc00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:44.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.079+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f6cc80d05a0 con 0x7f6cdc105890 2024-04-26T01:43:44.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.079+0000 7f6cdbe00700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 0x7f6cc00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:44.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.080+0000 7f6cdbe00700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 0x7f6cc00a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b790 tx=0x7f6ccc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:44.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.084+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6cc800ce20 con 0x7f6cdc105890 2024-04-26T01:43:44.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.315+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cdc02df80 con 0x7f6cdc105890 2024-04-26T01:43:44.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.320+0000 7f6cd8c00700 1 -- 172.21.15.52:0/3617916204 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22667 (secure 0 0 0) 0x7f6cc809cf30 con 0x7f6cdc105890 2024-04-26T01:43:44.317 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:44.318 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":94,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:47.490242+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:43:44.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.322+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 msgr2=0x7f6cc00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.322+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 0x7f6cc00a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b790 tx=0x7f6ccc00b260 comp rx=0 tx=0).stop 2024-04-26T01:43:44.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.322+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 msgr2=0x7f6cdc19b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.322+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800e9f0 tx=0x7f6cc800ed00 comp rx=0 tx=0).stop 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 shutdown_connections 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6cc00a3cc0 0x7f6cc00a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cdc109fc0 0x7f6cdc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6cdc104ee0 0x7f6cdc19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 --2- 172.21.15.52:0/3617916204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cdc105890 0x7f6cdc19b8e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 >> 172.21.15.52:0/3617916204 conn(0x7f6cdc100770 msgr2=0x7f6cdc102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 shutdown_connections 2024-04-26T01:43:44.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:44.323+0000 7f6ce3200700 1 -- 172.21.15.52:0/3617916204 wait complete. 2024-04-26T01:43:45.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:43:45.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:45 smithi191 ceph-mon[32745]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:45 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3617916204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:45 smithi183 ceph-mon[32752]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:45 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3617916204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:45.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:45 smithi052 ceph-mon[27851]: pgmap v413: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:45.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:45 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3617916204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:43:45.541 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:47.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:47 smithi052 ceph-mon[27851]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:47.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:47 smithi191 ceph-mon[32745]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:47.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:47 smithi183 ceph-mon[32752]: pgmap v414: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 -- 172.21.15.52:0/760574644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf5010e3b0 msgr2=0x7faf50115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 --2- 172.21.15.52:0/760574644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf5010e3b0 0x7faf50115f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faf4800b0d0 tx=0x7faf4800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 -- 172.21.15.52:0/760574644 shutdown_connections 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 --2- 172.21.15.52:0/760574644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf5010e3b0 0x7faf50115f80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 --2- 172.21.15.52:0/760574644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf501092d0 0x7faf501096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 --2- 172.21.15.52:0/760574644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 -- 172.21.15.52:0/760574644 >> 172.21.15.52:0/760574644 conn(0x7faf50076210 msgr2=0x7faf50107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 -- 172.21.15.52:0/760574644 shutdown_connections 2024-04-26T01:43:47.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.929+0000 7faf58200700 1 -- 172.21.15.52:0/760574644 wait complete. 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.930+0000 7faf58200700 1 Processor -- start 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.930+0000 7faf58200700 1 -- start start 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.930+0000 7faf58200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.930+0000 7faf58200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf58200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5010e3b0 0x7faf501a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf58200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf50119250 con 0x7faf501092d0 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf58200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf501190d0 con 0x7faf50109c80 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf58200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf501193d0 con 0x7faf5010e3b0 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:47.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44280/0 (socket says 172.21.15.52:44280) 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf54e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 -- 172.21.15.52:0/857620716 learned_addr learned my addr 172.21.15.52:0/857620716 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf56200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5010e3b0 0x7faf501a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 -- 172.21.15.52:0/857620716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5010e3b0 msgr2=0x7faf501a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5010e3b0 0x7faf501a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 -- 172.21.15.52:0/857620716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 msgr2=0x7faf5019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf55800700 1 -- 172.21.15.52:0/857620716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf501a4690 con 0x7faf501092d0 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.931+0000 7faf54e00700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:47.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf55800700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf4400d980 tx=0x7faf4400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44016070 con 0x7faf501092d0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf44004830 con 0x7faf501092d0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44005540 con 0x7faf501092d0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf501a4980 con 0x7faf501092d0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.932+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf501ac200 con 0x7faf501092d0 2024-04-26T01:43:47.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.934+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf50069730 con 0x7faf501092d0 2024-04-26T01:43:47.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.936+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf44004220 con 0x7faf501092d0 2024-04-26T01:43:47.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.937+0000 7faf4e000700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 0x7faf380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:47.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.937+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7faf440cff50 con 0x7faf501092d0 2024-04-26T01:43:47.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.937+0000 7faf54e00700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 0x7faf380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:47.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.938+0000 7faf54e00700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 0x7faf380a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faf501088a0 tx=0x7faf3c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:47.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:47.941+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf4401f030 con 0x7faf501092d0 2024-04-26T01:43:48.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.223+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf5002df80 con 0x7faf501092d0 2024-04-26T01:43:48.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.224+0000 7faf4e000700 1 -- 172.21.15.52:0/857620716 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7faf4409c8e0 con 0x7faf501092d0 2024-04-26T01:43:48.222 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:43:48.222 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:43:48.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 msgr2=0x7faf380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 0x7faf380a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faf501088a0 tx=0x7faf3c00b410 comp rx=0 tx=0).stop 2024-04-26T01:43:48.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 msgr2=0x7faf5019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:48.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf4400d980 tx=0x7faf4400dc90 comp rx=0 tx=0).stop 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 shutdown_connections 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7faf380a3cc0 0x7faf380a6180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf5010e3b0 0x7faf501a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf501092d0 0x7faf5019f680 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 --2- 172.21.15.52:0/857620716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf50109c80 0x7faf5019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 >> 172.21.15.52:0/857620716 conn(0x7faf50076210 msgr2=0x7faf50076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:48.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 shutdown_connections 2024-04-26T01:43:48.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:48.227+0000 7faf58200700 1 -- 172.21.15.52:0/857620716 wait complete. 2024-04-26T01:43:48.226 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:43:49.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:41:06.424614+0000', 'modified': '2024-04-26T01:41:31.153156+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24512, 'mds_2': 24496}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24472': {'gid': 24472, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/234906181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 234906181}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 234906181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6837/2929119914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2929119914}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2929119914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.183:6837/1263803776', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6836', 'nonce': 1263803776}, {'type': 'v1', 'addr': '172.21.15.183:6837', 'nonce': 1263803776}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.191:6839/1544004799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 1544004799}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 1544004799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'a', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.52:6841/1834450967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6840', 'nonce': 1834450967}, {'type': 'v1', 'addr': '172.21.15.52:6841', 'nonce': 1834450967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.183:6839/1410734884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6838', 'nonce': 1410734884}, {'type': 'v1', 'addr': '172.21.15.183:6839', 'nonce': 1410734884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24512, 24496, 24524]} 2024-04-26T01:43:49.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:43:49.119 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:43:49.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:49 smithi052 ceph-mon[27851]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:49 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/857620716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:49 smithi191 ceph-mon[32745]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:49 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/857620716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:49 smithi183 ceph-mon[32752]: pgmap v415: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:49 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/857620716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:43:49.541 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:51.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:51 smithi191 ceph-mon[32745]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:51 smithi183 ceph-mon[32752]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:51.452 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:51 smithi052 ceph-mon[27851]: pgmap v416: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:52.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.051+0000 7fce97600700 1 -- 172.21.15.52:0/2083052891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90109fa0 msgr2=0x7fce90111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.051+0000 7fce97600700 1 --2- 172.21.15.52:0/2083052891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90109fa0 0x7fce90111b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fce88009fc0 tx=0x7fce8800b330 comp rx=0 tx=0).stop 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.051+0000 7fce97600700 1 -- 172.21.15.52:0/2083052891 shutdown_connections 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.051+0000 7fce97600700 1 --2- 172.21.15.52:0/2083052891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90109fa0 0x7fce90111b70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.051+0000 7fce97600700 1 --2- 172.21.15.52:0/2083052891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90104ec0 0x7fce901052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.052+0000 7fce97600700 1 --2- 172.21.15.52:0/2083052891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 0x7fce901098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.052+0000 7fce97600700 1 -- 172.21.15.52:0/2083052891 >> 172.21.15.52:0/2083052891 conn(0x7fce90100750 msgr2=0x7fce90102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.052+0000 7fce97600700 1 -- 172.21.15.52:0/2083052891 shutdown_connections 2024-04-26T01:43:52.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.052+0000 7fce97600700 1 -- 172.21.15.52:0/2083052891 wait complete. 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.053+0000 7fce97600700 1 Processor -- start 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.053+0000 7fce97600700 1 -- start start 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 0x7fce9019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90109fa0 0x7fce9019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce90114e90 con 0x7fce90109fa0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce90114d10 con 0x7fce90105870 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce97600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce90115010 con 0x7fce90104ec0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce94c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce94c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60732/0 (socket says 172.21.15.52:60732) 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce94c00700 1 -- 172.21.15.52:0/2724704542 learned_addr learned my addr 172.21.15.52:0/2724704542 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce95600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90109fa0 0x7fce9019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.054+0000 7fce8fe00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 0x7fce9019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 -- 172.21.15.52:0/2724704542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 msgr2=0x7fce9019b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 0x7fce9019b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 -- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90109fa0 msgr2=0x7fce9019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90109fa0 0x7fce9019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 -- 172.21.15.52:0/2724704542 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce901a0280 con 0x7fce90104ec0 2024-04-26T01:43:52.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce94c00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce8400e9c0 tx=0x7fce8400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce8d600700 1 -- 172.21.15.52:0/2724704542 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84004a30 con 0x7fce90104ec0 2024-04-26T01:43:52.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce901a0570 con 0x7fce90104ec0 2024-04-26T01:43:52.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce8d600700 1 -- 172.21.15.52:0/2724704542 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce8400cc70 con 0x7fce90104ec0 2024-04-26T01:43:52.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.055+0000 7fce8d600700 1 -- 172.21.15.52:0/2724704542 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce840054a0 con 0x7fce90104ec0 2024-04-26T01:43:52.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.056+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce901a7df0 con 0x7fce90104ec0 2024-04-26T01:43:52.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.059+0000 7fce8d600700 1 -- 172.21.15.52:0/2724704542 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fce84004bd0 con 0x7fce90104ec0 2024-04-26T01:43:52.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.060+0000 7fce8d600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 0x7fce780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.060+0000 7fce8d600700 1 -- 172.21.15.52:0/2724704542 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fce840d0520 con 0x7fce90104ec0 2024-04-26T01:43:52.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.060+0000 7fce8fe00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 0x7fce780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 Processor -- start 2024-04-26T01:43:52.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 -- start start 2024-04-26T01:43:52.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fce8fe00700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 0x7fce780a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fce9010b4a0 tx=0x7fce7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcd0a0 0x55634dbc69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbc6ef0 0x55634dbc7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55634db18e40 con 0x55634dbcc780 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55634dabf6e0 con 0x55634dbc6ef0 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.061+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55634db14e00 con 0x55634dbcd0a0 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44314/0 (socket says 172.21.15.52:44314) 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 -- 172.21.15.52:0/1855781245 learned_addr learned my addr 172.21.15.52:0/1855781245 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce8fe00700 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcd0a0 0x55634dbc69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce95600700 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbc6ef0 0x55634dbc7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 -- 172.21.15.52:0/1855781245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcd0a0 msgr2=0x55634dbc69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcd0a0 0x55634dbc69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 -- 172.21.15.52:0/1855781245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbc6ef0 msgr2=0x55634dbc7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbc6ef0 0x55634dbc7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 -- 172.21.15.52:0/1855781245 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55634dab5630 con 0x55634dbcc780 2024-04-26T01:43:52.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.062+0000 7fce94c00700 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c740 tx=0x7fce8409c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ce0cec682e1117 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.063+0000 7fce82000700 1 -- 172.21.15.52:0/1855781245 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8409c7b0 con 0x55634dbcc780 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.063+0000 7fce82000700 1 -- 172.21.15.52:0/1855781245 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce840a0b60 con 0x55634dbcc780 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.063+0000 7fce82000700 1 -- 172.21.15.52:0/1855781245 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce840d0310 con 0x55634dbcc780 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.063+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/1855781245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 msgr2=0x55634dbccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.063+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c740 tx=0x7fce8409c110 comp rx=0 tx=0).stop 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/1855781245 shutdown_connections 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbc6ef0 0x55634dbc7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbccb60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/1855781245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcd0a0 0x55634dbc69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/1855781245 >> 172.21.15.52:0/1855781245 conn(0x55634daf4f00 msgr2=0x55634dbc18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/1855781245 shutdown_connections 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/1855781245 wait complete. 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.064+0000 7fcea1fb0b80 1 -- start start 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 Processor -- start 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcc780 0x55634dbd0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbd09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55634dbd3920 con 0x55634dbc6ef0 2024-04-26T01:43:52.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55634dad2810 con 0x55634dbd0570 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55634dbd3aa0 con 0x55634dbcc780 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fcea1fb0b80 10 client.0 _ll_register_callbacks cb 0x55634dab16d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.065+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44320/0 (socket says 172.21.15.52:44320) 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 -- 172.21.15.52:0/4237522948 learned_addr learned my addr 172.21.15.52:0/4237522948 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce95600700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbd09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce8fe00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcc780 0x55634dbd0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 -- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcc780 msgr2=0x55634dbd0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcc780 0x55634dbd0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 -- 172.21.15.52:0/4237522948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 msgr2=0x55634dbd09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbd09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 -- 172.21.15.52:0/4237522948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55634da66880 con 0x55634dbc6ef0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce95600700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbd09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce94c00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce84009fd0 tx=0x7fce8409cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4ed9f8e657dbee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.066+0000 7fce76a00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84005640 con 0x55634dbc6ef0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fce840d0110 con 0x55634dbc6ef0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce840a0b60 con 0x55634dbc6ef0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fce84082050 con 0x55634dbc6ef0 2024-04-26T01:43:52.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fcea1fb0b80 10 client.14757 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55634daafaa0 con 0x55634dbc6ef0 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce840a0b60 con 0x55634dbc6ef0 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.067+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fce840b3020 con 0x55634dbc6ef0 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 10 client.14757 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 10 client.14757 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 20 client.14757 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dbdf800 0x55634dbe19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 4 client.14757 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55634db26630 con 0x55634dbdf800 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fce8fe00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dbdf800 0x55634dbe19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fce8fe00700 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dbdf800 0x55634dbe19d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e71aa9677268ad9c server_cookie=28c9d73b937c56d6 in_seq=0 out_seq=0 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.068+0000 7fce76a00700 10 client.14757 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.069+0000 7fce76a00700 1 -- 172.21.15.52:0/4237522948 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55634db26630 con 0x55634dbdf800 2024-04-26T01:43:52.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.069+0000 7fce76a00700 10 client.14757 handle_command_reply: tid=0 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.069+0000 7fcea1fb0b80 1 client.14757 shutdown 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 msgr2=0x55634dbd2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce84009fd0 tx=0x7fce8409cf70 comp rx=0 tx=0).stop 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 shutdown_connections 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dbdf800 0x55634dbe19d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbd09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbd2d30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/4237522948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbcc780 0x55634dbd0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 >> 172.21.15.52:0/4237522948 conn(0x55634daf4f00 msgr2=0x55634dbc0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fce76a00700 0 client.14757 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 shutdown_connections 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/4237522948 wait complete. 2024-04-26T01:43:52.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.070+0000 7fcea1fb0b80 20 client.14757 trim_cache size 0 max 16384 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.093+0000 7fcea1fb0b80 1 Processor -- start 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.093+0000 7fcea1fb0b80 1 -- start start 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbcc780 0x55634dbdcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbd0570 0x55634dbcdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55634dbdfd10 con 0x55634dbc6ef0 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55634dbe03e0 con 0x55634dbcc780 2024-04-26T01:43:52.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55634dbe0ab0 con 0x55634dbd0570 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44336/0 (socket says 172.21.15.52:44336) 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 -- 172.21.15.52:0/2883430329 learned_addr learned my addr 172.21.15.52:0/2883430329 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce8fe00700 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbcc780 0x55634dbdcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce95600700 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbd0570 0x55634dbcdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 -- 172.21.15.52:0/2883430329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbd0570 msgr2=0x55634dbcdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbd0570 0x55634dbcdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 -- 172.21.15.52:0/2883430329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbcc780 msgr2=0x55634dbdcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbcc780 0x55634dbdcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.094+0000 7fce94c00700 1 -- 172.21.15.52:0/2883430329 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55634dbd2970 con 0x55634dbc6ef0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.095+0000 7fce94c00700 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c740 tx=0x7fce84098410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80819eb579dca89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.095+0000 7fce81600700 1 -- 172.21.15.52:0/2883430329 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce840a5420 con 0x55634dbc6ef0 2024-04-26T01:43:52.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.095+0000 7fce81600700 1 -- 172.21.15.52:0/2883430329 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce840bb070 con 0x55634dbc6ef0 2024-04-26T01:43:52.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.095+0000 7fce81600700 1 -- 172.21.15.52:0/2883430329 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84005740 con 0x55634dbc6ef0 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/2883430329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 msgr2=0x55634dbc69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c740 tx=0x7fce84098410 comp rx=0 tx=0).stop 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/2883430329 shutdown_connections 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbd0570 0x55634dbcdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbc6ef0 0x55634dbc69b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/2883430329 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbcc780 0x55634dbdcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/2883430329 >> 172.21.15.52:0/2883430329 conn(0x55634daf4f00 msgr2=0x55634dbcd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/2883430329 shutdown_connections 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.096+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/2883430329 wait complete. 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 -- start start 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 Processor -- start 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbe9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.097+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55634dbe1850 con 0x55634dbcc780 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55634dbe1180 con 0x55634dbd0570 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fcea1fb0b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55634dbe1f20 con 0x55634dbc6ef0 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fcea1fb0b80 10 client.0 _ll_register_callbacks cb 0x55634dab16d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce8fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60778/0 (socket says 172.21.15.52:60778) 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 -- 172.21.15.52:0/509691782 learned_addr learned my addr 172.21.15.52:0/509691782 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce8fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44348/0 (socket says 172.21.15.52:44348) 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce95600700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbe9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 -- 172.21.15.52:0/509691782 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 msgr2=0x55634dbe9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbe9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 -- 172.21.15.52:0/509691782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 msgr2=0x55634dbc1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:52.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce94c00700 1 -- 172.21.15.52:0/509691782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55634dbe9940 con 0x55634dbc6ef0 2024-04-26T01:43:52.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.098+0000 7fce8fe00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:52.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce94c00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c6f0 tx=0x7fce84004ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c61f6564d5f91843 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:52.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce76a00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce840a5420 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fce84016330 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce840bb070 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fce84016530 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fcea1fb0b80 10 client.24868 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55634dc505e0 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.099+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce840bb070 con 0x55634dbc6ef0 2024-04-26T01:43:52.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fce840b3070 con 0x55634dbc6ef0 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 10 client.24868 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 10 client.24868 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 20 client.24868 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dc54d70 0x55634dc53a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 4 client.24868 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "status"}) -- 0x55634db206d0 con 0x55634dc54d70 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.100+0000 7fce8fe00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dc54d70 0x55634dc53a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fce8fe00700 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dc54d70 0x55634dc53a80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e89564ebaddd139 server_cookie=9ac581574e499418 in_seq=0 out_seq=0 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fce76a00700 10 client.24868 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fce76a00700 1 -- 172.21.15.52:0/509691782 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55634db206d0 con 0x55634dc54d70 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fce76a00700 10 client.24868 handle_command_reply: tid=0 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 client.24868 shutdown 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 msgr2=0x55634dbc1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fce8409c6f0 tx=0x7fce84004ec0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 shutdown_connections 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55634dc54d70 0x55634dc53a80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55634dbd0570 0x55634dbe9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55634dbc6ef0 0x55634dbc1020 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fce76a00700 0 client.24868 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 --2- 172.21.15.52:0/509691782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55634dbcc780 0x55634dbc1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 >> 172.21.15.52:0/509691782 conn(0x55634daf4f00 msgr2=0x55634dbdbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 shutdown_connections 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.101+0000 7fcea1fb0b80 1 -- 172.21.15.52:0/509691782 wait complete. 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.102+0000 7fcea1fb0b80 20 client.24868 trim_cache size 0 max 16384 2024-04-26T01:43:52.102 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 2, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24496, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 147.73478893500001, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 199.30099543700001 2024-04-26T01:43:52.103 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.102+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 msgr2=0x7fce780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.102+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 0x7fce780a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fce9010b4a0 tx=0x7fce7c009260 comp rx=0 tx=0).stop 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.102+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 msgr2=0x7fce9019b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.102+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce8400e9c0 tx=0x7fce8400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 shutdown_connections 2024-04-26T01:43:52.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fce780a3c40 0x7fce780a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce90109fa0 0x7fce9019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce90104ec0 0x7fce9019b2c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 --2- 172.21.15.52:0/2724704542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce90105870 0x7fce9019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 >> 172.21.15.52:0/2724704542 conn(0x7fce90100750 msgr2=0x7fce901011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 shutdown_connections 2024-04-26T01:43:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:52.103+0000 7fce97600700 1 -- 172.21.15.52:0/2724704542 wait complete. 2024-04-26T01:43:53.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.a status 2024-04-26T01:43:53.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:53 smithi191 ceph-mon[32745]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:53 smithi183 ceph-mon[32752]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:53.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:53 smithi052 ceph-mon[27851]: pgmap v417: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:43:53.595 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:55.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:55 smithi052 ceph-mon[27851]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:55.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:55 smithi191 ceph-mon[32745]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:55.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:55 smithi183 ceph-mon[32752]: pgmap v418: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 -- 172.21.15.52:0/2127198698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b8006a750 msgr2=0x7f6b8006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 --2- 172.21.15.52:0/2127198698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b8006a750 0x7f6b8006abb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6b700098f0 tx=0x7f6b70009c00 comp rx=0 tx=0).stop 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 -- 172.21.15.52:0/2127198698 shutdown_connections 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 --2- 172.21.15.52:0/2127198698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006b0f0 0x7f6b8010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 --2- 172.21.15.52:0/2127198698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b80108a30 0x7f6b80108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 --2- 172.21.15.52:0/2127198698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b8006a750 0x7f6b8006abb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 -- 172.21.15.52:0/2127198698 >> 172.21.15.52:0/2127198698 conn(0x7f6b800788f0 msgr2=0x7f6b80078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 -- 172.21.15.52:0/2127198698 shutdown_connections 2024-04-26T01:43:55.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.911+0000 7f6b88800700 1 -- 172.21.15.52:0/2127198698 wait complete. 2024-04-26T01:43:55.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 Processor -- start 2024-04-26T01:43:55.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 -- start start 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 0x7f6b8019b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b80108a30 0x7f6b8019fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80112bd0 con 0x7f6b80108a30 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80112a50 con 0x7f6b8006a750 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.913+0000 7f6b88800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80112d50 con 0x7f6b8006b0f0 2024-04-26T01:43:55.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40540/0 (socket says 172.21.15.52:40540) 2024-04-26T01:43:55.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 -- 172.21.15.52:0/613982602 learned_addr learned my addr 172.21.15.52:0/613982602 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:55.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b86800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b80108a30 0x7f6b8019fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85400700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 0x7f6b8019b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 -- 172.21.15.52:0/613982602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 msgr2=0x7f6b8019b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 0x7f6b8019b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 -- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b80108a30 msgr2=0x7f6b8019fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b80108a30 0x7f6b8019fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 -- 172.21.15.52:0/613982602 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b801a0390 con 0x7f6b8006a750 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85400700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 0x7f6b8019b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.914+0000 7f6b85e00700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e9c0 tx=0x7f6b7400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.915+0000 7f6b7aa00700 1 -- 172.21.15.52:0/613982602 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400cbd0 con 0x7f6b8006a750 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.915+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b801a0680 con 0x7f6b8006a750 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.915+0000 7f6b7aa00700 1 -- 172.21.15.52:0/613982602 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b7400ee60 con 0x7f6b8006a750 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.915+0000 7f6b7aa00700 1 -- 172.21.15.52:0/613982602 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b740055c0 con 0x7f6b8006a750 2024-04-26T01:43:55.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.915+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b801a7f00 con 0x7f6b8006a750 2024-04-26T01:43:55.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.918+0000 7f6b7aa00700 1 -- 172.21.15.52:0/613982602 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6b74004750 con 0x7f6b8006a750 2024-04-26T01:43:55.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.919+0000 7f6b930c4b80 1 Processor -- start 2024-04-26T01:43:55.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b7aa00700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 0x7f6b680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 -- start start 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b7aa00700 1 -- 172.21.15.52:0/613982602 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f6b740d0620 con 0x7f6b8006a750 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b86800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 0x7f6b680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe50a0 0x563bb3fde9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563bb3f30e40 con 0x563bb3fdeef0 2024-04-26T01:43:55.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ed76e0 con 0x563bb3fe4780 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563bb3f2ce00 con 0x563bb3fe50a0 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.920+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b86800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40550/0 (socket says 172.21.15.52:40550) 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 -- 172.21.15.52:0/122514384 learned_addr learned my addr 172.21.15.52:0/122514384 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:55.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85400700 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe50a0 0x563bb3fde9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 -- 172.21.15.52:0/122514384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe50a0 msgr2=0x563bb3fde9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe50a0 0x563bb3fde9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 -- 172.21.15.52:0/122514384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 msgr2=0x563bb3fdf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 -- 172.21.15.52:0/122514384 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb3ecd630 con 0x563bb3fe4780 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b86800700 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b86800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 0x7f6b680a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00f870 tx=0x7f6b7c00fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.921+0000 7f6b85e00700 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6b74099280 tx=0x7f6b7409c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e242d4a51a25a0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.922+0000 7f6b6f400700 1 -- 172.21.15.52:0/122514384 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7409c8b0 con 0x563bb3fe4780 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.922+0000 7f6b6f400700 1 -- 172.21.15.52:0/122514384 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b740a0c60 con 0x563bb3fe4780 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.922+0000 7f6b6f400700 1 -- 172.21.15.52:0/122514384 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7409dbe0 con 0x563bb3fe4780 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.922+0000 7f6b930c4b80 1 -- 172.21.15.52:0/122514384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 msgr2=0x563bb3fe4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.922+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6b74099280 tx=0x7f6b7409c3f0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 -- 172.21.15.52:0/122514384 shutdown_connections 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe4b60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/122514384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe50a0 0x563bb3fde9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 -- 172.21.15.52:0/122514384 >> 172.21.15.52:0/122514384 conn(0x563bb3f0cf00 msgr2=0x563bb3fd98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 -- 172.21.15.52:0/122514384 shutdown_connections 2024-04-26T01:43:55.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 -- 172.21.15.52:0/122514384 wait complete. 2024-04-26T01:43:55.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.923+0000 7f6b930c4b80 1 -- start start 2024-04-26T01:43:55.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 Processor -- start 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 0x563bb3feac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 0x563bb3fe8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563bb3feb920 con 0x563bb3fe84a0 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bb3eea810 con 0x563bb3fe4780 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563bb3febaa0 con 0x563bb3fdeef0 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b930c4b80 10 client.0 _ll_register_callbacks cb 0x563bb3ec96d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.924+0000 7f6b86800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 0x563bb3fe8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40564/0 (socket says 172.21.15.52:40564) 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85400700 1 -- 172.21.15.52:0/3359639747 learned_addr learned my addr 172.21.15.52:0/3359639747 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:55.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85e00700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 0x563bb3feac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b86800700 1 -- 172.21.15.52:0/3359639747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 msgr2=0x563bb3feac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b86800700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 0x563bb3feac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b86800700 1 -- 172.21.15.52:0/3359639747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 msgr2=0x563bb3fe7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b86800700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b86800700 1 -- 172.21.15.52:0/3359639747 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb3e7e880 con 0x563bb3fe84a0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85e00700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 0x563bb3feac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.925+0000 7f6b85400700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b86800700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 0x563bb3fe8920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a20 tx=0x7f6b7c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a9064319127952 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b63e00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0089e0 con 0x563bb3fe84a0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f6b7c008b80 con 0x563bb3fe84a0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b740a0c60 con 0x563bb3fe84a0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b7c008d80 con 0x563bb3fe84a0 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b930c4b80 10 client.14775 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:55.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.926+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563bb3ec7aa0 con 0x563bb3fe84a0 2024-04-26T01:43:55.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b740a0c60 con 0x563bb3fe84a0 2024-04-26T01:43:55.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6b7c01a440 con 0x563bb3fe84a0 2024-04-26T01:43:55.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 10 client.14775 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 10 client.14775 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 20 client.14775 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb3ff89f0 0x563bb3ff9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 4 client.14775 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.927+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bb3f3e630 con 0x563bb3ff89f0 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.928+0000 7f6b85e00700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb3ff89f0 0x563bb3ff9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.928+0000 7f6b85e00700 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb3ff89f0 0x563bb3ff9e60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80d91292d342a58e server_cookie=af2326861ce27592 in_seq=0 out_seq=0 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.928+0000 7f6b63e00700 10 client.14775 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b63e00700 1 -- 172.21.15.52:0/3359639747 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bb3f3e630 con 0x563bb3ff89f0 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b63e00700 10 client.14775 handle_command_reply: tid=0 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 client.14775 shutdown 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 msgr2=0x563bb3fe8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 0x563bb3fe8920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a20 tx=0x7f6b7c004b40 comp rx=0 tx=0).stop 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 shutdown_connections 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb3ff89f0 0x563bb3ff9e60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe84a0 0x563bb3fe8920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b63e00700 0 client.14775 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.929+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fdeef0 0x563bb3feac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.930+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/3359639747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe4780 0x563bb3fe7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.930+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 >> 172.21.15.52:0/3359639747 conn(0x563bb3f0cf00 msgr2=0x563bb3fd8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.930+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 shutdown_connections 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.930+0000 7f6b930c4b80 1 -- 172.21.15.52:0/3359639747 wait complete. 2024-04-26T01:43:55.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.930+0000 7f6b930c4b80 20 client.14775 trim_cache size 0 max 16384 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 Processor -- start 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 -- start start 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe84a0 0x563bb40032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ff81a0 con 0x563bb3fe4780 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ff8870 con 0x563bb3fdeef0 2024-04-26T01:43:55.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.953+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ff8f40 con 0x563bb3fe84a0 2024-04-26T01:43:55.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36466/0 (socket says 172.21.15.52:36466) 2024-04-26T01:43:55.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 -- 172.21.15.52:0/2629290829 learned_addr learned my addr 172.21.15.52:0/2629290829 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:55.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85e00700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b86800700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe84a0 0x563bb40032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 -- 172.21.15.52:0/2629290829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe84a0 msgr2=0x563bb40032c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe84a0 0x563bb40032c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 -- 172.21.15.52:0/2629290829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 msgr2=0x563bb3fdf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85400700 1 -- 172.21.15.52:0/2629290829 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb3fe2a40 con 0x563bb3fe4780 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.954+0000 7f6b85e00700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.955+0000 7f6b85400700 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009810 tx=0x7f6b70005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711db15ce4599828 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.955+0000 7f6b6ea00700 1 -- 172.21.15.52:0/2629290829 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70028070 con 0x563bb3fe4780 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.955+0000 7f6b6ea00700 1 -- 172.21.15.52:0/2629290829 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b70003680 con 0x563bb3fe4780 2024-04-26T01:43:55.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.955+0000 7f6b6ea00700 1 -- 172.21.15.52:0/2629290829 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000b7a0 con 0x563bb3fe4780 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 -- 172.21.15.52:0/2629290829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 msgr2=0x563bb3fd95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009810 tx=0x7f6b70005dc0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 -- 172.21.15.52:0/2629290829 shutdown_connections 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe84a0 0x563bb40032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fdeef0 0x563bb3fdf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/2629290829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fe4780 0x563bb3fd95d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 -- 172.21.15.52:0/2629290829 >> 172.21.15.52:0/2629290829 conn(0x563bb3f0cf00 msgr2=0x563bb3fe6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 -- 172.21.15.52:0/2629290829 shutdown_connections 2024-04-26T01:43:55.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.956+0000 7f6b930c4b80 1 -- 172.21.15.52:0/2629290829 wait complete. 2024-04-26T01:43:55.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 -- start start 2024-04-26T01:43:55.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 Processor -- start 2024-04-26T01:43:55.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb40032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 0x563bb4003820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 0x563bb3ffe490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ff9ce0 con 0x563bb3fdeef0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ff9610 con 0x563bb3fe84a0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563bb3ffa3b0 con 0x563bb3fe4780 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.957+0000 7f6b930c4b80 10 client.0 _ll_register_callbacks cb 0x563bb3ec96d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb40032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb40032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36472/0 (socket says 172.21.15.52:36472) 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85e00700 1 -- 172.21.15.52:0/1507004731 learned_addr learned my addr 172.21.15.52:0/1507004731 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 0x563bb4003820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b86800700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 0x563bb3ffe490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 -- 172.21.15.52:0/1507004731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 msgr2=0x563bb3ffe490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 0x563bb3ffe490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 -- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 msgr2=0x563bb40032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb40032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b85400700 1 -- 172.21.15.52:0/1507004731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb3ffecd0 con 0x563bb3fe4780 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.958+0000 7f6b86800700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 0x563bb3ffe490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b85400700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 0x563bb4003820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005e90 tx=0x7f6b70003ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92ea335ccffdd220 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b63e00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70028040 con 0x563bb3fe4780 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f6b700052a0 con 0x563bb3fe4780 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b70003680 con 0x563bb3fe4780 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b700054a0 con 0x563bb3fe4780 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b930c4b80 10 client.24886 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:55.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.959+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563bb4068f70 con 0x563bb3fe4780 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b70003680 con 0x563bb3fe4780 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6b7002d450 con 0x563bb3fe4780 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 10 client.24886 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 10 client.24886 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 20 client.24886 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb406d5c0 0x563bb406c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 4 client.24886 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "status"}) -- 0x563bb3f386d0 con 0x563bb406d5c0 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.960+0000 7f6b85e00700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb406d5c0 0x563bb406c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b85e00700 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb406d5c0 0x563bb406c2d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2439f1169f1edc76 server_cookie=23ac10a2dc409e5c in_seq=0 out_seq=0 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b63e00700 10 client.24886 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:43:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b63e00700 1 -- 172.21.15.52:0/1507004731 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563bb3f386d0 con 0x563bb406d5c0 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b63e00700 10 client.24886 handle_command_reply: tid=0 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b930c4b80 1 client.24886 shutdown 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 msgr2=0x563bb4003820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 0x563bb4003820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005e90 tx=0x7f6b70003ef0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 shutdown_connections 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.961+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x563bb406d5c0 0x563bb406c2d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563bb3fe84a0 0x563bb3ffe490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563bb3fdeef0 0x563bb40032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 --2- 172.21.15.52:0/1507004731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563bb3fe4780 0x563bb4003820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b63e00700 0 client.24886 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 >> 172.21.15.52:0/1507004731 conn(0x563bb3f0cf00 msgr2=0x563bb3fe5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 shutdown_connections 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 1 -- 172.21.15.52:0/1507004731 wait complete. 2024-04-26T01:43:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.962+0000 7f6b930c4b80 20 client.24886 trim_cache size 0 max 16384 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 1, 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24512, 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 152.60759741800001, 2024-04-26T01:43:55.962 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:43:55.963 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:43:55.963 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:43:55.963 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 192.95001287599999 2024-04-26T01:43:55.963 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:43:55.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 msgr2=0x7f6b680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 0x7f6b680a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00f870 tx=0x7f6b7c00fc30 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 msgr2=0x7f6b8019b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e9c0 tx=0x7f6b7400ecd0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 shutdown_connections 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b680a3c40 0x7f6b680a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b80108a30 0x7f6b8019fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b8006a750 0x7f6b8019b380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 --2- 172.21.15.52:0/613982602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b8006b0f0 0x7f6b8019b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 >> 172.21.15.52:0/613982602 conn(0x7f6b800788f0 msgr2=0x7f6b800ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 shutdown_connections 2024-04-26T01:43:55.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:55.963+0000 7f6b88800700 1 -- 172.21.15.52:0/613982602 wait complete. 2024-04-26T01:43:56.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:43:57.306 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:43:57.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:57 smithi191 ceph-mon[32745]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:57.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:57 smithi183 ceph-mon[32752]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:57.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:57 smithi052 ceph-mon[27851]: pgmap v419: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:43:59.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:43:59 smithi052 ceph-mon[27851]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:59.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:43:59 smithi191 ceph-mon[32745]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:59.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:43:59 smithi183 ceph-mon[32752]: pgmap v420: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:43:59.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 -- 172.21.15.52:0/457053320 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc010e3b0 msgr2=0x7fbfc0115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 --2- 172.21.15.52:0/457053320 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc010e3b0 0x7fbfc0115f80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80099c0 tx=0x7fbfb8009cd0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 -- 172.21.15.52:0/457053320 shutdown_connections 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 --2- 172.21.15.52:0/457053320 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc010e3b0 0x7fbfc0115f80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 --2- 172.21.15.52:0/457053320 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc01092d0 0x7fbfc01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 --2- 172.21.15.52:0/457053320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc0109c80 0x7fbfc010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 -- 172.21.15.52:0/457053320 >> 172.21.15.52:0/457053320 conn(0x7fbfc0076210 msgr2=0x7fbfc0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 -- 172.21.15.52:0/457053320 shutdown_connections 2024-04-26T01:43:59.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.644+0000 7fbfc5000700 1 -- 172.21.15.52:0/457053320 wait complete. 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 Processor -- start 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 -- start start 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc0109c80 0x7fbfc019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc010e3b0 0x7fbfc01a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01192a0 con 0x7fbfc01092d0 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0119120 con 0x7fbfc0109c80 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.646+0000 7fbfc5000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0119420 con 0x7fbfc010e3b0 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36494/0 (socket says 172.21.15.52:36494) 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 -- 172.21.15.52:0/1256874276 learned_addr learned my addr 172.21.15.52:0/1256874276 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfb5400700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc0109c80 0x7fbfc019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbe800700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc010e3b0 0x7fbfc01a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 -- 172.21.15.52:0/1256874276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc010e3b0 msgr2=0x7fbfc01a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc010e3b0 0x7fbfc01a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 -- 172.21.15.52:0/1256874276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc0109c80 msgr2=0x7fbfc019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc0109c80 0x7fbfc019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 -- 172.21.15.52:0/1256874276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc01a46e0 con 0x7fbfc01092d0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.647+0000 7fbfbde00700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000eb30 tx=0x7fbfb000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.648+0000 7fbfb7400700 1 -- 172.21.15.52:0/1256874276 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000cc90 con 0x7fbfc01092d0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.648+0000 7fbfb7400700 1 -- 172.21.15.52:0/1256874276 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00044d0 con 0x7fbfc01092d0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.648+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc01a49d0 con 0x7fbfc01092d0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.648+0000 7fbfb7400700 1 -- 172.21.15.52:0/1256874276 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0010430 con 0x7fbfc01092d0 2024-04-26T01:43:59.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.648+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc01ac250 con 0x7fbfc01092d0 2024-04-26T01:43:59.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.651+0000 7fbfb7400700 1 -- 172.21.15.52:0/1256874276 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbfb0003ea0 con 0x7fbfc01092d0 2024-04-26T01:43:59.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfb7400700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 0x7fbfa40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfcfa80b80 1 Processor -- start 2024-04-26T01:43:59.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfb7400700 1 -- 172.21.15.52:0/1256874276 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fbfb00d0650 con 0x7fbfc01092d0 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfb5400700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 0x7fbfa40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfcfa80b80 1 -- start start 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.653+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fee0a0 0x55e575fe79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fe7ef0 0x55e575fe8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e575f39e40 con 0x55e575fe7ef0 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e575ee06e0 con 0x55e575fed780 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e575f35e00 con 0x55e575fee0a0 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40606/0 (socket says 172.21.15.52:40606) 2024-04-26T01:43:59.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 -- 172.21.15.52:0/3427779411 learned_addr learned my addr 172.21.15.52:0/3427779411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbe800700 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fe7ef0 0x55e575fe8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfb5400700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 0x7fbfa40a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbfc01087a0 tx=0x7fbfa800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfb5400700 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fee0a0 0x55e575fe79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 -- 172.21.15.52:0/3427779411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fee0a0 msgr2=0x55e575fe79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fee0a0 0x55e575fe79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 -- 172.21.15.52:0/3427779411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fe7ef0 msgr2=0x55e575fe8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fe7ef0 0x55e575fe8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.654+0000 7fbfbde00700 1 -- 172.21.15.52:0/3427779411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e575ed6630 con 0x55e575fed780 2024-04-26T01:43:59.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.655+0000 7fbfbde00700 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009c870 tx=0x7fbfb0098cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf87492d3790935e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.655+0000 7fbfaf400700 1 -- 172.21.15.52:0/3427779411 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb009cfe0 con 0x55e575fed780 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.655+0000 7fbfaf400700 1 -- 172.21.15.52:0/3427779411 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb009c8e0 con 0x55e575fed780 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.655+0000 7fbfaf400700 1 -- 172.21.15.52:0/3427779411 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0004190 con 0x55e575fed780 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/3427779411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 msgr2=0x55e575fedb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009c870 tx=0x7fbfb0098cb0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/3427779411 shutdown_connections 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fe7ef0 0x55e575fe8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fedb60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/3427779411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fee0a0 0x55e575fe79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/3427779411 >> 172.21.15.52:0/3427779411 conn(0x55e575f15f00 msgr2=0x55e575fe28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/3427779411 shutdown_connections 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/3427779411 wait complete. 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.656+0000 7fbfcfa80b80 1 -- start start 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 Processor -- start 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575ff3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575ff0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e575ff4920 con 0x55e575ff1420 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e575ef3810 con 0x55e575fed780 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e575ff4aa0 con 0x55e575fe7ef0 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.657+0000 7fbfcfa80b80 10 client.0 _ll_register_callbacks cb 0x55e575ed26d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfb5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575ff0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575ff3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36510/0 (socket says 172.21.15.52:36510) 2024-04-26T01:43:59.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfb5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575ff0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40616/0 (socket says 172.21.15.52:40616) 2024-04-26T01:43:59.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbe800700 1 -- 172.21.15.52:0/4188661875 learned_addr learned my addr 172.21.15.52:0/4188661875 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:59.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 -- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 msgr2=0x55e575ff0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575ff0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 -- 172.21.15.52:0/4188661875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 msgr2=0x55e575ff18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 -- 172.21.15.52:0/4188661875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e575e87880 con 0x55e575fe7ef0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbe800700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfbde00700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575ff3be0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009c870 tx=0x7fbfb00a0d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a2781dc66fd081 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.658+0000 7fbfa3e00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00ba070 con 0x55e575fe7ef0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fbfb0003680 con 0x55e575fe7ef0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb009c8e0 con 0x55e575fe7ef0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfb009cfe0 con 0x55e575fe7ef0 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfcfa80b80 10 client.24898 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:59.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e575ed0aa0 con 0x55e575fe7ef0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.659+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb009c8e0 con 0x55e575fe7ef0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fbfb009bd30 con 0x55e575fe7ef0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 10 client.24898 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 10 client.24898 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 20 client.24898 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e5760006b0 0x55e576002890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 4 client.24898 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e575f47630 con 0x55e5760006b0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.660+0000 7fbfb5400700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e5760006b0 0x55e576002890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.661+0000 7fbfb5400700 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e5760006b0 0x55e576002890 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47f9da48a8ed82e2 server_cookie=63b32d618207be7 in_seq=0 out_seq=0 2024-04-26T01:43:59.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.661+0000 7fbfa3e00700 10 client.24898 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:43:59.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfa3e00700 1 -- 172.21.15.52:0/4188661875 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e575f47630 con 0x55e5760006b0 2024-04-26T01:43:59.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfa3e00700 10 client.24898 handle_command_reply: tid=0 2024-04-26T01:43:59.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 client.24898 shutdown 2024-04-26T01:43:59.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 msgr2=0x55e575ff3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575ff3be0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009c870 tx=0x7fbfb00a0d00 comp rx=0 tx=0).stop 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 shutdown_connections 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e5760006b0 0x55e576002890 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e575ff18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfa3e00700 0 client.24898 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575ff3be0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/4188661875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575ff0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 >> 172.21.15.52:0/4188661875 conn(0x55e575f15f00 msgr2=0x55e575fe1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 shutdown_connections 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.662+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/4188661875 wait complete. 2024-04-26T01:43:59.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.663+0000 7fbfcfa80b80 20 client.24898 trim_cache size 0 max 16384 2024-04-26T01:43:59.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.685+0000 7fbfcfa80b80 1 Processor -- start 2024-04-26T01:43:59.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.685+0000 7fbfcfa80b80 1 -- start start 2024-04-26T01:43:59.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.685+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.685+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fed780 0x55e576000d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.685+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e576000bd0 con 0x55e575fed780 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e5760012a0 con 0x55e575ff1420 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e576001970 con 0x55e575fe7ef0 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbde00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40626/0 (socket says 172.21.15.52:40626) 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfb5400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fed780 0x55e576000d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbe800700 1 -- 172.21.15.52:0/2869445621 learned_addr learned my addr 172.21.15.52:0/2869445621 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbde00700 1 -- 172.21.15.52:0/2869445621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 msgr2=0x55e575ffdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbde00700 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.686+0000 7fbfbde00700 1 -- 172.21.15.52:0/2869445621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fed780 msgr2=0x55e576000d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfbde00700 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fed780 0x55e576000d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfbde00700 1 -- 172.21.15.52:0/2869445621 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e575ff3820 con 0x55e575fe7ef0 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfbe800700 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:59.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfbde00700 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe79b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009ab90 tx=0x7fbfb0098cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403693777d3dcddc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfaea00700 1 -- 172.21.15.52:0/2869445621 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00ba070 con 0x55e575fe7ef0 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfaea00700 1 -- 172.21.15.52:0/2869445621 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00d0410 con 0x55e575fe7ef0 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.687+0000 7fbfaea00700 1 -- 172.21.15.52:0/2869445621 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb009bd30 con 0x55e575fe7ef0 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/2869445621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 msgr2=0x55e575fe79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe79b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbfb009ab90 tx=0x7fbfb0098cb0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/2869445621 shutdown_connections 2024-04-26T01:43:59.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575ff1420 0x55e575ffdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe79b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/2869445621 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575fed780 0x55e576000d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/2869445621 >> 172.21.15.52:0/2869445621 conn(0x55e575f15f00 msgr2=0x55e575ff0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/2869445621 shutdown_connections 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/2869445621 wait complete. 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.688+0000 7fbfcfa80b80 1 -- start start 2024-04-26T01:43:59.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 Processor -- start 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fe2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e576002710 con 0x55e575ff1420 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e576002040 con 0x55e575fed780 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e576002de0 con 0x55e575fe7ef0 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.689+0000 7fbfcfa80b80 10 client.0 _ll_register_callbacks cb 0x55e575ed26d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36526/0 (socket says 172.21.15.52:36526) 2024-04-26T01:43:59.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 -- 172.21.15.52:0/903615984 learned_addr learned my addr 172.21.15.52:0/903615984 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfb5400700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fe2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbde00700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 -- 172.21.15.52:0/903615984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 msgr2=0x55e575fe1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 -- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 msgr2=0x55e575fe2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fe2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 -- 172.21.15.52:0/903615984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e57600a800 con 0x55e575ff1420 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfb5400700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fe2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfbe800700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009750 tx=0x7fbfb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58312a8ea6fedb66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.690+0000 7fbfa3e00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb802a030 con 0x55e575ff1420 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fbfb8003ad0 con 0x55e575ff1420 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb00d0410 con 0x55e575ff1420 2024-04-26T01:43:59.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfb8003d10 con 0x55e575ff1420 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfcfa80b80 10 client.14799 fetch_fsmap learned FSMap version 24 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e576068d10 con 0x55e575ff1420 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.691+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00d0410 con 0x55e575ff1420 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fbfb800f6d0 con 0x55e575ff1420 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 10 client.14799 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 10 client.14799 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 20 client.14799 populate_metadata read hostname 'smithi052' 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e57606d8c0 0x55e57606c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 4 client.14799 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:43:59.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "status"}) -- 0x55e575f416d0 con 0x55e57606d8c0 2024-04-26T01:43:59.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.692+0000 7fbfbde00700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e57606d8c0 0x55e57606c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:43:59.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfbde00700 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e57606d8c0 0x55e57606c5d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31da115d0014538 server_cookie=da610cf949b4dd1d in_seq=0 out_seq=0 2024-04-26T01:43:59.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfa3e00700 10 client.14799 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfa3e00700 1 -- 172.21.15.52:0/903615984 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e575f416d0 con 0x55e57606d8c0 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfa3e00700 10 client.14799 handle_command_reply: tid=0 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 client.14799 shutdown 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 msgr2=0x55e57600a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009750 tx=0x7fbfb80057f0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 shutdown_connections 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e57606d8c0 0x55e57606c5d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e575ff1420 0x55e57600a2c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e575fe7ef0 0x55e575fe1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 --2- 172.21.15.52:0/903615984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e575fed780 0x55e575fe2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfa3e00700 0 client.14799 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 >> 172.21.15.52:0/903615984 conn(0x55e575f15f00 msgr2=0x55e575fee490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 shutdown_connections 2024-04-26T01:43:59.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.693+0000 7fbfcfa80b80 1 -- 172.21.15.52:0/903615984 wait complete. 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfcfa80b80 20 client.14799 trim_cache size 0 max 16384 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24524, 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 172.39427398999999, 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:43:59.694 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 193.76480458899999 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 msgr2=0x7fbfa40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 0x7fbfa40a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbfc01087a0 tx=0x7fbfa800b040 comp rx=0 tx=0).stop 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 msgr2=0x7fbfc019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:43:59.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000eb30 tx=0x7fbfb000ee40 comp rx=0 tx=0).stop 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 shutdown_connections 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbfa40a3cc0 0x7fbfa40a6180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfc010e3b0 0x7fbfc01a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbfc01092d0 0x7fbfc019f6d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 --2- 172.21.15.52:0/1256874276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbfc0109c80 0x7fbfc019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.694+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 >> 172.21.15.52:0/1256874276 conn(0x7fbfc0076210 msgr2=0x7fbfc0076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.695+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 shutdown_connections 2024-04-26T01:43:59.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:43:59.695+0000 7fbfc5000700 1 -- 172.21.15.52:0/1256874276 wait complete. 2024-04-26T01:44:00.525 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T01:44:00.525 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:44:00.535 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:44:00.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T01:44:01.002 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:01.319 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:01 smithi052 ceph-mon[27851]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:01.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:01 smithi191 ceph-mon[32745]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:01.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:01 smithi183 ceph-mon[32752]: pgmap v421: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:03.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 -- 172.21.15.52:0/2081288115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c10f080 msgr2=0x7fe03c111470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 --2- 172.21.15.52:0/2081288115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c10f080 0x7fe03c111470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe03400b0a0 tx=0x7fe03400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 -- 172.21.15.52:0/2081288115 shutdown_connections 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 --2- 172.21.15.52:0/2081288115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c10f080 0x7fe03c111470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 --2- 172.21.15.52:0/2081288115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 --2- 172.21.15.52:0/2081288115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c0754a0 0x7fe03c075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.093+0000 7fe040e00700 1 -- 172.21.15.52:0/2081288115 >> 172.21.15.52:0/2081288115 conn(0x7fe03c0fe040 msgr2=0x7fe03c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.094+0000 7fe040e00700 1 -- 172.21.15.52:0/2081288115 shutdown_connections 2024-04-26T01:44:03.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.094+0000 7fe040e00700 1 -- 172.21.15.52:0/2081288115 wait complete. 2024-04-26T01:44:03.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.095+0000 7fe040e00700 1 Processor -- start 2024-04-26T01:44:03.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 -- start start 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 0x7fe03c1a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c114ba0 con 0x7fe03c10f080 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c114a20 con 0x7fe03c076a70 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.096+0000 7fe040e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c114d20 con 0x7fe03c0754a0 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe03a800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 0x7fe03c1a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44970/0 (socket says 172.21.15.52:44970) 2024-04-26T01:44:03.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039e00700 1 -- 172.21.15.52:0/2537544923 learned_addr learned my addr 172.21.15.52:0/2537544923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 -- 172.21.15.52:0/2537544923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 msgr2=0x7fe03c19f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 -- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 msgr2=0x7fe03c1a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 0x7fe03c1a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039400700 1 -- 172.21.15.52:0/2537544923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c1a6e90 con 0x7fe03c076a70 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe03a800700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 0x7fe03c1a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.097+0000 7fe039e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe039400700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c19f770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe02800e9c0 tx=0x7fe02800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028004220 con 0x7fe03c076a70 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0280043c0 con 0x7fe03c076a70 2024-04-26T01:44:03.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028003c30 con 0x7fe03c076a70 2024-04-26T01:44:03.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03c1a7180 con 0x7fe03c076a70 2024-04-26T01:44:03.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.098+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03c1a7710 con 0x7fe03c076a70 2024-04-26T01:44:03.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.100+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe03c069730 con 0x7fe03c076a70 2024-04-26T01:44:03.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.101+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe02800c7c0 con 0x7fe03c076a70 2024-04-26T01:44:03.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.102+0000 7fe032a00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 0x7fe0200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:03.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.103+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fe0280d05a0 con 0x7fe03c076a70 2024-04-26T01:44:03.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.103+0000 7fe039e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 0x7fe0200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:03.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.103+0000 7fe039e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 0x7fe0200a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe02c011550 tx=0x7fe02c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:03.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.107+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe02809cf30 con 0x7fe03c076a70 2024-04-26T01:44:03.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:03 smithi052 ceph-mon[27851]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:03.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.359+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fe03c062e80 con 0x7fe0200a3cc0 2024-04-26T01:44:03.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.361+0000 7fe032a00700 1 -- 172.21.15.52:0/2537544923 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fe03c062e80 con 0x7fe0200a3cc0 2024-04-26T01:44:03.359 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: cephadm 2024-04-26T01:44:03.359 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2024-04-26T01:44:03.359 INFO:teuthology.orchestra.run.smithi052.stdout:Paused: No 2024-04-26T01:44:03.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 msgr2=0x7fe0200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 0x7fe0200a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe02c011550 tx=0x7fe02c009260 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 msgr2=0x7fe03c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c19f770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe02800e9c0 tx=0x7fe02800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 shutdown_connections 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe0200a3cc0 0x7fe0200a6180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe03c10f080 0x7fe03c1a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe03c0754a0 0x7fe03c19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 --2- 172.21.15.52:0/2537544923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe03c076a70 0x7fe03c19f770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 >> 172.21.15.52:0/2537544923 conn(0x7fe03c0fe040 msgr2=0x7fe03c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 shutdown_connections 2024-04-26T01:44:03.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:03.365+0000 7fe040e00700 1 -- 172.21.15.52:0/2537544923 wait complete. 2024-04-26T01:44:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:03 smithi191 ceph-mon[32745]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:03 smithi183 ceph-mon[32752]: pgmap v422: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:04.219 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T01:44:04.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:04 smithi191 ceph-mon[32745]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:04 smithi183 ceph-mon[32752]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:04.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:04 smithi052 ceph-mon[27851]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:04.664 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:05.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:05 smithi052 ceph-mon[27851]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:05 smithi191 ceph-mon[32745]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:05 smithi183 ceph-mon[32752]: pgmap v423: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 -- 172.21.15.52:0/3549106070 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978105ee0 msgr2=0x7f5978106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 --2- 172.21.15.52:0/3549106070 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978105ee0 0x7f5978106340 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f59640099c0 tx=0x7f5964009cd0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 -- 172.21.15.52:0/3549106070 shutdown_connections 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 --2- 172.21.15.52:0/3549106070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978106880 0x7f597810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 --2- 172.21.15.52:0/3549106070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978104ce0 0x7f5978105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 --2- 172.21.15.52:0/3549106070 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978105ee0 0x7f5978106340 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 -- 172.21.15.52:0/3549106070 >> 172.21.15.52:0/3549106070 conn(0x7f59781002a0 msgr2=0x7f59781026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 -- 172.21.15.52:0/3549106070 shutdown_connections 2024-04-26T01:44:06.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.875+0000 7f597ea00700 1 -- 172.21.15.52:0/3549106070 wait complete. 2024-04-26T01:44:06.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.876+0000 7f597ea00700 1 Processor -- start 2024-04-26T01:44:06.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.876+0000 7f597ea00700 1 -- start start 2024-04-26T01:44:06.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978106880 0x7f59781a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f59781107a0 con 0x7f5978106880 2024-04-26T01:44:06.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5978110620 con 0x7f5978105ee0 2024-04-26T01:44:06.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ea00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5978110920 con 0x7f5978104ce0 2024-04-26T01:44:06.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f5977400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f5977e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f5977400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38700/0 (socket says 172.21.15.52:38700) 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f5977400700 1 -- 172.21.15.52:0/4154356462 learned_addr learned my addr 172.21.15.52:0/4154356462 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f5977e00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57974/0 (socket says 172.21.15.52:57974) 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.877+0000 7f597ca00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978106880 0x7f59781a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 -- 172.21.15.52:0/4154356462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 msgr2=0x7f597819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 -- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978106880 msgr2=0x7f59781a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978106880 0x7f59781a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 -- 172.21.15.52:0/4154356462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59781a2a40 con 0x7f5978105ee0 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977e00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:06.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5977400700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f59640036f0 tx=0x7f5964003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964009e60 con 0x7f5978105ee0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596400be50 con 0x7f5978105ee0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59781a2cd0 con 0x7f5978105ee0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.878+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596400fda0 con 0x7f5978105ee0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.879+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59781a3120 con 0x7f5978105ee0 2024-04-26T01:44:06.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.880+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5978069730 con 0x7f5978105ee0 2024-04-26T01:44:06.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.882+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f596400f460 con 0x7f5978105ee0 2024-04-26T01:44:06.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.883+0000 7f5974c00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 0x7f595c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:06.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.884+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f59640d1300 con 0x7f5978105ee0 2024-04-26T01:44:06.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.884+0000 7f5977e00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 0x7f595c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:06.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.884+0000 7f5977e00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 0x7f595c0a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5978105d40 tx=0x7f5968009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:06.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:06.885+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5964013070 con 0x7f5978105ee0 2024-04-26T01:44:07.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.123+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5978062e80 con 0x7f595c0a3c40 2024-04-26T01:44:07.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.163+0000 7f5974c00700 1 -- 172.21.15.52:0/4154356462 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f5978062e80 con 0x7f595c0a3c40 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.a smithi052 running (3m) 42s ago 3m 18.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b6df1cc916a6 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.b smithi183 running (3m) 62s ago 3m 28.5M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cef119fcec4f 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.c smithi191 running (3m) 61s ago 3m 21.5M - 19.0.0-3156-gd69f13d8 da4b004c09e8 f0ff8f7696e6 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.d smithi052 running (3m) 42s ago 3m 13.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 541dde23f7cf 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.e smithi183 running (3m) 62s ago 3m 13.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 8c567928518c 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.f smithi191 running (3m) 61s ago 3m 21.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2757e8f5f1bc 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.g smithi052 running (3m) 42s ago 3m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 3195b405ccc9 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.h smithi183 running (3m) 62s ago 3m 20.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 afe162391a93 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.i smithi191 running (3m) 61s ago 3m 19.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 03e3269b72bf 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.j smithi052 running (3m) 42s ago 3m 13.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 67b9a82133ea 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.k smithi183 running (3m) 62s ago 3m 15.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b67a73380ba3 2024-04-26T01:44:07.162 INFO:teuthology.orchestra.run.smithi052.stdout:mds.l smithi191 running (4m) 61s ago 4m 16.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 dbad05a3ba9c 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.x smithi052 *:9283,8765 running (12m) 42s ago 12m 517M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cce5110b139c 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.y smithi183 *:8443,8765 running (9m) 62s ago 9m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 25eeb8bc80e7 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.z smithi191 *:8443,8765 running (10m) 61s ago 10m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 071cf2d73ef6 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (12m) 42s ago 12m 51.0M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 d282b1058cde 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b smithi183 running (10m) 62s ago 10m 41.6M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 00c660ae7e9e 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c smithi191 running (10m) 61s ago 10m 41.9M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 7dfd944d208e 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (9m) 42s ago 9m 49.4M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 3020e08b797e 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi183 running (9m) 62s ago 9m 57.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 49ea6141af81 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi191 running (8m) 61s ago 8m 60.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 5aa3835bcce0 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi052 running (8m) 42s ago 8m 52.9M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 24a9718409f0 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 smithi183 running (7m) 62s ago 7m 54.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 7f5c8f6fe7ff 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 smithi191 running (7m) 61s ago 7m 58.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 76dab11e4986 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6 smithi052 running (6m) 42s ago 6m 45.4M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 4ebad74a2cfd 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7 smithi183 running (6m) 62s ago 6m 48.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 3e1e71c8e990 2024-04-26T01:44:07.163 INFO:teuthology.orchestra.run.smithi052.stdout:osd.8 smithi191 running (5m) 61s ago 5m 48.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 574e917e6317 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stdout:osd.9 smithi052 running (5m) 42s ago 5m 51.4M 4273M 19.0.0-3156-gd69f13d8 da4b004c09e8 6a5fb73e59c7 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stdout:osd.10 smithi183 running (4m) 62s ago 4m 54.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 93db56c9974a 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stdout:osd.11 smithi191 running (4m) 61s ago 4m 51.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 fbf7f49b39c6 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 msgr2=0x7f595c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 0x7f595c0a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5978105d40 tx=0x7f5968009260 comp rx=0 tx=0).stop 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 msgr2=0x7f597819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:07.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f59640036f0 tx=0x7f5964003e60 comp rx=0 tx=0).stop 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 shutdown_connections 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f595c0a3c40 0x7f595c0a6100 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5978106880 0x7f59781a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5978104ce0 0x7f597819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 --2- 172.21.15.52:0/4154356462 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5978105ee0 0x7f597819b320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 >> 172.21.15.52:0/4154356462 conn(0x7f59781002a0 msgr2=0x7f5978109260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 shutdown_connections 2024-04-26T01:44:07.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:07.167+0000 7f597ea00700 1 -- 172.21.15.52:0/4154356462 wait complete. 2024-04-26T01:44:07.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:07 smithi191 ceph-mon[32745]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:07 smithi183 ceph-mon[32752]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:07 smithi052 ceph-mon[27851]: pgmap v424: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:08.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T01:44:08.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:08 smithi191 ceph-mon[32745]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:08.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:08 smithi183 ceph-mon[32752]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:08 smithi052 ceph-mon[27851]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:08.508 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:09.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:09 smithi052 ceph-mon[27851]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:09 smithi191 ceph-mon[32745]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:09.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:09 smithi183 ceph-mon[32752]: pgmap v425: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.738+0000 7f5cbca00700 1 -- 172.21.15.52:0/2954674776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 msgr2=0x7f5cb8074e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.738+0000 7f5cbca00700 1 --2- 172.21.15.52:0/2954674776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb8074e70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009990 tx=0x7f5ca8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 -- 172.21.15.52:0/2954674776 shutdown_connections 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 --2- 172.21.15.52:0/2954674776 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 0x7f5cb810ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 --2- 172.21.15.52:0/2954674776 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb8074e70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 --2- 172.21.15.52:0/2954674776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb80758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 -- 172.21.15.52:0/2954674776 >> 172.21.15.52:0/2954674776 conn(0x7f5cb80fe090 msgr2=0x7f5cb81004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:10.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 -- 172.21.15.52:0/2954674776 shutdown_connections 2024-04-26T01:44:10.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.739+0000 7f5cbca00700 1 -- 172.21.15.52:0/2954674776 wait complete. 2024-04-26T01:44:10.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.740+0000 7f5cbca00700 1 Processor -- start 2024-04-26T01:44:10.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 -- start start 2024-04-26T01:44:10.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:10.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 0x7f5cb81a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb81107b0 con 0x7f5cb8075440 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8110630 con 0x7f5cb8076a10 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cbca00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8110930 con 0x7f5cb810a9d0 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb5e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb5400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38718/0 (socket says 172.21.15.52:38718) 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb5e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60078/0 (socket says 172.21.15.52:60078) 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb5400700 1 -- 172.21.15.52:0/3842619384 learned_addr learned my addr 172.21.15.52:0/3842619384 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.741+0000 7f5cb6800700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 0x7f5cb81a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 -- 172.21.15.52:0/3842619384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 msgr2=0x7f5cb81a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 0x7f5cb81a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 -- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 msgr2=0x7f5cb819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 -- 172.21.15.52:0/3842619384 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb81a2a30 con 0x7f5cb8076a10 2024-04-26T01:44:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cb5400700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400e9a0 tx=0x7f5ca400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400ca10 con 0x7f5cb8076a10 2024-04-26T01:44:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ca400cbb0 con 0x7f5cb8076a10 2024-04-26T01:44:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.742+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb81a2d20 con 0x7f5cb8076a10 2024-04-26T01:44:10.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.743+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4018680 con 0x7f5cb8076a10 2024-04-26T01:44:10.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.743+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb81a3250 con 0x7f5cb8076a10 2024-04-26T01:44:10.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.744+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb8069730 con 0x7f5cb8076a10 2024-04-26T01:44:10.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.746+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5ca4010430 con 0x7f5cb8076a10 2024-04-26T01:44:10.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.747+0000 7f5caea00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 0x7f5c9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:10.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.747+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f5ca40d0500 con 0x7f5cb8076a10 2024-04-26T01:44:10.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.748+0000 7f5cb5e00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 0x7f5c9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:10.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.748+0000 7f5cb5e00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 0x7f5c9c0a6180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800b2e0 tx=0x7f5ca8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:10.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.750+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ca409ce90 con 0x7f5cb8076a10 2024-04-26T01:44:10.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.988+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f5cb8062e80 con 0x7f5c9c0a3cc0 2024-04-26T01:44:10.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:10.998+0000 7f5caea00700 1 -- 172.21.15.52:0/3842619384 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f5cb8062e80 con 0x7f5c9c0a3cc0 2024-04-26T01:44:10.996 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T01:44:10.996 INFO:teuthology.orchestra.run.smithi052.stdout:mds.all 12/12 65s ago 4m smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi183=b;smithi183=e;smithi183=h;smithi183=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-04-26T01:44:10.996 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 3/3 65s ago 10m smithi052=x;smithi183=y;smithi191=z;count:3 2024-04-26T01:44:10.996 INFO:teuthology.orchestra.run.smithi052.stdout:mon 3/3 65s ago 10m smithi052:172.21.15.52=a;smithi183:172.21.15.183=b;smithi191:172.21.15.191=c;count:3 2024-04-26T01:44:10.996 INFO:teuthology.orchestra.run.smithi052.stdout:osd 12 65s ago - 2024-04-26T01:44:10.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.001+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 msgr2=0x7f5c9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:10.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.001+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 0x7f5c9c0a6180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800b2e0 tx=0x7f5ca8005fb0 comp rx=0 tx=0).stop 2024-04-26T01:44:10.999 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.001+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 msgr2=0x7f5cb819b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.001+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400e9a0 tx=0x7f5ca400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.001+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 shutdown_connections 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f5c9c0a3cc0 0x7f5c9c0a6180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5cb810a9d0 0x7f5cb81a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5cb8075440 0x7f5cb819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 --2- 172.21.15.52:0/3842619384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb8076a10 0x7f5cb819b310 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 >> 172.21.15.52:0/3842619384 conn(0x7f5cb80fe090 msgr2=0x7f5cb8106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 shutdown_connections 2024-04-26T01:44:11.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:11.002+0000 7f5cbca00700 1 -- 172.21.15.52:0/3842619384 wait complete. 2024-04-26T01:44:11.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:11 smithi052 ceph-mon[27851]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:11.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:11 smithi191 ceph-mon[32745]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:11 smithi183 ceph-mon[32752]: pgmap v426: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:11.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T01:44:12.308 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:12.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:12 smithi191 ceph-mon[32745]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:12.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:12 smithi183 ceph-mon[32752]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:12.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:12 smithi052 ceph-mon[27851]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:13.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:13 smithi052 ceph-mon[27851]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:13.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:13 smithi191 ceph-mon[32745]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:13.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:13 smithi183 ceph-mon[32752]: pgmap v427: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 -- 172.21.15.52:0/3986261803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 msgr2=0x7f10f0111420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3986261803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f0111420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b0d0 tx=0x7f10e800b3e0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 -- 172.21.15.52:0/3986261803 shutdown_connections 2024-04-26T01:44:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3986261803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f0111420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3986261803 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f01093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3986261803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f010a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 -- 172.21.15.52:0/3986261803 >> 172.21.15.52:0/3986261803 conn(0x7f10f0077fa0 msgr2=0x7f10f007a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 -- 172.21.15.52:0/3986261803 shutdown_connections 2024-04-26T01:44:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.530+0000 7f10f6e00700 1 -- 172.21.15.52:0/3986261803 wait complete. 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.531+0000 7f10f6e00700 1 Processor -- start 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 -- start start 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0114aa0 con 0x7f10f010ab70 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0114920 con 0x7f10f0108fd0 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0114c20 con 0x7f10f010a1d0 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f6400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57840/0 (socket says 172.21.15.52:57840) 2024-04-26T01:44:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 -- 172.21.15.52:0/3646235095 learned_addr learned my addr 172.21.15.52:0/3646235095 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f5000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.532+0000 7f10f5a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 -- 172.21.15.52:0/3646235095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 msgr2=0x7f10f019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 -- 172.21.15.52:0/3646235095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 msgr2=0x7f10f019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 -- 172.21.15.52:0/3646235095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f01a6cb0 con 0x7f10f010ab70 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f5000700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f5a00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.533+0000 7f10f6400700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10e8004cb0 tx=0x7f10e8004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.534+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e801f030 con 0x7f10f010ab70 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.534+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e8005040 con 0x7f10f010ab70 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.534+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e801b690 con 0x7f10f010ab70 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.534+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f01a6f40 con 0x7f10f010ab70 2024-04-26T01:44:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.534+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f01a7340 con 0x7f10f010ab70 2024-04-26T01:44:14.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.535+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f0069730 con 0x7f10f010ab70 2024-04-26T01:44:14.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.537+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f10e8005480 con 0x7f10f010ab70 2024-04-26T01:44:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.538+0000 7f10ee000700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 0x7f10d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.539+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f10e800f020 con 0x7f10f010ab70 2024-04-26T01:44:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.539+0000 7f10f5a00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 0x7f10d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.539+0000 7f10f5a00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 0x7f10d80a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10f010a030 tx=0x7f10e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:14.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.543+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10e8029020 con 0x7f10f010ab70 2024-04-26T01:44:14.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.778+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f10f0062e80 con 0x7f10d80a3cc0 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.781+0000 7f10ee000700 1 -- 172.21.15.52:0/3646235095 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f10f0062e80 con 0x7f10d80a3cc0 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stdout:smithi183 172.21.15.183 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 172.21.15.191 2024-04-26T01:44:14.779 INFO:teuthology.orchestra.run.smithi052.stdout:3 hosts in cluster 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 msgr2=0x7f10d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 0x7f10d80a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10f010a030 tx=0x7f10e4009260 comp rx=0 tx=0).stop 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 msgr2=0x7f10f01a6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10e8004cb0 tx=0x7f10e8004ce0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 shutdown_connections 2024-04-26T01:44:14.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f10d80a3cc0 0x7f10d80a6180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10f010ab70 0x7f10f01a6610 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10f0108fd0 0x7f10f019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 --2- 172.21.15.52:0/3646235095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10f010a1d0 0x7f10f019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.784+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 >> 172.21.15.52:0/3646235095 conn(0x7f10f0077fa0 msgr2=0x7f10f010d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.785+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 shutdown_connections 2024-04-26T01:44:14.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:14.785+0000 7f10f6e00700 1 -- 172.21.15.52:0/3646235095 wait complete. 2024-04-26T01:44:15.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:15 smithi191 ceph-mon[32745]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:15 smithi183 ceph-mon[32752]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:15.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:15 smithi052 ceph-mon[27851]: pgmap v428: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:15.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T01:44:16.148 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:16.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:16 smithi191 ceph-mon[32745]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:16.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:16 smithi183 ceph-mon[32752]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:16.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:16 smithi052 ceph-mon[27851]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:17.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:17 smithi191 ceph-mon[32745]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:17 smithi183 ceph-mon[32752]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:17 smithi052 ceph-mon[27851]: pgmap v429: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 -- 172.21.15.52:0/590451395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 msgr2=0x7fa30c105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 --2- 172.21.15.52:0/590451395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c105850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa300009990 tx=0x7fa300009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 -- 172.21.15.52:0/590451395 shutdown_connections 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 --2- 172.21.15.52:0/590451395 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 0x7fa30c10ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 --2- 172.21.15.52:0/590451395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c105850 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 --2- 172.21.15.52:0/590451395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 -- 172.21.15.52:0/590451395 >> 172.21.15.52:0/590451395 conn(0x7fa30c0fce50 msgr2=0x7fa30c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 -- 172.21.15.52:0/590451395 shutdown_connections 2024-04-26T01:44:18.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.402+0000 7fa312e00700 1 -- 172.21.15.52:0/590451395 wait complete. 2024-04-26T01:44:18.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 Processor -- start 2024-04-26T01:44:18.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 -- start start 2024-04-26T01:44:18.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:18.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c19b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:18.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 0x7fa30c1a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:18.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c10e4a0 con 0x7fa30c103460 2024-04-26T01:44:18.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c10e320 con 0x7fa30c1086c0 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.404+0000 7fa312e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c10e620 con 0x7fa30c105d90 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57860/0 (socket says 172.21.15.52:57860) 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 -- 172.21.15.52:0/103661598 learned_addr learned my addr 172.21.15.52:0/103661598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311000700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c19b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:18.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa312400700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 0x7fa30c1a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 -- 172.21.15.52:0/103661598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 msgr2=0x7fa30c19b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c19b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 -- 172.21.15.52:0/103661598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 msgr2=0x7fa30c1a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 0x7fa30c1a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311a00700 1 -- 172.21.15.52:0/103661598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c1a2990 con 0x7fa30c103460 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.405+0000 7fa311000700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c19b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:18.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa311a00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b760 tx=0x7fa3000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300027070 con 0x7fa30c103460 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa300022470 con 0x7fa30c103460 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300017650 con 0x7fa30c103460 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c1a2c20 con 0x7fa30c103460 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.406+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c1a3020 con 0x7fa30c103460 2024-04-26T01:44:18.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.407+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30c069730 con 0x7fa30c103460 2024-04-26T01:44:18.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.409+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa30002a070 con 0x7fa30c103460 2024-04-26T01:44:18.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.411+0000 7fa30a000700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 0x7fa2f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:18.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.411+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fa3000d1160 con 0x7fa30c103460 2024-04-26T01:44:18.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.411+0000 7fa311000700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 0x7fa2f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:18.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.412+0000 7fa311000700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 0x7fa2f40a6180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8011490 tx=0x7fa2f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:18.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.416+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa300013070 con 0x7fa30c103460 2024-04-26T01:44:18.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.657+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 --> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa30c062e80 con 0x7fa2f40a3cc0 2024-04-26T01:44:18.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.659+0000 7fa30a000700 1 -- 172.21.15.52:0/103661598 <== mgr.14152 v2:172.21.15.52:6800/2415060450 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa30c062e80 con 0x7fa2f40a3cc0 2024-04-26T01:44:18.657 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T01:44:18.657 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-04-26T01:44:18.657 INFO:teuthology.orchestra.run.smithi052.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T01:44:18.657 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T01:44:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 msgr2=0x7fa2f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 0x7fa2f40a6180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8011490 tx=0x7fa2f8009260 comp rx=0 tx=0).stop 2024-04-26T01:44:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 msgr2=0x7fa30c19ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b760 tx=0x7fa3000047b0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 shutdown_connections 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fa2f40a3cc0 0x7fa2f40a6180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa30c1086c0 0x7fa30c1a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa30c103460 0x7fa30c19ad40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 --2- 172.21.15.52:0/103661598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa30c105d90 0x7fa30c19b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.662+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 >> 172.21.15.52:0/103661598 conn(0x7fa30c0fce50 msgr2=0x7fa30c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.663+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 shutdown_connections 2024-04-26T01:44:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:18.663+0000 7fa312e00700 1 -- 172.21.15.52:0/103661598 wait complete. 2024-04-26T01:44:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:19 smithi191 ceph-mon[32745]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:44:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:19 smithi183 ceph-mon[32752]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:44:19.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:19 smithi052 ceph-mon[27851]: pgmap v430: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T01:44:19.486 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T01:44:19.496 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:44:19.496 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T01:44:19.926 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:20.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:20 smithi052 ceph-mon[27851]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:20.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:20 smithi191 ceph-mon[32745]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:20.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:20 smithi183 ceph-mon[32752]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:44:21.328 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:21 smithi052 ceph-mon[27851]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:21 smithi191 ceph-mon[32745]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:21 smithi183 ceph-mon[32752]: pgmap v431: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T01:44:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 -- 172.21.15.52:0/2832017483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc104bc0 msgr2=0x7fb3cc104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2832017483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc104fe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009990 tx=0x7fb3c0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 -- 172.21.15.52:0/2832017483 shutdown_connections 2024-04-26T01:44:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2832017483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc106760 0x7fb3cc10d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2832017483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc104fe0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2832017483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 -- 172.21.15.52:0/2832017483 >> 172.21.15.52:0/2832017483 conn(0x7fb3cc100160 msgr2=0x7fb3cc1025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:22.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 -- 172.21.15.52:0/2832017483 shutdown_connections 2024-04-26T01:44:22.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.152+0000 7fb3d2000700 1 -- 172.21.15.52:0/2832017483 wait complete. 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.153+0000 7fb3d2000700 1 Processor -- start 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 -- start start 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 0x7fb3cc1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc1106a0 con 0x7fb3cc105dc0 2024-04-26T01:44:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc110520 con 0x7fb3cc104bc0 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d2000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc110820 con 0x7fb3cc106760 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d0c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3cbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d0c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37040/0 (socket says 172.21.15.52:37040) 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d0c00700 1 -- 172.21.15.52:0/2942272080 learned_addr learned my addr 172.21.15.52:0/2942272080 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.154+0000 7fb3d1600700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 0x7fb3cc1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 -- 172.21.15.52:0/2942272080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 msgr2=0x7fb3cc1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 0x7fb3cc1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 -- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 msgr2=0x7fb3cc19b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc19b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 -- 172.21.15.52:0/2942272080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc1a29e0 con 0x7fb3cc104bc0 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d1600700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 0x7fb3cc1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:22.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3cbe00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3d0c00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000b850 tx=0x7fb3c0009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.155+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0026070 con 0x7fb3cc104bc0 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.156+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc1a2c70 con 0x7fb3cc104bc0 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.156+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c0021410 con 0x7fb3cc104bc0 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.156+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0017650 con 0x7fb3cc104bc0 2024-04-26T01:44:22.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.156+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc1a3120 con 0x7fb3cc104bc0 2024-04-26T01:44:22.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.157+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc069730 con 0x7fb3cc104bc0 2024-04-26T01:44:22.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.159+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb3c0029070 con 0x7fb3cc104bc0 2024-04-26T01:44:22.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.160+0000 7fb3c9600700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 0x7fb3b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:22.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.160+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fb3c00d13e0 con 0x7fb3cc104bc0 2024-04-26T01:44:22.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.161+0000 7fb3cbe00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 0x7fb3b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:22.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.161+0000 7fb3cbe00700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 0x7fb3b40a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8011490 tx=0x7fb3b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:22.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.163+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3c0013070 con 0x7fb3cc104bc0 2024-04-26T01:44:22.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.445+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fb3cc02df80 con 0x7fb3cc104bc0 2024-04-26T01:44:22.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.446+0000 7fb3c9600700 1 -- 172.21.15.52:0/2942272080 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3058 (secure 0 0 0) 0x7fb3c009de00 con 0x7fb3cc104bc0 2024-04-26T01:44:22.445 INFO:teuthology.orchestra.run.smithi052.stdout:e24 2024-04-26T01:44:22.445 INFO:teuthology.orchestra.run.smithi052.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T01:44:22.445 INFO:teuthology.orchestra.run.smithi052.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:44:22.445 INFO:teuthology.orchestra.run.smithi052.stdout:legacy client fscid: 1 2024-04-26T01:44:22.445 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:Filesystem 'cephfs' (1) 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:fs_name cephfs 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 24 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-04-26T01:41:06.424614+0000 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2024-04-26T01:41:31.153156+0000 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:tableserver 0 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:root 0 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:session_timeout 300 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:session_autoclose 300 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:max_file_size 1099511627776 2024-04-26T01:44:22.446 INFO:teuthology.orchestra.run.smithi052.stdout:max_xattr_size 65536 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:required_client_features {} 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure 0 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure_osd_epoch 0 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:max_mds 3 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:in 0,1,2 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:up {0=24524,1=24512,2=24496} 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:failed 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:damaged 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:stopped 2024-04-26T01:44:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:data_pools [3] 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:metadata_pool 2 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:inline_data disabled 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:balancer 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:bal_rank_mask -1 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:standby_count_wanted 0 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:qdb_cluster leader: 24524 members: 24512,24496,24524 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.b{0:24524} state up:active seq 7 addr [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.h{0:24494} state up:standby-replay seq 1 addr [v2:172.21.15.183:6836/1263803776,v1:172.21.15.183:6837/1263803776] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.a{1:24512} state up:active seq 12 addr [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.c{1:24478} state up:standby-replay seq 1 addr [v2:172.21.15.191:6836/2929119914,v1:172.21.15.191:6837/2929119914] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.448 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.i{2:24496} state up:active seq 14 addr [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.f{2:24472} state up:standby-replay seq 1 addr [v2:172.21.15.191:6834/234906181,v1:172.21.15.191:6835/234906181] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:Standby daemons: 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.j{-1:14361} state up:standby seq 1 addr [v2:172.21.15.52:6834/1730690276,v1:172.21.15.52:6835/1730690276] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.d{-1:14373} state up:standby seq 1 addr [v2:172.21.15.52:6836/2681513710,v1:172.21.15.52:6837/2681513710] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.g{-1:14397} state up:standby seq 1 addr [v2:172.21.15.52:6838/3379340628,v1:172.21.15.52:6839/3379340628] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.l{-1:24433} state up:standby seq 1 addr [v2:172.21.15.191:6832/4135915621,v1:172.21.15.191:6833/4135915621] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.183:6832/2084061199,v1:172.21.15.183:6833/2084061199] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.449 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.e{-1:24470} state up:standby seq 1 addr [v2:172.21.15.183:6834/2708532468,v1:172.21.15.183:6835/2708532468] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T01:44:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 msgr2=0x7fb3b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 0x7fb3b40a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8011490 tx=0x7fb3b8009260 comp rx=0 tx=0).stop 2024-04-26T01:44:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 msgr2=0x7fb3cc19ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000b850 tx=0x7fb3c0009580 comp rx=0 tx=0).stop 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 shutdown_connections 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fb3b40a3cc0 0x7fb3b40a6180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3cc106760 0x7fb3cc1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3cc104bc0 0x7fb3cc19ad90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 --2- 172.21.15.52:0/2942272080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3cc105dc0 0x7fb3cc19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.450+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 >> 172.21.15.52:0/2942272080 conn(0x7fb3cc100160 msgr2=0x7fb3cc1090e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.451+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 shutdown_connections 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:22.451+0000 7fb3d2000700 1 -- 172.21.15.52:0/2942272080 wait complete. 2024-04-26T01:44:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:44:23.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T01:44:23.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:23 smithi191 ceph-mon[32745]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:44:23.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:23 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2942272080' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:44:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:23 smithi183 ceph-mon[32752]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:44:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:23 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2942272080' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:44:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:23 smithi052 ceph-mon[27851]: pgmap v432: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T01:44:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:23 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2942272080' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T01:44:23.681 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:25.382 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:25 smithi052 ceph-mon[27851]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:25.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:25 smithi191 ceph-mon[32745]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:25.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:25 smithi183 ceph-mon[32752]: pgmap v433: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 -- 172.21.15.52:0/1988025188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 msgr2=0x7f56f0074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 --2- 172.21.15.52:0/1988025188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f0074ed0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009990 tx=0x7f56e4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 -- 172.21.15.52:0/1988025188 shutdown_connections 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 --2- 172.21.15.52:0/1988025188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 0x7f56f0111390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 --2- 172.21.15.52:0/1988025188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f0074ed0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 --2- 172.21.15.52:0/1988025188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 0x7f56f0075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 -- 172.21.15.52:0/1988025188 >> 172.21.15.52:0/1988025188 conn(0x7f56f00fdf80 msgr2=0x7f56f01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:25.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 -- 172.21.15.52:0/1988025188 shutdown_connections 2024-04-26T01:44:25.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.882+0000 7f56f6800700 1 -- 172.21.15.52:0/1988025188 wait complete. 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.883+0000 7f56f6800700 1 Processor -- start 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 -- start start 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 0x7f56f019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 0x7f56f01a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f56f0114a30 con 0x7f56f0076a70 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f56f01148b0 con 0x7f56f010efa0 2024-04-26T01:44:25.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f6800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f56f0114bb0 con 0x7f56f00754a0 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f5400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 0x7f56f019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f4a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42430/0 (socket says 172.21.15.52:42430) 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f4a00700 1 -- 172.21.15.52:0/2575629310 learned_addr learned my addr 172.21.15.52:0/2575629310 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.884+0000 7f56f5e00700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 0x7f56f01a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 -- 172.21.15.52:0/2575629310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 msgr2=0x7f56f019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 0x7f56f019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 -- 172.21.15.52:0/2575629310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 msgr2=0x7f56f01a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 0x7f56f01a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:25.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 -- 172.21.15.52:0/2575629310 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f01a6d10 con 0x7f56f0076a70 2024-04-26T01:44:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56f4a00700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00e9a0 tx=0x7f56dc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc004220 con 0x7f56f0076a70 2024-04-26T01:44:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.885+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56dc0043c0 con 0x7f56f0076a70 2024-04-26T01:44:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.886+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f01a6fa0 con 0x7f56f0076a70 2024-04-26T01:44:25.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.886+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc003cc0 con 0x7f56f0076a70 2024-04-26T01:44:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.886+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f01a7500 con 0x7f56f0076a70 2024-04-26T01:44:25.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.887+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f0069730 con 0x7f56f0076a70 2024-04-26T01:44:25.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.889+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f56dc00c810 con 0x7f56f0076a70 2024-04-26T01:44:25.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.890+0000 7f56ee000700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 0x7f56d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:25.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.890+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f56dc0d0550 con 0x7f56f0076a70 2024-04-26T01:44:25.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.891+0000 7f56f5400700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 0x7f56d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:25.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.891+0000 7f56f5400700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 0x7f56d80a6180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56e400b2e0 tx=0x7f56e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:25.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:25.895+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56dc09cee0 con 0x7f56f0076a70 2024-04-26T01:44:26.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.131+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f56f0063650 con 0x7f56f0076a70 2024-04-26T01:44:26.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.133+0000 7f56ee000700 1 -- 172.21.15.52:0/2575629310 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5284 (secure 0 0 0) 0x7f56dc09c7e0 con 0x7f56f0076a70 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 94 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-04-26T01:31:27.358609+0000 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2024-04-26T01:41:47.490242+0000 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:crush_version 26 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:full_ratio 0.95 2024-04-26T01:44:26.132 INFO:teuthology.orchestra.run.smithi052.stdout:backfillfull_ratio 0.9 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:nearfull_ratio 0.85 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:require_min_compat_client luminous 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:min_compat_client luminous 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:require_osd_release squid 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:stretch_mode_enabled false 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 92 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 92 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:max_osd 12 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 up in weight 1 up_from 8 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6802/4212634049,v1:172.21.15.52:6803/4212634049] [v2:172.21.15.52:6804/4212634049,v1:172.21.15.52:6805/4212634049] exists,up db649844-3866-493f-aa45-bbdcc973f556 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/3104160515,v1:172.21.15.183:6801/3104160515] [v2:172.21.15.183:6802/3104160515,v1:172.21.15.183:6803/3104160515] exists,up 285c6829-a343-4ccb-9883-8a1e24fb28ba 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/932162512,v1:172.21.15.191:6801/932162512] [v2:172.21.15.191:6802/932162512,v1:172.21.15.191:6803/932162512] exists,up bc51fafd-4a20-45b8-9b1f-d22980ab4b86 2024-04-26T01:44:26.133 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 up in weight 1 up_from 27 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6810/1589015031,v1:172.21.15.52:6811/1589015031] [v2:172.21.15.52:6812/1589015031,v1:172.21.15.52:6813/1589015031] exists,up 376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 up in weight 1 up_from 35 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6808/2309146877,v1:172.21.15.183:6809/2309146877] [v2:172.21.15.183:6810/2309146877,v1:172.21.15.183:6811/2309146877] exists,up 014075df-27cd-4f11-8542-ecc8ac24a820 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 up in weight 1 up_from 41 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/3556847761,v1:172.21.15.191:6809/3556847761] [v2:172.21.15.191:6810/3556847761,v1:172.21.15.191:6811/3556847761] exists,up 59ea2f70-c81c-4121-890e-1216b8b142f2 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6 up in weight 1 up_from 49 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6818/915546379,v1:172.21.15.52:6819/915546379] [v2:172.21.15.52:6820/915546379,v1:172.21.15.52:6821/915546379] exists,up b37abf58-d1b9-4290-bcc1-c1163dfc256f 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7 up in weight 1 up_from 54 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6816/1224783051,v1:172.21.15.183:6817/1224783051] [v2:172.21.15.183:6818/1224783051,v1:172.21.15.183:6819/1224783051] exists,up 569c973e-98a6-4957-bb2e-3eb411ea369c 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.8 up in weight 1 up_from 62 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/3371568470,v1:172.21.15.191:6817/3371568470] [v2:172.21.15.191:6818/3371568470,v1:172.21.15.191:6819/3371568470] exists,up 85ed3a78-d945-4ea1-a9a6-1391924890fa 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.9 up in weight 1 up_from 70 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6826/405746213,v1:172.21.15.52:6827/405746213] [v2:172.21.15.52:6828/405746213,v1:172.21.15.52:6829/405746213] exists,up dd9bae00-f288-4771-a5c7-da5fb4c4f7d8 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.10 up in weight 1 up_from 76 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6824/2192270029,v1:172.21.15.183:6825/2192270029] [v2:172.21.15.183:6826/2192270029,v1:172.21.15.183:6827/2192270029] exists,up 4d3f07a6-f55c-443d-9e9d-16036d02a9ab 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:osd.11 up in weight 1 up_from 84 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/1249163110,v1:172.21.15.191:6825/1249163110] [v2:172.21.15.191:6826/1249163110,v1:172.21.15.191:6827/1249163110] exists,up 6c795e48-0fbd-4ff6-b778-f57b2a07a4fd 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.4 [2,11] 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.c [2,4] 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.1d [8,11] 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.11 [5,9] 2024-04-26T01:44:26.134 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.17 [4,8,5,6] 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.19 [3,7] 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.2c [4,9] 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.3c [4,9] 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6801/3772941101 expires 2024-04-27T01:32:45.509758+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/4041331855 expires 2024-04-27T01:32:45.509758+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/3772941101 expires 2024-04-27T01:32:45.509758+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/292261907 expires 2024-04-27T01:32:45.509758+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6801/496114326 expires 2024-04-27T01:32:09.875117+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/459130865 expires 2024-04-27T01:32:09.875117+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/496114326 expires 2024-04-27T01:32:09.875117+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3936152513 expires 2024-04-27T01:32:09.875117+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/483957502 expires 2024-04-27T01:32:09.875117+0000 2024-04-26T01:44:26.135 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/320501252 expires 2024-04-27T01:32:45.509758+0000 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 msgr2=0x7f56d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 0x7f56d80a6180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56e400b2e0 tx=0x7f56e4005fb0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 msgr2=0x7f56f019f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00e9a0 tx=0x7f56dc00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 shutdown_connections 2024-04-26T01:44:26.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f56d80a3cc0 0x7f56d80a6180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56f010efa0 0x7f56f01a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f00754a0 0x7f56f019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 --2- 172.21.15.52:0/2575629310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56f0076a70 0x7f56f019f5f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.138+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 >> 172.21.15.52:0/2575629310 conn(0x7f56f00fdf80 msgr2=0x7f56f0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.139+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 shutdown_connections 2024-04-26T01:44:26.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:26.139+0000 7f56f6800700 1 -- 172.21.15.52:0/2575629310 wait complete. 2024-04-26T01:44:26.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:26 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2575629310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:44:26.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:26 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2575629310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:44:26.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:26 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2575629310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T01:44:26.969 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T01:44:26.980 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:27.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:27.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:27.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:27 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.432 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:27.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:27 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: pgmap v434: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:27 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:28.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:28 smithi052 ceph-mon[27851]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:28.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:28.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:28.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:28 smithi191 ceph-mon[32745]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:28.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:44:28.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:44:28.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:28 smithi183 ceph-mon[32752]: from='mgr.14152 172.21.15.52:0/3955570308' entity='mgr.x' 2024-04-26T01:44:29.421 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:29 smithi052 ceph-mon[27851]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 -- 172.21.15.52:0/1949797025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c101940 msgr2=0x7fe32c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 --2- 172.21.15.52:0/1949797025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c101940 0x7fe32c101dc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe318009990 tx=0x7fe318009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 -- 172.21.15.52:0/1949797025 shutdown_connections 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 --2- 172.21.15.52:0/1949797025 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 --2- 172.21.15.52:0/1949797025 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c107990 0x7fe32c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 --2- 172.21.15.52:0/1949797025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c101940 0x7fe32c101dc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 -- 172.21.15.52:0/1949797025 >> 172.21.15.52:0/1949797025 conn(0x7fe32c0788f0 msgr2=0x7fe32c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:29.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 -- 172.21.15.52:0/1949797025 shutdown_connections 2024-04-26T01:44:29.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.597+0000 7fe333c00700 1 -- 172.21.15.52:0/1949797025 wait complete. 2024-04-26T01:44:29.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 Processor -- start 2024-04-26T01:44:29.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 -- start start 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c101940 0x7fe32c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c111b20 con 0x7fe32c107990 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c1119a0 con 0x7fe32c102300 2024-04-26T01:44:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.599+0000 7fe333c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c111ca0 con 0x7fe32c101940 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42452/0 (socket says 172.21.15.52:42452) 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 -- 172.21.15.52:0/2052913673 learned_addr learned my addr 172.21.15.52:0/2052913673 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe32be00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331200700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c101940 0x7fe32c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 -- 172.21.15.52:0/2052913673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c101940 msgr2=0x7fe32c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c101940 0x7fe32c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 -- 172.21.15.52:0/2052913673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 msgr2=0x7fe32c19b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c19b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe331c00700 1 -- 172.21.15.52:0/2052913673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c1a0430 con 0x7fe32c107990 2024-04-26T01:44:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.600+0000 7fe32be00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c19b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe331c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c370 tx=0x7fe32400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3240092e0 con 0x7fe32c107990 2024-04-26T01:44:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe324003a00 con 0x7fe32c107990 2024-04-26T01:44:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3240046f0 con 0x7fe32c107990 2024-04-26T01:44:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c1a06c0 con 0x7fe32c107990 2024-04-26T01:44:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.601+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c1a7fa0 con 0x7fe32c107990 2024-04-26T01:44:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.602+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c069730 con 0x7fe32c107990 2024-04-26T01:44:29.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.604+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe32401a020 con 0x7fe32c107990 2024-04-26T01:44:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.606+0000 7fe329600700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 0x7fe3140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.606+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fe3240d02a0 con 0x7fe32c107990 2024-04-26T01:44:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.606+0000 7fe331200700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 0x7fe3140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.607+0000 7fe331200700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 0x7fe3140a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe320009be0 tx=0x7fe320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.609+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe324010040 con 0x7fe32c107990 2024-04-26T01:44:29.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:29 smithi191 ceph-mon[32745]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:29.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:29 smithi183 ceph-mon[32752]: pgmap v435: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:29.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.891+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe32c1a8170 con 0x7fe32c107990 2024-04-26T01:44:29.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.893+0000 7fe329600700 1 -- 172.21.15.52:0/2052913673 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fe32401a310 con 0x7fe32c107990 2024-04-26T01:44:29.891 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:29.891 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:44:29.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 msgr2=0x7fe3140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 0x7fe3140a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe320009be0 tx=0x7fe320009260 comp rx=0 tx=0).stop 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 msgr2=0x7fe32c19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c370 tx=0x7fe32400c680 comp rx=0 tx=0).stop 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 shutdown_connections 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fe3140a3cc0 0x7fe3140a6180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe32c107990 0x7fe32c19fcf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe32c101940 0x7fe32c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 --2- 172.21.15.52:0/2052913673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe32c102300 0x7fe32c19b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:29.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 >> 172.21.15.52:0/2052913673 conn(0x7fe32c0788f0 msgr2=0x7fe32c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:29.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.896+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 shutdown_connections 2024-04-26T01:44:29.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:29.897+0000 7fe333c00700 1 -- 172.21.15.52:0/2052913673 wait complete. 2024-04-26T01:44:29.896 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:44:30.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:30 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2052913673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:30.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:30 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2052913673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:30.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:30 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2052913673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:30.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:44:31.233 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:31.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:31 smithi052 ceph-mon[27851]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:31.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:31 smithi191 ceph-mon[32745]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:31 smithi183 ceph-mon[32752]: pgmap v436: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:33.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:33 smithi052 ceph-mon[27851]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.538+0000 7ff44a400700 1 -- 172.21.15.52:0/665737557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444109f90 msgr2=0x7ff444111b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.538+0000 7ff44a400700 1 --2- 172.21.15.52:0/665737557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444109f90 0x7ff444111b60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00b0a0 tx=0x7ff43c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 -- 172.21.15.52:0/665737557 shutdown_connections 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 --2- 172.21.15.52:0/665737557 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444109f90 0x7ff444111b60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 --2- 172.21.15.52:0/665737557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444104eb0 0x7ff444105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 --2- 172.21.15.52:0/665737557 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff4441098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 -- 172.21.15.52:0/665737557 >> 172.21.15.52:0/665737557 conn(0x7ff444100740 msgr2=0x7ff444102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 -- 172.21.15.52:0/665737557 shutdown_connections 2024-04-26T01:44:33.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.539+0000 7ff44a400700 1 -- 172.21.15.52:0/665737557 wait complete. 2024-04-26T01:44:33.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.540+0000 7ff44a400700 1 Processor -- start 2024-04-26T01:44:33.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 -- start start 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 0x7ff44419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff44419b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff444114e20 con 0x7ff444109f90 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff444114ca0 con 0x7ff444104eb0 2024-04-26T01:44:33.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.541+0000 7ff44a400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff444114fa0 con 0x7ff444105860 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 0x7ff44419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42478/0 (socket says 172.21.15.52:42478) 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443e00700 1 -- 172.21.15.52:0/974681411 learned_addr learned my addr 172.21.15.52:0/974681411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff43be00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff44419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 -- 172.21.15.52:0/974681411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 msgr2=0x7ff44419b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff44419b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 -- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 msgr2=0x7ff44419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443400700 1 -- 172.21.15.52:0/974681411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4441a02b0 con 0x7ff444104eb0 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff43be00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff44419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.542+0000 7ff443e00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:33.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff443400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 0x7ff44419b2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9f0 tx=0x7ff43400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43400c9c0 con 0x7ff444104eb0 2024-04-26T01:44:33.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff43400cb60 con 0x7ff444104eb0 2024-04-26T01:44:33.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff434018680 con 0x7ff444104eb0 2024-04-26T01:44:33.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4441a05a0 con 0x7ff444104eb0 2024-04-26T01:44:33.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.543+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4441a7e20 con 0x7ff444104eb0 2024-04-26T01:44:33.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.544+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff444069730 con 0x7ff444104eb0 2024-04-26T01:44:33.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.546+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff434010430 con 0x7ff444104eb0 2024-04-26T01:44:33.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.548+0000 7ff440c00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 0x7ff4280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:33.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.548+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7ff4340d0530 con 0x7ff444104eb0 2024-04-26T01:44:33.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.548+0000 7ff43be00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 0x7ff4280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:33.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.549+0000 7ff43be00700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 0x7ff4280a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff44410b540 tx=0x7ff42c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:33.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.552+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff43409cec0 con 0x7ff444104eb0 2024-04-26T01:44:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:33 smithi191 ceph-mon[32745]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:33 smithi183 ceph-mon[32752]: pgmap v437: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:33.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.837+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff44402df80 con 0x7ff444104eb0 2024-04-26T01:44:33.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.838+0000 7ff440c00700 1 -- 172.21.15.52:0/974681411 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7ff43409c7c0 con 0x7ff444104eb0 2024-04-26T01:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:33.837 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:41:31.153156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:44:33.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 msgr2=0x7ff4280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 0x7ff4280a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff44410b540 tx=0x7ff42c009260 comp rx=0 tx=0).stop 2024-04-26T01:44:33.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 msgr2=0x7ff44419b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:33.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 0x7ff44419b2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9f0 tx=0x7ff43400ed00 comp rx=0 tx=0).stop 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 shutdown_connections 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7ff4280a3cc0 0x7ff4280a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff444109f90 0x7ff44419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff444104eb0 0x7ff44419b2f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.841+0000 7ff44a400700 1 --2- 172.21.15.52:0/974681411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff444105860 0x7ff44419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.842+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 >> 172.21.15.52:0/974681411 conn(0x7ff444100740 msgr2=0x7ff444102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.842+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 shutdown_connections 2024-04-26T01:44:33.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:33.842+0000 7ff44a400700 1 -- 172.21.15.52:0/974681411 wait complete. 2024-04-26T01:44:33.841 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 24 2024-04-26T01:44:34.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:34 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/974681411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:34.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:34 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/974681411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:34.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:44:34.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:34 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/974681411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:44:35.127 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:35.502 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:35 smithi052 ceph-mon[27851]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:35.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:35 smithi191 ceph-mon[32745]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:35 smithi183 ceph-mon[32752]: pgmap v438: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 -- 172.21.15.52:0/3939755521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06b0f0 msgr2=0x7fc83c10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 --2- 172.21.15.52:0/3939755521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06b0f0 0x7fc83c10f900 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc83400b0a0 tx=0x7fc83400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 -- 172.21.15.52:0/3939755521 shutdown_connections 2024-04-26T01:44:37.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 --2- 172.21.15.52:0/3939755521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06b0f0 0x7fc83c10f900 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 --2- 172.21.15.52:0/3939755521 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 --2- 172.21.15.52:0/3939755521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06a750 0x7fc83c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 -- 172.21.15.52:0/3939755521 >> 172.21.15.52:0/3939755521 conn(0x7fc83c0788f0 msgr2=0x7fc83c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:37.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 -- 172.21.15.52:0/3939755521 shutdown_connections 2024-04-26T01:44:37.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.426+0000 7fc844400700 1 -- 172.21.15.52:0/3939755521 wait complete. 2024-04-26T01:44:37.425 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:37 smithi052 ceph-mon[27851]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 Processor -- start 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 -- start start 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06b0f0 0x7fc83c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.428+0000 7fc844400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c112c20 con 0x7fc83c06a750 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc844400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c112aa0 con 0x7fc83c108a30 2024-04-26T01:44:37.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc844400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c112da0 con 0x7fc83c06b0f0 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60610/0 (socket says 172.21.15.52:60610) 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 -- 172.21.15.52:0/3801096352 learned_addr learned my addr 172.21.15.52:0/3801096352 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc842400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841000700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06b0f0 0x7fc83c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 -- 172.21.15.52:0/3801096352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06b0f0 msgr2=0x7fc83c1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06b0f0 0x7fc83c1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 -- 172.21.15.52:0/3801096352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 msgr2=0x7fc83c19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc841a00700 1 -- 172.21.15.52:0/3801096352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c19e130 con 0x7fc83c06a750 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.429+0000 7fc842400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc841a00700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e9a0 tx=0x7fc83000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:37.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83000ca10 con 0x7fc83c06a750 2024-04-26T01:44:37.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc83000cbb0 con 0x7fc83c06a750 2024-04-26T01:44:37.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830018680 con 0x7fc83c06a750 2024-04-26T01:44:37.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c19e420 con 0x7fc83c06a750 2024-04-26T01:44:37.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.430+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c1a5ca0 con 0x7fc83c06a750 2024-04-26T01:44:37.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.431+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c069730 con 0x7fc83c06a750 2024-04-26T01:44:37.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.433+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc830010430 con 0x7fc83c06a750 2024-04-26T01:44:37.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.435+0000 7fc83a000700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 0x7fc8240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:37.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.435+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fc8300d07d0 con 0x7fc83c06a750 2024-04-26T01:44:37.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.435+0000 7fc841000700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 0x7fc8240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:37.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.436+0000 7fc841000700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 0x7fc8240a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc83c078ed0 tx=0x7fc828009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.438+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc83009d160 con 0x7fc83c06a750 2024-04-26T01:44:37.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:37 smithi191 ceph-mon[32745]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:37.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.669+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc83c02df80 con 0x7fc83c06a750 2024-04-26T01:44:37.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.673+0000 7fc83a000700 1 -- 172.21.15.52:0/3801096352 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22667 (secure 0 0 0) 0x7fc83009ca60 con 0x7fc83c06a750 2024-04-26T01:44:37.671 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:44:37.671 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":94,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:41:47.490242+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:44:37.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 msgr2=0x7fc8240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 0x7fc8240a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc83c078ed0 tx=0x7fc828009260 comp rx=0 tx=0).stop 2024-04-26T01:44:37.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 msgr2=0x7fc83c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:37.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc83000e9a0 tx=0x7fc83000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:37 smithi183 ceph-mon[32752]: pgmap v439: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 shutdown_connections 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fc8240a3cc0 0x7fc8240a6180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc83c108a30 0x7fc83c19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc83c06a750 0x7fc83c199170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 --2- 172.21.15.52:0/3801096352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc83c06b0f0 0x7fc83c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 >> 172.21.15.52:0/3801096352 conn(0x7fc83c0788f0 msgr2=0x7fc83c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.676+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 shutdown_connections 2024-04-26T01:44:37.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:37.677+0000 7fc844400700 1 -- 172.21.15.52:0/3801096352 wait complete. 2024-04-26T01:44:38.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:38 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3801096352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:41:06.424614+0000', 'modified': '2024-04-26T01:41:31.153156+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24512, 'mds_2': 24496}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24472': {'gid': 24472, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/234906181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 234906181}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 234906181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6837/2929119914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2929119914}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2929119914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.183:6837/1263803776', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6836', 'nonce': 1263803776}, {'type': 'v1', 'addr': '172.21.15.183:6837', 'nonce': 1263803776}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.191:6839/1544004799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 1544004799}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 1544004799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'a', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.52:6841/1834450967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6840', 'nonce': 1834450967}, {'type': 'v1', 'addr': '172.21.15.52:6841', 'nonce': 1834450967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.183:6839/1410734884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6838', 'nonce': 1410734884}, {'type': 'v1', 'addr': '172.21.15.183:6839', 'nonce': 1410734884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24512, 24496, 24524]} 2024-04-26T01:44:38.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:44:38.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:44:38.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:38 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3801096352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:38 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3801096352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:44:38.974 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:39.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:39 smithi052 ceph-mon[27851]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:39.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:39 smithi191 ceph-mon[32745]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:39 smithi183 ceph-mon[32752]: pgmap v440: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 -- 172.21.15.52:0/2191357939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 msgr2=0x7fcb9406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/2191357939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9406abb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009990 tx=0x7fcb80009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 -- 172.21.15.52:0/2191357939 shutdown_connections 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/2191357939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb9406b0f0 0x7fcb9410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/2191357939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb94108a30 0x7fcb94108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/2191357939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9406abb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 -- 172.21.15.52:0/2191357939 >> 172.21.15.52:0/2191357939 conn(0x7fcb940788f0 msgr2=0x7fcb94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 -- 172.21.15.52:0/2191357939 shutdown_connections 2024-04-26T01:44:41.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.030+0000 7fcb9ae00700 1 -- 172.21.15.52:0/2191357939 wait complete. 2024-04-26T01:44:41.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 Processor -- start 2024-04-26T01:44:41.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 -- start start 2024-04-26T01:44:41.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 0x7fcb9419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb94108a30 0x7fcb9419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94112c60 con 0x7fcb9406a750 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94112ae0 con 0x7fcb9406b0f0 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.032+0000 7fcb9ae00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94112de0 con 0x7fcb94108a30 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60638/0 (socket says 172.21.15.52:60638) 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 -- 172.21.15.52:0/753221902 learned_addr learned my addr 172.21.15.52:0/753221902 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb98e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb94108a30 0x7fcb9419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93400700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 0x7fcb9419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 -- 172.21.15.52:0/753221902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb94108a30 msgr2=0x7fcb9419fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb94108a30 0x7fcb9419fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 -- 172.21.15.52:0/753221902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 msgr2=0x7fcb9419b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 0x7fcb9419b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93e00700 1 -- 172.21.15.52:0/753221902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb941a0400 con 0x7fcb9406a750 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.033+0000 7fcb93400700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 0x7fcb9419b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:41.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb93e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e9a0 tx=0x7fcb8400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb90c00700 1 -- 172.21.15.52:0/753221902 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8400cbd0 con 0x7fcb9406a750 2024-04-26T01:44:41.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb90c00700 1 -- 172.21.15.52:0/753221902 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8400cd70 con 0x7fcb9406a750 2024-04-26T01:44:41.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb941a06f0 con 0x7fcb9406a750 2024-04-26T01:44:41.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb90c00700 1 -- 172.21.15.52:0/753221902 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84018680 con 0x7fcb9406a750 2024-04-26T01:44:41.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.034+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb941a7f70 con 0x7fcb9406a750 2024-04-26T01:44:41.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.039+0000 7fcb90c00700 1 -- 172.21.15.52:0/753221902 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcb84010430 con 0x7fcb9406a750 2024-04-26T01:44:41.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.040+0000 7fcba571bb80 1 Processor -- start 2024-04-26T01:44:41.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.040+0000 7fcba571bb80 1 -- start start 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 0x55647ba81570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55647b9c9e40 con 0x55647ba762f0 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55647b9706e0 con 0x55647ba76c10 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcba571bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55647b9c5e00 con 0x55647ba81ab0 2024-04-26T01:44:41.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb90c00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 0x7fcb780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb90c00700 1 -- 172.21.15.52:0/753221902 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fcb840d0580 con 0x7fcb9406a750 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb93400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb98e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb93400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45878/0 (socket says 172.21.15.52:45878) 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb93400700 1 -- 172.21.15.52:0/2855661406 learned_addr learned my addr 172.21.15.52:0/2855661406 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb93e00700 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 0x55647ba81570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.041+0000 7fcb98e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 0x7fcb780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 -- 172.21.15.52:0/2855661406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 msgr2=0x55647ba83ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 -- 172.21.15.52:0/2855661406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 msgr2=0x55647ba766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba766d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 -- 172.21.15.52:0/2855661406 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55647b966630 con 0x55647ba76c10 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93400700 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb93e00700 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 0x55647ba81570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb840a4f50 tx=0x7fcb84097210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e77467f978ea460 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb98e00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 0x7fcb780a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009ec0 tx=0x7fcb8c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb89600700 1 -- 172.21.15.52:0/2855661406 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8409cf10 con 0x55647ba76c10 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb89600700 1 -- 172.21.15.52:0/2855661406 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8409c810 con 0x55647ba76c10 2024-04-26T01:44:41.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.042+0000 7fcb89600700 1 -- 172.21.15.52:0/2855661406 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840d00f0 con 0x55647ba76c10 2024-04-26T01:44:41.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 -- 172.21.15.52:0/2855661406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 msgr2=0x55647ba81570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 0x55647ba81570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb840a4f50 tx=0x7fcb84097210 comp rx=0 tx=0).stop 2024-04-26T01:44:41.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 -- 172.21.15.52:0/2855661406 shutdown_connections 2024-04-26T01:44:41.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba83ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 --2- 172.21.15.52:0/2855661406 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba76c10 0x55647ba81570 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 -- 172.21.15.52:0/2855661406 >> 172.21.15.52:0/2855661406 conn(0x55647b9a5550 msgr2=0x55647ba72790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 -- 172.21.15.52:0/2855661406 shutdown_connections 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.043+0000 7fcba571bb80 1 -- 172.21.15.52:0/2855661406 wait complete. 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.044+0000 7fcba571bb80 1 -- start start 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.044+0000 7fcba571bb80 1 Processor -- start 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba762f0 0x55647ba81220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba7ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55647ba845a0 con 0x55647ba76c10 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55647b983810 con 0x55647ba81ab0 2024-04-26T01:44:41.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55647ba84720 con 0x55647ba762f0 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcba571bb80 10 client.0 _ll_register_callbacks cb 0x55647b9626d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60668/0 (socket says 172.21.15.52:60668) 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 -- 172.21.15.52:0/963990494 learned_addr learned my addr 172.21.15.52:0/963990494 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93e00700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba762f0 0x55647ba81220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb98e00700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba7ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 -- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba762f0 msgr2=0x55647ba81220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba762f0 0x55647ba81220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 -- 172.21.15.52:0/963990494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 msgr2=0x55647ba7ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba7ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.045+0000 7fcb93400700 1 -- 172.21.15.52:0/963990494 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55647b917880 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcb93400700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb800098e0 tx=0x7fcb8000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7b8043785766f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcb7e000700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80027030 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fcb8000f8c0 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8409c810 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcb8000fac0 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.046+0000 7fcba571bb80 10 client.14871 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55647b960aa0 con 0x55647ba76c10 2024-04-26T01:44:41.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8409c810 con 0x55647ba76c10 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fcb8002cc60 con 0x55647ba76c10 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 10 client.14871 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 10 client.14871 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 20 client.14871 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647ba91370 0x55647ba927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 4 client.14871 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.047+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55647b9d7630 con 0x55647ba91370 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.048+0000 7fcb93e00700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647ba91370 0x55647ba927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.048+0000 7fcb93e00700 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647ba91370 0x55647ba927e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=afec5e8aceb736d3 server_cookie=b15302813c49c104 in_seq=0 out_seq=0 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.048+0000 7fcb7e000700 10 client.14871 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcb7e000700 1 -- 172.21.15.52:0/963990494 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55647b9d7630 con 0x55647ba91370 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcb7e000700 10 client.14871 handle_command_reply: tid=0 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 client.14871 shutdown 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 msgr2=0x55647ba7e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb800098e0 tx=0x7fcb8000be70 comp rx=0 tx=0).stop 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 shutdown_connections 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647ba91370 0x55647ba927e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba7ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba762f0 0x55647ba81220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 --2- 172.21.15.52:0/963990494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7e520 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcb7e000700 0 client.14871 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.049+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 >> 172.21.15.52:0/963990494 conn(0x55647b9a5550 msgr2=0x55647ba71950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.050+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 shutdown_connections 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.050+0000 7fcba571bb80 1 -- 172.21.15.52:0/963990494 wait complete. 2024-04-26T01:44:41.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.050+0000 7fcba571bb80 20 client.14871 trim_cache size 0 max 16384 2024-04-26T01:44:41.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.072+0000 7fcba571bb80 1 Processor -- start 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.072+0000 7fcba571bb80 1 -- start start 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba76c10 0x55647ba720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba72610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55647ba90b20 con 0x55647ba762f0 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55647ba911f0 con 0x55647ba81ab0 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcba571bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55647ba918c0 con 0x55647ba76c10 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcb93e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcb93e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60674/0 (socket says 172.21.15.52:60674) 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcb93e00700 1 -- 172.21.15.52:0/1025937577 learned_addr learned my addr 172.21.15.52:0/1025937577 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:41.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcb98e00700 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba72610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.073+0000 7fcb93400700 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba76c10 0x55647ba720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 -- 172.21.15.52:0/1025937577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba76c10 msgr2=0x55647ba720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba76c10 0x55647ba720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 -- 172.21.15.52:0/1025937577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 msgr2=0x55647ba72610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba72610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 -- 172.21.15.52:0/1025937577 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55647ba77db0 con 0x55647ba762f0 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c5b0 tx=0x7fcb84010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333173069a284b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb88c00700 1 -- 172.21.15.52:0/1025937577 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8409cf10 con 0x55647ba762f0 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb88c00700 1 -- 172.21.15.52:0/1025937577 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8409c810 con 0x55647ba762f0 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.074+0000 7fcb88c00700 1 -- 172.21.15.52:0/1025937577 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8409ccb0 con 0x55647ba762f0 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 -- 172.21.15.52:0/1025937577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 msgr2=0x55647ba7ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c5b0 tx=0x7fcb84010cb0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 -- 172.21.15.52:0/1025937577 shutdown_connections 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba81ab0 0x55647ba72610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba762f0 0x55647ba7ea10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1025937577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba76c10 0x55647ba720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 -- 172.21.15.52:0/1025937577 >> 172.21.15.52:0/1025937577 conn(0x55647b9a5550 msgr2=0x55647ba7e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 -- 172.21.15.52:0/1025937577 shutdown_connections 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.075+0000 7fcba571bb80 1 -- 172.21.15.52:0/1025937577 wait complete. 2024-04-26T01:44:41.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 -- start start 2024-04-26T01:44:41.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 Processor -- start 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba762f0 0x55647ba7b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba98030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55647ba92660 con 0x55647ba76c10 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55647ba91f90 con 0x55647ba762f0 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55647ba92d30 con 0x55647ba81ab0 2024-04-26T01:44:41.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.076+0000 7fcba571bb80 10 client.0 _ll_register_callbacks cb 0x55647b9626d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60688/0 (socket says 172.21.15.52:60688) 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 -- 172.21.15.52:0/1953922513 learned_addr learned my addr 172.21.15.52:0/1953922513 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba762f0 0x55647ba7b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb98e00700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba98030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 -- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 msgr2=0x55647ba98030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba98030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 -- 172.21.15.52:0/1953922513 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba762f0 msgr2=0x55647ba7b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba762f0 0x55647ba7b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 -- 172.21.15.52:0/1953922513 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55647ba7c2a0 con 0x55647ba76c10 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb93400700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002cff0 tx=0x7fcb8000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ecd4532555c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:41.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.077+0000 7fcb7e000700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80027070 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fcb80004830 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8409c810 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcb8002c450 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcba571bb80 10 client.14877 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55647baf8a30 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8409c810 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.078+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fcb80037620 con 0x55647ba76c10 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 10 client.14877 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 10 client.14877 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:44:41.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 20 client.14877 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647bafd570 0x55647bafc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 4 client.14877 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "status"}) -- 0x55647b9d16d0 con 0x55647bafd570 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.079+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647bafd570 0x55647bafc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcb93e00700 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647bafd570 0x55647bafc280 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5591b8ae4da20c50 server_cookie=387a0b2ec70105d1 in_seq=0 out_seq=0 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcb7e000700 10 client.14877 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcb7e000700 1 -- 172.21.15.52:0/1953922513 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55647b9d16d0 con 0x55647bafd570 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcb7e000700 10 client.14877 handle_command_reply: tid=0 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 client.14877 shutdown 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 msgr2=0x55647ba7b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002cff0 tx=0x7fcb8000bf50 comp rx=0 tx=0).stop 2024-04-26T01:44:41.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 shutdown_connections 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55647bafd570 0x55647bafc280 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55647ba81ab0 0x55647ba98030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55647ba762f0 0x55647ba7b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 --2- 172.21.15.52:0/1953922513 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55647ba76c10 0x55647ba7b9e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 >> 172.21.15.52:0/1953922513 conn(0x55647b9a5550 msgr2=0x55647ba793f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcb7e000700 0 client.14877 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 shutdown_connections 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.080+0000 7fcba571bb80 1 -- 172.21.15.52:0/1953922513 wait complete. 2024-04-26T01:44:41.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.081+0000 7fcba571bb80 20 client.14877 trim_cache size 0 max 16384 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 2, 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24496, 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 196.71376731399999, 2024-04-26T01:44:41.083 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:44:41.084 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:44:41.084 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:44:41.084 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 248.27997587499999 2024-04-26T01:44:41.084 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:44:41.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.081+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 msgr2=0x7fcb780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.081+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 0x7fcb780a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009ec0 tx=0x7fcb8c00b040 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.081+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 msgr2=0x7fcb9419b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.081+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e9a0 tx=0x7fcb8400ecb0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 shutdown_connections 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fcb780a3cc0 0x7fcb780a6180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb94108a30 0x7fcb9419fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb9406a750 0x7fcb9419b3f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 --2- 172.21.15.52:0/753221902 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcb9406b0f0 0x7fcb9419b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 >> 172.21.15.52:0/753221902 conn(0x7fcb940788f0 msgr2=0x7fcb9410dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 shutdown_connections 2024-04-26T01:44:41.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:41.082+0000 7fcb9ae00700 1 -- 172.21.15.52:0/753221902 wait complete. 2024-04-26T01:44:41.643 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:41 smithi052 ceph-mon[27851]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:41.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:41 smithi191 ceph-mon[32745]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:41 smithi183 ceph-mon[32752]: pgmap v441: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:42.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.a status 2024-04-26T01:44:42.502 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:43 smithi052 ceph-mon[27851]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:43.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:43 smithi191 ceph-mon[32745]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:43.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:43 smithi183 ceph-mon[32752]: pgmap v442: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T01:44:44.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 -- 172.21.15.52:0/4007451862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa010f4a0 msgr2=0x7f4fa0111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 --2- 172.21.15.52:0/4007451862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa010f4a0 0x7f4fa0111890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800b0a0 tx=0x7f4f9800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 -- 172.21.15.52:0/4007451862 shutdown_connections 2024-04-26T01:44:44.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 --2- 172.21.15.52:0/4007451862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa010f4a0 0x7f4fa0111890 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 --2- 172.21.15.52:0/4007451862 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa0074c70 0x7f4fa0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 --2- 172.21.15.52:0/4007451862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0075590 0x7f4fa010ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 -- 172.21.15.52:0/4007451862 >> 172.21.15.52:0/4007451862 conn(0x7f4fa0079ba0 msgr2=0x7f4fa0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.601+0000 7f4fa8400700 1 -- 172.21.15.52:0/4007451862 shutdown_connections 2024-04-26T01:44:44.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.602+0000 7f4fa8400700 1 -- 172.21.15.52:0/4007451862 wait complete. 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.602+0000 7f4fa8400700 1 Processor -- start 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 -- start start 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 0x7f4fa019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa010f4a0 0x7f4fa019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0114e30 con 0x7f4fa0075590 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0114cb0 con 0x7f4fa010f4a0 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa8400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0114fb0 con 0x7f4fa0074c70 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa5000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa5000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55794/0 (socket says 172.21.15.52:55794) 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa5000700 1 -- 172.21.15.52:0/2267608064 learned_addr learned my addr 172.21.15.52:0/2267608064 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.603+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 0x7f4fa019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa6400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa010f4a0 0x7f4fa019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 -- 172.21.15.52:0/2267608064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 msgr2=0x7f4fa019b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 0x7f4fa019b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 -- 172.21.15.52:0/2267608064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa010f4a0 msgr2=0x7f4fa019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa010f4a0 0x7f4fa019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 -- 172.21.15.52:0/2267608064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa01a02b0 con 0x7f4fa0075590 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 0x7f4fa019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.604+0000 7f4fa5000700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9c0 tx=0x7f4f8c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.605+0000 7f4f9e000700 1 -- 172.21.15.52:0/2267608064 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00cbd0 con 0x7f4fa0075590 2024-04-26T01:44:44.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.605+0000 7f4f9e000700 1 -- 172.21.15.52:0/2267608064 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f8c00ee60 con 0x7f4fa0075590 2024-04-26T01:44:44.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.605+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa01a05a0 con 0x7f4fa0075590 2024-04-26T01:44:44.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.605+0000 7f4f9e000700 1 -- 172.21.15.52:0/2267608064 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0055c0 con 0x7f4fa0075590 2024-04-26T01:44:44.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.605+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa01a7e20 con 0x7f4fa0075590 2024-04-26T01:44:44.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.608+0000 7f4f9e000700 1 -- 172.21.15.52:0/2267608064 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f8c004750 con 0x7f4fa0075590 2024-04-26T01:44:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4f9e000700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 0x7f4f880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 Processor -- start 2024-04-26T01:44:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4f9e000700 1 -- 172.21.15.52:0/2267608064 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f4f8c0d0630 con 0x7f4fa0075590 2024-04-26T01:44:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 -- start start 2024-04-26T01:44:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 0x7f4f880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 0x55c08b55f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b560330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c08b4b1e40 con 0x55c08b55fef0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c08b4586e0 con 0x55c08b5660a0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.610+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c08b4ade00 con 0x55c08b565780 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44298/0 (socket says 172.21.15.52:44298) 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5a00700 1 -- 172.21.15.52:0/875242681 learned_addr learned my addr 172.21.15.52:0/875242681 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 0x55c08b55f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa6400700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b560330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 0x7f4f880a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4f94011550 tx=0x7f4f94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 -- 172.21.15.52:0/875242681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 msgr2=0x55c08b565b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 -- 172.21.15.52:0/875242681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 msgr2=0x55c08b560330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b560330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5000700 1 -- 172.21.15.52:0/875242681 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08b44e630 con 0x55c08b5660a0 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.611+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:44.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.612+0000 7f4fa5000700 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 0x55c08b55f9b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009fd0 tx=0x7f4f8c09c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f692fc23fd871d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.612+0000 7f4f92a00700 1 -- 172.21.15.52:0/875242681 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c09c8c0 con 0x55c08b5660a0 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.612+0000 7f4f92a00700 1 -- 172.21.15.52:0/875242681 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f8c085e50 con 0x55c08b5660a0 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.612+0000 7f4f92a00700 1 -- 172.21.15.52:0/875242681 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c087300 con 0x55c08b5660a0 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/875242681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 msgr2=0x55c08b55f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 0x55c08b55f9b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009fd0 tx=0x7f4f8c09c480 comp rx=0 tx=0).stop 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/875242681 shutdown_connections 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b560330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b565b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/875242681 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b5660a0 0x55c08b55f9b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/875242681 >> 172.21.15.52:0/875242681 conn(0x55c08b48df00 msgr2=0x55c08b55a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/875242681 shutdown_connections 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/875242681 wait complete. 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.613+0000 7f4fb2d06b80 1 -- start start 2024-04-26T01:44:44.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 Processor -- start 2024-04-26T01:44:44.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 0x55c08b569030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 0x55c08b5699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c08b56c920 con 0x55c08b55fef0 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c08b46b810 con 0x55c08b565780 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c08b56caa0 con 0x55c08b569570 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.614+0000 7f4fb2d06b80 10 client.0 _ll_register_callbacks cb 0x55c08b44a6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55802/0 (socket says 172.21.15.52:55802) 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 0x55c08b569030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 -- 172.21.15.52:0/3525673917 learned_addr learned my addr 172.21.15.52:0/3525673917 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa6400700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 0x55c08b5699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 -- 172.21.15.52:0/3525673917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 msgr2=0x55c08b5699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 0x55c08b5699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 -- 172.21.15.52:0/3525673917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 msgr2=0x55c08b569030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 0x55c08b569030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5a00700 1 -- 172.21.15.52:0/3525673917 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08b3ff880 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa6400700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 0x55c08b5699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.615+0000 7f4fa5000700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 0x55c08b569030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e70 tx=0x7f4f940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d945764559a0022 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4f87400700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9401c610 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f4f9401c7b0 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f8c085e50 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4f9401c9b0 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4fb2d06b80 10 client.14889 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.616+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c08b448aa0 con 0x55c08b55fef0 2024-04-26T01:44:44.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f8c085e50 con 0x55c08b55fef0 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4f9401a040 con 0x55c08b55fef0 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 10 client.14889 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 10 client.14889 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 20 client.14889 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b579b20 0x55c08b57af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 4 client.14889 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c08b4bf630 con 0x55c08b579b20 2024-04-26T01:44:44.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.617+0000 7f4fa5000700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b579b20 0x55c08b57af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.618+0000 7f4fa5000700 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b579b20 0x55c08b57af90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7686b2e9307ef5d2 server_cookie=d9217602e313152 in_seq=0 out_seq=0 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.618+0000 7f4f87400700 10 client.14889 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4f87400700 1 -- 172.21.15.52:0/3525673917 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c08b4bf630 con 0x55c08b579b20 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4f87400700 10 client.14889 handle_command_reply: tid=0 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 client.14889 shutdown 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 msgr2=0x55c08b56bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e70 tx=0x7f4f940047c0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 shutdown_connections 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b579b20 0x55c08b57af90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b569570 0x55c08b5699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b56bd30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/3525673917 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b565780 0x55c08b569030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 >> 172.21.15.52:0/3525673917 conn(0x55c08b48df00 msgr2=0x55c08b559e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4f87400700 0 client.14889 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:44:44.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 shutdown_connections 2024-04-26T01:44:44.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.619+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/3525673917 wait complete. 2024-04-26T01:44:44.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.620+0000 7f4fb2d06b80 20 client.14889 trim_cache size 0 max 16384 2024-04-26T01:44:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.642+0000 7f4fb2d06b80 1 Processor -- start 2024-04-26T01:44:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.642+0000 7f4fb2d06b80 1 -- start start 2024-04-26T01:44:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 0x55c08b55f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b569570 0x55c08b55a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c08b5792d0 con 0x55c08b569570 2024-04-26T01:44:44.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c08b5799a0 con 0x55c08b55fef0 2024-04-26T01:44:44.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c08b57a070 con 0x55c08b565780 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fa5000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 0x55c08b55f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fa5000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44312/0 (socket says 172.21.15.52:44312) 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fa5000700 1 -- 172.21.15.52:0/2119541775 learned_addr learned my addr 172.21.15.52:0/2119541775 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.643+0000 7f4fa6400700 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b569570 0x55c08b55a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 -- 172.21.15.52:0/2119541775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 msgr2=0x55c08b55c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 -- 172.21.15.52:0/2119541775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b569570 msgr2=0x55c08b55a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b569570 0x55c08b55a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 -- 172.21.15.52:0/2119541775 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08b5675a0 con 0x55c08b55fef0 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5000700 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:44.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 0x55c08b55f9b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4f94005c30 tx=0x7f4f94005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7762f7be2c3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4f92000700 1 -- 172.21.15.52:0/2119541775 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94009ea0 con 0x55c08b55fef0 2024-04-26T01:44:44.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4f92000700 1 -- 172.21.15.52:0/2119541775 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f9402a070 con 0x55c08b55fef0 2024-04-26T01:44:44.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.644+0000 7f4f92000700 1 -- 172.21.15.52:0/2119541775 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94031700 con 0x55c08b55fef0 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/2119541775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 msgr2=0x55c08b55f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 0x55c08b55f9b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4f94005c30 tx=0x7f4f94005900 comp rx=0 tx=0).stop 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/2119541775 shutdown_connections 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b569570 0x55c08b55a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b55fef0 0x55c08b55f9b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/2119541775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b55c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.645+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/2119541775 >> 172.21.15.52:0/2119541775 conn(0x55c08b48df00 msgr2=0x55c08b5602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.646+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/2119541775 shutdown_connections 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.646+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/2119541775 wait complete. 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.646+0000 7f4fb2d06b80 1 -- start start 2024-04-26T01:44:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.646+0000 7f4fb2d06b80 1 Processor -- start 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b5596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b569570 0x55c08b559be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c08b57ae10 con 0x55c08b55fef0 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c08b57a740 con 0x55c08b569570 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c08b57b4e0 con 0x55c08b565780 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fb2d06b80 10 client.0 _ll_register_callbacks cb 0x55c08b44a6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:44.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fa5a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55828/0 (socket says 172.21.15.52:55828) 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fa5a00700 1 -- 172.21.15.52:0/1900522059 learned_addr learned my addr 172.21.15.52:0/1900522059 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fa6400700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b569570 0x55c08b559be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.647+0000 7f4fa5000700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b5596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 -- 172.21.15.52:0/1900522059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 msgr2=0x55c08b5596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b5596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 -- 172.21.15.52:0/1900522059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b569570 msgr2=0x55c08b559be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b569570 0x55c08b559be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 -- 172.21.15.52:0/1900522059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08b55a3b0 con 0x55c08b55fef0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4fa5a00700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e40 tx=0x7f4f94003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb65bb33da3cbe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4f87400700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9403e020 con 0x55c08b55fef0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.648+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f4f94031480 con 0x55c08b55fef0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f9402a070 con 0x55c08b55fef0 2024-04-26T01:44:44.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4f940316a0 con 0x55c08b55fef0 2024-04-26T01:44:44.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 10 client.14895 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:44.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c08b5e0c50 con 0x55c08b55fef0 2024-04-26T01:44:44.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f9402a070 con 0x55c08b55fef0 2024-04-26T01:44:44.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4f9401a050 con 0x55c08b55fef0 2024-04-26T01:44:44.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 10 client.14895 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 10 client.14895 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 20 client.14895 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b5e5510 0x55c08b5e4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.649+0000 7f4fb2d06b80 4 client.14895 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.650+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "status"}) -- 0x55c08b4b96d0 con 0x55c08b5e5510 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.650+0000 7f4fa5000700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b5e5510 0x55c08b5e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.650+0000 7f4fa5000700 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b5e5510 0x55c08b5e4220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13f2d911894709ac server_cookie=b4c130591b984b53 in_seq=0 out_seq=0 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.650+0000 7f4f87400700 10 client.14895 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4f87400700 1 -- 172.21.15.52:0/1900522059 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c08b4b96d0 con 0x55c08b5e5510 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4f87400700 10 client.14895 handle_command_reply: tid=0 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 client.14895 shutdown 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 msgr2=0x55c08b5846f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009e40 tx=0x7f4f94003cf0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 shutdown_connections 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x55c08b5e5510 0x55c08b5e4220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c08b569570 0x55c08b559be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c08b55fef0 0x55c08b5846f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 --2- 172.21.15.52:0/1900522059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c08b565780 0x55c08b5596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 >> 172.21.15.52:0/1900522059 conn(0x55c08b48df00 msgr2=0x55c08b566490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4f87400700 0 client.14895 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 shutdown_connections 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 1 -- 172.21.15.52:0/1900522059 wait complete. 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.651+0000 7f4fb2d06b80 20 client.14895 trim_cache size 0 max 16384 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:44:44.651 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 1, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24512, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 201.29697556599999, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 241.639395079 2024-04-26T01:44:44.652 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 msgr2=0x7f4f880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 0x7f4f880a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4f94011550 tx=0x7f4f94009260 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 msgr2=0x7f4fa019b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9c0 tx=0x7f4f8c00ecd0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 shutdown_connections 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f4f880a3cc0 0x7f4f880a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fa010f4a0 0x7f4fa019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fa0074c70 0x7f4fa019b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 --2- 172.21.15.52:0/2267608064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4fa0075590 0x7f4fa019b7e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.652+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 >> 172.21.15.52:0/2267608064 conn(0x7f4fa0079ba0 msgr2=0x7f4fa00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:44.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.653+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 shutdown_connections 2024-04-26T01:44:44.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:44.653+0000 7f4fa8400700 1 -- 172.21.15.52:0/2267608064 wait complete. 2024-04-26T01:44:45.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:45 smithi052 ceph-mon[27851]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:45.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:45 smithi191 ceph-mon[32745]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:45 smithi183 ceph-mon[32752]: pgmap v443: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T01:44:45.717 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:44:46.204 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:47.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:47 smithi191 ceph-mon[32745]: pgmap v444: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:47.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:47 smithi183 ceph-mon[32752]: pgmap v444: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:47 smithi052 ceph-mon[27851]: pgmap v444: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 -- 172.21.15.52:0/3152557938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4102960 msgr2=0x7efbf4102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 --2- 172.21.15.52:0/3152557938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4102960 0x7efbf4102de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009990 tx=0x7efbe0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 -- 172.21.15.52:0/3152557938 shutdown_connections 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 --2- 172.21.15.52:0/3152557938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4103320 0x7efbf410f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 --2- 172.21.15.52:0/3152557938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4108960 0x7efbf4108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 --2- 172.21.15.52:0/3152557938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4102960 0x7efbf4102de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 -- 172.21.15.52:0/3152557938 >> 172.21.15.52:0/3152557938 conn(0x7efbf40fe450 msgr2=0x7efbf4100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 -- 172.21.15.52:0/3152557938 shutdown_connections 2024-04-26T01:44:48.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.527+0000 7efbf9600700 1 -- 172.21.15.52:0/3152557938 wait complete. 2024-04-26T01:44:48.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.528+0000 7efbf9600700 1 Processor -- start 2024-04-26T01:44:48.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 -- start start 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4103320 0x7efbf4075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4108960 0x7efbf40757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efbf4112b70 con 0x7efbf4108960 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efbf41129f0 con 0x7efbf4103320 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf9600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efbf4112cf0 con 0x7efbf4102960 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.529+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44322/0 (socket says 172.21.15.52:44322) 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 -- 172.21.15.52:0/313549580 learned_addr learned my addr 172.21.15.52:0/313549580 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf8c00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4108960 0x7efbf40757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3400700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4103320 0x7efbf4075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 -- 172.21.15.52:0/313549580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4103320 msgr2=0x7efbf4075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4103320 0x7efbf4075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 -- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4108960 msgr2=0x7efbf40757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4108960 0x7efbf40757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 -- 172.21.15.52:0/313549580 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf41a7590 con 0x7efbf4102960 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.530+0000 7efbf3e00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efbe400eb30 tx=0x7efbe400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.531+0000 7efbf0c00700 1 -- 172.21.15.52:0/313549580 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe400cbd0 con 0x7efbf4102960 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.531+0000 7efbf0c00700 1 -- 172.21.15.52:0/313549580 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbe40044d0 con 0x7efbf4102960 2024-04-26T01:44:48.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.531+0000 7efbf0c00700 1 -- 172.21.15.52:0/313549580 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4003e30 con 0x7efbf4102960 2024-04-26T01:44:48.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.531+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf41a7880 con 0x7efbf4102960 2024-04-26T01:44:48.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.531+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf41a7de0 con 0x7efbf4102960 2024-04-26T01:44:48.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.534+0000 7efbf0c00700 1 -- 172.21.15.52:0/313549580 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efbe4004000 con 0x7efbf4102960 2024-04-26T01:44:48.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.536+0000 7efc055f5b80 1 Processor -- start 2024-04-26T01:44:48.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 -- start start 2024-04-26T01:44:48.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efbf0c00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 0x7efbd80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efbf0c00700 1 -- 172.21.15.52:0/313549580 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7efbe40d06a0 con 0x7efbf4102960 2024-04-26T01:44:48.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efbf8c00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 0x7efbd80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bef8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745befb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55745be40e40 con 0x55745beed8b0 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55745bde76e0 con 0x55745beee1d0 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efc055f5b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55745be3ce00 con 0x55745bef9080 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.537+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55860/0 (socket says 172.21.15.52:55860) 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 -- 172.21.15.52:0/2840439170 learned_addr learned my addr 172.21.15.52:0/2840439170 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf8c00700 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745befb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3400700 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bef8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf8c00700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 0x7efbd80a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efbec00f520 tx=0x7efbec009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 -- 172.21.15.52:0/2840439170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 msgr2=0x55745befb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745befb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 -- 172.21.15.52:0/2840439170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 msgr2=0x55745bef8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bef8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 -- 172.21.15.52:0/2840439170 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745bddd630 con 0x55745beed8b0 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.538+0000 7efbf3e00700 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efbe4099300 tx=0x7efbe4098560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ac8a1d848fcb8b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.539+0000 7efbe9600700 1 -- 172.21.15.52:0/2840439170 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe409c470 con 0x55745beed8b0 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.539+0000 7efbe9600700 1 -- 172.21.15.52:0/2840439170 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbe40042f0 con 0x55745beed8b0 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.539+0000 7efbe9600700 1 -- 172.21.15.52:0/2840439170 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe409c290 con 0x55745beed8b0 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 -- 172.21.15.52:0/2840439170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 msgr2=0x55745beedc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efbe4099300 tx=0x7efbe4098560 comp rx=0 tx=0).stop 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 -- 172.21.15.52:0/2840439170 shutdown_connections 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745befb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beed8b0 0x55745beedc90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 --2- 172.21.15.52:0/2840439170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bef8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 -- 172.21.15.52:0/2840439170 >> 172.21.15.52:0/2840439170 conn(0x55745be1cf00 msgr2=0x55745bee9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 -- 172.21.15.52:0/2840439170 shutdown_connections 2024-04-26T01:44:48.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.540+0000 7efc055f5b80 1 -- 172.21.15.52:0/2840439170 wait complete. 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.541+0000 7efc055f5b80 1 -- start start 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.541+0000 7efc055f5b80 1 Processor -- start 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.541+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beed8b0 0x55745bef8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.541+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745bef6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efc055f5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55745befbc00 con 0x55745beee1d0 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efc055f5b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55745bdfa810 con 0x55745beed8b0 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efc055f5b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55745befbd80 con 0x55745bef9080 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efc055f5b80 10 client.0 _ll_register_callbacks cb 0x55745bdd96d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beed8b0 0x55745bef8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55876/0 (socket says 172.21.15.52:55876) 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 -- 172.21.15.52:0/431260835 learned_addr learned my addr 172.21.15.52:0/431260835 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf8c00700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745bef6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 -- 172.21.15.52:0/431260835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 msgr2=0x55745bef6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745bef6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 -- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beed8b0 msgr2=0x55745bef8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beed8b0 0x55745bef8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.542+0000 7efbf3400700 1 -- 172.21.15.52:0/431260835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55745bd8e880 con 0x55745beee1d0 2024-04-26T01:44:48.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.543+0000 7efbf3400700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009990 tx=0x7efbe0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baae34bd1b7f2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.543+0000 7efbde000700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:44:48.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.543+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe0029060 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.543+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7efbe000bd00 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.543+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbe40042f0 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efbe0003680 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 10 client.14907 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55745bdd7aa0 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbe40042f0 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7efbe000f920 con 0x55745beee1d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 10 client.14907 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 10 client.14907 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 20 client.14907 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.544+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf088d0 0x55745bf09d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.545+0000 7efc055f5b80 4 client.14907 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.545+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55745be4e630 con 0x55745bf088d0 2024-04-26T01:44:48.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.545+0000 7efbf3e00700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf088d0 0x55745bf09d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.545+0000 7efbf3e00700 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf088d0 0x55745bf09d40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62c4dc4749a1af14 server_cookie=90a167dbe6cff97e in_seq=0 out_seq=0 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.545+0000 7efbde000700 10 client.14907 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.546+0000 7efbde000700 1 -- 172.21.15.52:0/431260835 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55745be4e630 con 0x55745bf088d0 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.546+0000 7efbde000700 10 client.14907 handle_command_reply: tid=0 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.546+0000 7efc055f5b80 1 client.14907 shutdown 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 msgr2=0x55745bef5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009990 tx=0x7efbe0005800 comp rx=0 tx=0).stop 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 shutdown_connections 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf088d0 0x55745bf09d40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745bef9080 0x55745bef6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beed8b0 0x55745bef8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 --2- 172.21.15.52:0/431260835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745beee1d0 0x55745bef5b20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efbde000700 0 client.14907 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:44:48.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 >> 172.21.15.52:0/431260835 conn(0x55745be1cf00 msgr2=0x55745bee9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 shutdown_connections 2024-04-26T01:44:48.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 1 -- 172.21.15.52:0/431260835 wait complete. 2024-04-26T01:44:48.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.547+0000 7efc055f5b80 20 client.14907 trim_cache size 0 max 16384 2024-04-26T01:44:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 Processor -- start 2024-04-26T01:44:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 -- start start 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55745bf08080 con 0x55745bef9080 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55745bf08750 con 0x55745beee1d0 2024-04-26T01:44:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.570+0000 7efc055f5b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55745bf08e20 con 0x55745beed8b0 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf8c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44344/0 (socket says 172.21.15.52:44344) 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54620/0 (socket says 172.21.15.52:54620) 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf8c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf136f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55884/0 (socket says 172.21.15.52:55884) 2024-04-26T01:44:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 -- 172.21.15.52:0/3439838349 learned_addr learned my addr 172.21.15.52:0/3439838349 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 -- 172.21.15.52:0/3439838349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 msgr2=0x55745bf131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 -- 172.21.15.52:0/3439838349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 msgr2=0x55745bf136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.571+0000 7efbf3e00700 1 -- 172.21.15.52:0/3439838349 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745beef410 con 0x55745beed8b0 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.572+0000 7efbf3e00700 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efbe4098610 tx=0x7efbe40bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16bc0aa74bcf23e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.572+0000 7efbe8c00700 1 -- 172.21.15.52:0/3439838349 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe40042f0 con 0x55745beed8b0 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.572+0000 7efbe8c00700 1 -- 172.21.15.52:0/3439838349 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbe409c930 con 0x55745beed8b0 2024-04-26T01:44:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.572+0000 7efbe8c00700 1 -- 172.21.15.52:0/3439838349 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe409ac30 con 0x55745beed8b0 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 -- 172.21.15.52:0/3439838349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 msgr2=0x55745bef61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efbe4098610 tx=0x7efbe40bd040 comp rx=0 tx=0).stop 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 -- 172.21.15.52:0/3439838349 shutdown_connections 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bef61f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 --2- 172.21.15.52:0/3439838349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 -- 172.21.15.52:0/3439838349 >> 172.21.15.52:0/3439838349 conn(0x55745be1cf00 msgr2=0x55745bef5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 -- 172.21.15.52:0/3439838349 shutdown_connections 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.573+0000 7efc055f5b80 1 -- 172.21.15.52:0/3439838349 wait complete. 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.574+0000 7efc055f5b80 1 -- start start 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.574+0000 7efc055f5b80 1 Processor -- start 2024-04-26T01:44:48.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bf0f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55745bf09bc0 con 0x55745bef9080 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55745bf094f0 con 0x55745beee1d0 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55745bf0a290 con 0x55745beed8b0 2024-04-26T01:44:48.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efc055f5b80 10 client.0 _ll_register_callbacks cb 0x55745bdd96d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efbf8c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efbf3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bf0f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efbf3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efbf3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54624/0 (socket says 172.21.15.52:54624) 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.575+0000 7efbf8c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55890/0 (socket says 172.21.15.52:55890) 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 -- 172.21.15.52:0/990729172 learned_addr learned my addr 172.21.15.52:0/990729172 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 -- 172.21.15.52:0/990729172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 msgr2=0x55745bf0f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bf0f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 -- 172.21.15.52:0/990729172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 msgr2=0x55745bf6ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 -- 172.21.15.52:0/990729172 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55745bf6ef40 con 0x55745beee1d0 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3e00700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bf0f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf8c00700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbf3400700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efbe00055a0 tx=0x7efbe0005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5860791be9ddb758 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.576+0000 7efbde000700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe0034040 con 0x55745beee1d0 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7efbe000bc00 con 0x55745beee1d0 2024-04-26T01:44:48.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55745be486d0 con 0x55745beee1d0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efbe0029040 con 0x55745beee1d0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efc055f5b80 10 client.25010 fetch_fsmap learned FSMap version 24 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.577+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55745bf70220 con 0x55745beee1d0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55745be486d0 con 0x55745beee1d0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7efbe000f6d0 con 0x55745beee1d0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 10 client.25010 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 10 client.25010 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 20 client.25010 populate_metadata read hostname 'smithi052' 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf74d80 0x55745bf73a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 4 client.25010 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "status"}) -- 0x55745bcc1200 con 0x55745bf74d80 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.578+0000 7efbf3e00700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf74d80 0x55745bf73a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:48.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.579+0000 7efbde000700 10 client.25010 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:44:48.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.579+0000 7efbf3e00700 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf74d80 0x55745bf73a90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e5bd3aa24100f8 server_cookie=48a84e6a20c839c4 in_seq=0 out_seq=0 2024-04-26T01:44:48.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.579+0000 7efbde000700 1 -- 172.21.15.52:0/990729172 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55745bcc1200 con 0x55745bf74d80 2024-04-26T01:44:48.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.579+0000 7efbde000700 10 client.25010 handle_command_reply: tid=0 2024-04-26T01:44:48.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.579+0000 7efc055f5b80 1 client.25010 shutdown 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 msgr2=0x55745bf0f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efbe00055a0 tx=0x7efbe0005a40 comp rx=0 tx=0).stop 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 shutdown_connections 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55745bf74d80 0x55745bf73a90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55745bef9080 0x55745bf6ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55745beed8b0 0x55745bf0f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 --2- 172.21.15.52:0/990729172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55745beee1d0 0x55745bf0f7b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 >> 172.21.15.52:0/990729172 conn(0x55745be1cf00 msgr2=0x55745bef5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efbde000700 0 client.25010 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 shutdown_connections 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 1 -- 172.21.15.52:0/990729172 wait complete. 2024-04-26T01:44:48.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.580+0000 7efc055f5b80 20 client.25010 trim_cache size 0 max 16384 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24524, 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 221.2808258, 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 24, 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 94, 2024-04-26T01:44:48.580 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T01:44:48.581 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 242.65135543700001 2024-04-26T01:44:48.581 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:44:48.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 msgr2=0x7efbd80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 0x7efbd80a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efbec00f520 tx=0x7efbec009920 comp rx=0 tx=0).stop 2024-04-26T01:44:48.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 msgr2=0x7efbf4074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efbe400eb30 tx=0x7efbe400ee40 comp rx=0 tx=0).stop 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 shutdown_connections 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7efbd80a3cc0 0x7efbd80a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efbf4108960 0x7efbf40757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efbf4102960 0x7efbf4074d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 --2- 172.21.15.52:0/313549580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efbf4103320 0x7efbf4075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 >> 172.21.15.52:0/313549580 conn(0x7efbf40fe450 msgr2=0x7efbf40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.581+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 shutdown_connections 2024-04-26T01:44:48.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:48.582+0000 7efbf9600700 1 -- 172.21.15.52:0/313549580 wait complete. 2024-04-26T01:44:49.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:49 smithi052 ceph-mon[27851]: pgmap v445: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:49.595 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T01:44:49.605 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T01:44:49.605 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T01:44:49.605 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T01:44:49.605 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T01:44:49.606 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:44:49.606 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:44:49.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T01:44:49.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:49 smithi191 ceph-mon[32745]: pgmap v445: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:49.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:49 smithi183 ceph-mon[32752]: pgmap v445: 97 pgs: 97 active+clean; 644 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T01:44:49.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 -- 172.21.15.52:0/3836209026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 msgr2=0x7f14cc10a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 --2- 172.21.15.52:0/3836209026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc10a420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009900 tx=0x7f14c402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 -- 172.21.15.52:0/3836209026 shutdown_connections 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 --2- 172.21.15.52:0/3836209026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc10a420 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 --2- 172.21.15.52:0/3836209026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc1031d0 0x7f14cc103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 --2- 172.21.15.52:0/3836209026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc101fe0 0x7f14cc102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 -- 172.21.15.52:0/3836209026 >> 172.21.15.52:0/3836209026 conn(0x7f14cc0fd570 msgr2=0x7f14cc0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 -- 172.21.15.52:0/3836209026 shutdown_connections 2024-04-26T01:44:49.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.802+0000 7f14d2c00640 1 -- 172.21.15.52:0/3836209026 wait complete. 2024-04-26T01:44:49.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.803+0000 7f14d2c00640 1 Processor -- start 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.803+0000 7f14d2c00640 1 -- start start 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc101fe0 0x7f14cc19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc1031d0 0x7f14cc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10db20 con 0x7f14cc103b90 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10d9a0 con 0x7f14cc1031d0 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d2c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10dca0 con 0x7f14cc101fe0 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14cbe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc101fe0 0x7f14cc19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55916/0 (socket says 172.21.15.52:55916) 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 -- 172.21.15.52:0/1737287099 learned_addr learned my addr 172.21.15.52:0/1737287099 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14cb400640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc1031d0 0x7f14cc19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 -- 172.21.15.52:0/1737287099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc101fe0 msgr2=0x7f14cc19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc101fe0 0x7f14cc19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 -- 172.21.15.52:0/1737287099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc1031d0 msgr2=0x7f14cc19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc1031d0 0x7f14cc19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.804+0000 7f14d0c00640 1 -- 172.21.15.52:0/1737287099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1a1f60 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14d0c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f14c4031500 tx=0x7f14c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c403f030 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14c4003cb0 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1a21f0 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4035070 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.805+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a26e0 con 0x7f14cc103b90 2024-04-26T01:44:49.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.807+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc1a2940 con 0x7f14cc103b90 2024-04-26T01:44:49.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.808+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f14c4042030 con 0x7f14cc103b90 2024-04-26T01:44:49.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.811+0000 7f14c8c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 0x7f14b00a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:49.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.811+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f14c40f0830 con 0x7f14cc103b90 2024-04-26T01:44:49.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.811+0000 7f14cbe00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 0x7f14b00a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:49.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.812+0000 7f14cbe00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 0x7f14b00a6110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14cc103030 tx=0x7f14bc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:49.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:49.816+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14c40bcb70 con 0x7f14cc103b90 2024-04-26T01:44:50.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.077+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f14cc1a2b10 con 0x7f14cc103b90 2024-04-26T01:44:50.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.283+0000 7f14c8c00640 1 -- 172.21.15.52:0/1737287099 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f14c40c0e70 con 0x7f14cc103b90 2024-04-26T01:44:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.289+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 msgr2=0x7f14b00a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.289+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 0x7f14b00a6110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14cc103030 tx=0x7f14bc009500 comp rx=0 tx=0).stop 2024-04-26T01:44:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.289+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 msgr2=0x7f14cc1a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.289+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f14c4031500 tx=0x7f14c40042f0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 shutdown_connections 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f14b00a3c50 0x7f14b00a6110 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f14cc103b90 0x7f14cc1a18f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14cc1031d0 0x7f14cc19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 --2- 172.21.15.52:0/1737287099 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14cc101fe0 0x7f14cc19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 >> 172.21.15.52:0/1737287099 conn(0x7f14cc0fd570 msgr2=0x7f14cc074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.291+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 shutdown_connections 2024-04-26T01:44:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.293+0000 7f14d2c00640 1 -- 172.21.15.52:0/1737287099 wait complete. 2024-04-26T01:44:50.315 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 -- 172.21.15.52:0/429787585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14102a30 msgr2=0x7f8c14104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 --2- 172.21.15.52:0/429787585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14102a30 0x7f8c14104e20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009ca0 tx=0x7f8c0c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 -- 172.21.15.52:0/429787585 shutdown_connections 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 --2- 172.21.15.52:0/429787585 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 0x7f8c1410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 --2- 172.21.15.52:0/429787585 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14105360 0x7f8c14107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 --2- 172.21.15.52:0/429787585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14102a30 0x7f8c14104e20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 -- 172.21.15.52:0/429787585 >> 172.21.15.52:0/429787585 conn(0x7f8c140fc420 msgr2=0x7f8c140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:50.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 -- 172.21.15.52:0/429787585 shutdown_connections 2024-04-26T01:44:50.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.519+0000 7f8c1a800640 1 -- 172.21.15.52:0/429787585 wait complete. 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.520+0000 7f8c1a800640 1 Processor -- start 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.520+0000 7f8c1a800640 1 -- start start 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14102a30 0x7f8c141023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 0x7f8c14100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1410daf0 con 0x7f8c14102a30 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1410d970 con 0x7f8c14107c90 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c1a800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1410dc70 con 0x7f8c14105360 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44414/0 (socket says 172.21.15.52:44414) 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c13400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14102a30 0x7f8c141023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 -- 172.21.15.52:0/1880672853 learned_addr learned my addr 172.21.15.52:0/1880672853 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c13e00640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 0x7f8c14100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 msgr2=0x7f8c14100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 0x7f8c14100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14102a30 msgr2=0x7f8c141023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14102a30 0x7f8c141023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.521+0000 7f8c12a00640 1 -- 172.21.15.52:0/1880672853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c141017b0 con 0x7f8c14105360 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c12a00640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000e9c0 tx=0x7f8c0000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000cd80 con 0x7f8c14105360 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c000054d0 con 0x7f8c14105360 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c141a2250 con 0x7f8c14105360 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00010430 con 0x7f8c14105360 2024-04-26T01:44:50.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.522+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c141a27a0 con 0x7f8c14105360 2024-04-26T01:44:50.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.523+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c140fdf10 con 0x7f8c14105360 2024-04-26T01:44:50.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.526+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f8c00004fd0 con 0x7f8c14105360 2024-04-26T01:44:50.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.527+0000 7f8c0be00640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 0x7f8bf80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.527+0000 7f8c13400640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 msgr2=0x7f8bf80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:50.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.527+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f8c000d08d0 con 0x7f8c14105360 2024-04-26T01:44:50.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.527+0000 7f8c13400640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 0x7f8bf80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:44:50.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.531+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c0009d260 con 0x7f8c14105360 2024-04-26T01:44:50.613 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:50 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:44:50.614 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setuser ceph since I am not root 2024-04-26T01:44:50.614 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: ignoring --setgroup ceph since I am not root 2024-04-26T01:44:50.614 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:50.615+0000 7f178960d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:44:50.644 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:44:50.644 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: ignoring --setuser ceph since I am not root 2024-04-26T01:44:50.644 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: ignoring --setgroup ceph since I am not root 2024-04-26T01:44:50.644 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-mgr[34142]: -- 172.21.15.183:0/2608459361 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffd0ca25a0 con 0x55ffd0c76c00 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:50.463+0000 7facbe0a6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:50.646+0000 7facbe0a6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: ignoring --setuser ceph since I am not root 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: ignoring --setgroup ceph since I am not root 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-mgr[34612]: -- 172.21.15.191:0/4251567000 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f87288a5a0 con 0x55f87285f000 2024-04-26T01:44:50.645 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:50.465+0000 7f69be3c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:44:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.728+0000 7f8c13400640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 msgr2=0x7f8bf80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.728+0000 7f8c13400640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 0x7f8bf80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:44:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.729+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f8c14062b50 con 0x7f8c14105360 2024-04-26T01:44:50.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.737+0000 7f8c0be00640 1 -- 172.21.15.52:0/1880672853 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f8c0009cb60 con 0x7f8c14105360 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.739+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 msgr2=0x7f8bf80a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.739+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 0x7f8bf80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.739+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 msgr2=0x7f8c141009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.739+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000e9c0 tx=0x7f8c0000ee80 comp rx=0 tx=0).stop 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 shutdown_connections 2024-04-26T01:44:50.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f8bf80a3ca0 0x7f8bf80a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c14107c90 0x7f8c14100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c14105360 0x7f8c141009e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 --2- 172.21.15.52:0/1880672853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c14102a30 0x7f8c141023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 >> 172.21.15.52:0/1880672853 conn(0x7f8c140fc420 msgr2=0x7f8c141032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 shutdown_connections 2024-04-26T01:44:50.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.740+0000 7f8c1a800640 1 -- 172.21.15.52:0/1880672853 wait complete. 2024-04-26T01:44:50.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T01:44:50.901 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:50.647+0000 7f69be3c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:50.901 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:50.749+0000 7f69be3c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:44:50.924 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:50.750+0000 7facbe0a6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 -- 172.21.15.52:0/2950362253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78074950 msgr2=0x7fbd78074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 --2- 172.21.15.52:0/2950362253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78074950 0x7fbd78074db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009900 tx=0x7fbd6002a480 comp rx=0 tx=0).stop 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 -- 172.21.15.52:0/2950362253 shutdown_connections 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 --2- 172.21.15.52:0/2950362253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78109c70 0x7fbd7810c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 --2- 172.21.15.52:0/2950362253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78074950 0x7fbd78074db0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 --2- 172.21.15.52:0/2950362253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78075f30 0x7fbd78074380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 -- 172.21.15.52:0/2950362253 >> 172.21.15.52:0/2950362253 conn(0x7fbd780fd350 msgr2=0x7fbd780ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 -- 172.21.15.52:0/2950362253 shutdown_connections 2024-04-26T01:44:50.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.948+0000 7fbd77200640 1 -- 172.21.15.52:0/2950362253 wait complete. 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.949+0000 7fbd77200640 1 Processor -- start 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.949+0000 7fbd77200640 1 -- start start 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.949+0000 7fbd77200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.949+0000 7fbd77200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 0x7fbd7819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.949+0000 7fbd77200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 0x7fbd781a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd77200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810fb70 con 0x7fbd78109c70 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd77200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810f9f0 con 0x7fbd78074950 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd77200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810fcf0 con 0x7fbd78075f30 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd6fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 0x7fbd7819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54682/0 (socket says 172.21.15.52:54682) 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd76800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 0x7fbd781a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:50.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 -- 172.21.15.52:0/3508592230 learned_addr learned my addr 172.21.15.52:0/3508592230 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 msgr2=0x7fbd7819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 0x7fbd7819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 msgr2=0x7fbd781a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 0x7fbd781a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd75e00640 1 -- 172.21.15.52:0/3508592230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd781a1e30 con 0x7fbd78074950 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd6fe00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 0x7fbd7819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.950+0000 7fbd76800640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 0x7fbd781a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd75e00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800ea60 tx=0x7fbd6800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800cde0 con 0x7fbd78074950 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd680035f0 con 0x7fbd78074950 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd781a2120 con 0x7fbd78074950 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68010690 con 0x7fbd78074950 2024-04-26T01:44:50.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.951+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd781a2640 con 0x7fbd78074950 2024-04-26T01:44:50.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.952+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd78108a30 con 0x7fbd78074950 2024-04-26T01:44:50.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.956+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fbd680037a0 con 0x7fbd78074950 2024-04-26T01:44:50.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.957+0000 7fbd6ea00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 0x7fbd5c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:50.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.957+0000 7fbd6fe00640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 msgr2=0x7fbd5c0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:50.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.957+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7fbd680d0740 con 0x7fbd78074950 2024-04-26T01:44:50.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.957+0000 7fbd6fe00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 0x7fbd5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:44:50.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:50.961+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd6809d0d0 con 0x7fbd78074950 2024-04-26T01:44:51.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.158+0000 7fbd6fe00640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 msgr2=0x7fbd5c0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:51.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.158+0000 7fbd6fe00640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 0x7fbd5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:44:51.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.170+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fbd7802d5c0 con 0x7fbd78074950 2024-04-26T01:44:51.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.178+0000 7fbd6ea00640 1 -- 172.21.15.52:0/3508592230 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7fbd6809c9d0 con 0x7fbd78074950 2024-04-26T01:44:51.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.182+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 msgr2=0x7fbd5c0a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:44:51.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 0x7fbd5c0a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:51.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 msgr2=0x7fbd7819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800ea60 tx=0x7fbd6800ef20 comp rx=0 tx=0).stop 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 shutdown_connections 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7fbd5c0a3bf0 0x7fbd5c0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd78109c70 0x7fbd781a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd78075f30 0x7fbd7819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 --2- 172.21.15.52:0/3508592230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd78074950 0x7fbd7819a210 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 >> 172.21.15.52:0/3508592230 conn(0x7fbd780fd350 msgr2=0x7fbd78105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 shutdown_connections 2024-04-26T01:44:51.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:51.183+0000 7fbd77200640 1 -- 172.21.15.52:0/3508592230 wait complete. 2024-04-26T01:44:51.202 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:50.865+0000 7f178960d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:51.202 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:51.007+0000 7f178960d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:44:51.205 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T01:44:51.227 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T01:44:51.227 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T01:44:51.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:51.227 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-26T01:44:51.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:51.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-04-26T01:44:51.363 INFO:teuthology.orchestra.run.smithi052.stderr:Cannot find device "ceph-brx" 2024-04-26T01:44:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:44:51.366 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-26T01:44:51.366 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T01:44:51.366 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T01:44:51.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T01:44:51.401 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:51.052+0000 7f69be3c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:44:51.401 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:51.135+0000 7f69be3c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:44:51.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:44:51.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-mon[32745]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:44:51.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:51 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:51.057+0000 7facbe0a6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:51.142+0000 7facbe0a6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-mon[32752]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:51 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:51.435+0000 7f178960d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:44:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1737287099' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T01:44:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-mon[27851]: mgrmap e19: x(active, since 12m), standbys: y, z 2024-04-26T01:44:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' 2024-04-26T01:44:51.811 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:44:51.961 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:51 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:51.554+0000 7f178960d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:44:52.634 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:52.301+0000 7f69be3c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:44:52.635 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:52.467+0000 7f69be3c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:52.666 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:52.328+0000 7facbe0a6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:44:52.666 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:52.497+0000 7facbe0a6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:52.666 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:52.668+0000 7facbe0a6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:44:52.901 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:52.637+0000 7f69be3c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:44:53.210 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:52.920+0000 7f69be3c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:44:53.210 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:53.010+0000 7f69be3c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:44:53.249 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:52.957+0000 7facbe0a6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:44:53.250 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:53.048+0000 7facbe0a6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:44:53.250 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:53.252+0000 7facbe0a6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:44:53.422 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:53.202+0000 7f178960d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:44:53.651 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:53.212+0000 7f69be3c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:44:53.651 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:53.403+0000 7f69be3c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:53.674 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:53.447+0000 7facbe0a6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:53.676 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:53.445+0000 7f178960d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:53.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.943+0000 7f1b3e400700 1 -- 172.21.15.52:0/432828289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 msgr2=0x7f1b3806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:53.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.943+0000 7f1b3e400700 1 --2- 172.21.15.52:0/432828289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b3806abb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009990 tx=0x7f1b24009ca0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.942 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:53 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:53.677+0000 7f178960d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:44:53.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 -- 172.21.15.52:0/432828289 shutdown_connections 2024-04-26T01:44:53.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 --2- 172.21.15.52:0/432828289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b3806b0f0 0x7f1b3810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 --2- 172.21.15.52:0/432828289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b38108a30 0x7f1b38108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 --2- 172.21.15.52:0/432828289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b3806abb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 -- 172.21.15.52:0/432828289 >> 172.21.15.52:0/432828289 conn(0x7f1b380788f0 msgr2=0x7f1b38078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 -- 172.21.15.52:0/432828289 shutdown_connections 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.944+0000 7f1b3e400700 1 -- 172.21.15.52:0/432828289 wait complete. 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.945+0000 7f1b3e400700 1 Processor -- start 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.945+0000 7f1b3e400700 1 -- start start 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b3806b0f0 0x7f1b38101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b38108a30 0x7f1b38102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38112c10 con 0x7f1b3806b0f0 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38112a90 con 0x7f1b38108a30 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b3e400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38112d90 con 0x7f1b3806a750 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:53.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44446/0 (socket says 172.21.15.52:44446) 2024-04-26T01:44:53.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 -- 172.21.15.52:0/2692427630 learned_addr learned my addr 172.21.15.52:0/2692427630 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:53.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b36a00700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b3806b0f0 0x7f1b38101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:53.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37e00700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b38108a30 0x7f1b38102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b38108a30 msgr2=0x7f1b38102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b38108a30 0x7f1b38102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b3806b0f0 msgr2=0x7f1b38101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b3806b0f0 0x7f1b38101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.946+0000 7f1b37400700 1 -- 172.21.15.52:0/2692427630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b381a75f0 con 0x7f1b3806a750 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b37400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800eb30 tx=0x7f1b2800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800cbd0 con 0x7f1b3806a750 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b280044d0 con 0x7f1b3806a750 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28003e30 con 0x7f1b3806a750 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381a78e0 con 0x7f1b3806a750 2024-04-26T01:44:53.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.947+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381a7e40 con 0x7f1b3806a750 2024-04-26T01:44:53.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.948+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b38069730 con 0x7f1b3806a750 2024-04-26T01:44:53.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.952+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f1b28004000 con 0x7f1b3806a750 2024-04-26T01:44:53.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.953+0000 7f1b2fe00700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 0x7f1b1c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:53.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.953+0000 7f1b36a00700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 msgr2=0x7f1b1c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:53.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.953+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f1b280d08a0 con 0x7f1b3806a750 2024-04-26T01:44:53.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.953+0000 7f1b36a00700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 0x7f1b1c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:44:53.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:53.954+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b280042f0 con 0x7f1b3806a750 2024-04-26T01:44:54.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.154+0000 7f1b36a00700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 msgr2=0x7f1b1c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:54.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.154+0000 7f1b36a00700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 0x7f1b1c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:44:54.211 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:54.087+0000 7f178960d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:44:54.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.240+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1b3802df80 con 0x7f1b3806a750 2024-04-26T01:44:54.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.241+0000 7f1b2fe00700 1 -- 172.21.15.52:0/2692427630 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+73 (secure 0 0 0) 0x7f1b2809d230 con 0x7f1b3806a750 2024-04-26T01:44:54.238 INFO:teuthology.orchestra.run.smithi052.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T01:44:54.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 msgr2=0x7f1b1c0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 0x7f1b1c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 msgr2=0x7f1b38101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800eb30 tx=0x7f1b2800ee40 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 shutdown_connections 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f1b1c0a3d10 0x7f1b1c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b38108a30 0x7f1b38102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b3806a750 0x7f1b38101a10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 --2- 172.21.15.52:0/2692427630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b3806b0f0 0x7f1b38101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 >> 172.21.15.52:0/2692427630 conn(0x7f1b380788f0 msgr2=0x7f1b380ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 shutdown_connections 2024-04-26T01:44:54.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:54.244+0000 7f1b3e400700 1 -- 172.21.15.52:0/2692427630 wait complete. 2024-04-26T01:44:54.290 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:53.994+0000 7f69be3c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:44:54.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:54 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2692427630' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:44:54.424 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:54.047+0000 7facbe0a6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:44:54.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:54 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2692427630' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:44:54.481 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:54.215+0000 7f178960d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:44:54.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:54 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2692427630' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T01:44:54.652 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:54.409+0000 7f69be3c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:44:54.652 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:54.504+0000 7f69be3c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:44:54.749 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:54.483+0000 7f178960d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:44:54.765 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:54.470+0000 7facbe0a6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:44:54.766 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:54.566+0000 7facbe0a6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:44:54.766 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:54.767+0000 7facbe0a6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:44:55.089 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:54.751+0000 7f178960d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi052.front.sepia.ceph.com 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T01:44:55.144 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T01:44:55.145 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T01:44:55.145 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T01:44:55.145 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T01:44:55.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:55.145 DEBUG:teuthology.orchestra.run.smithi052:> ip addr 2024-04-26T01:44:55.151 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:54.705+0000 7f69be3c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:44:55.199 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 41858sec preferred_lft 41858sec 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2024-04-26T01:44:55.200 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2024-04-26T01:44:55.201 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add name ceph-brx type bridge 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr flush dev ceph-brx 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set ceph-brx up 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T01:44:55.202 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-26T01:44:55.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:55.560 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.231+0000 7f69be3c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:44:55.560 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.324+0000 7f69be3c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:44:55.634 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.298+0000 7facbe0a6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:44:55.634 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.393+0000 7facbe0a6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:44:55.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:55.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:55.675 DEBUG:teuthology.orchestra.run.smithi052:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T01:44:55.738 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-04-26T01:44:55.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:55.742 DEBUG:teuthology.orchestra.run.smithi052:> route 2024-04-26T01:44:55.797 INFO:teuthology.orchestra.run.smithi052.stdout:Kernel IP routing table 2024-04-26T01:44:55.797 INFO:teuthology.orchestra.run.smithi052.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T01:44:55.798 INFO:teuthology.orchestra.run.smithi052.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T01:44:55.798 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T01:44:55.798 INFO:teuthology.orchestra.run.smithi052.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T01:44:55.798 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:44:55.798 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-26T01:44:55.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T01:44:55.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T01:44:55.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T01:44:55.799 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-26T01:44:55.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:55.897 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.562+0000 7f69be3c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:44:55.897 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.693+0000 7f69be3c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:44:55.897 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.785+0000 7f69be3c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:44:55.897 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:55.899+0000 7f69be3c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:55.915 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:55.587+0000 7f178960d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T01:44:55.924 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.636+0000 7facbe0a6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:44:55.924 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.769+0000 7facbe0a6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:44:55.924 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.865+0000 7facbe0a6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:44:56.282 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:55.980+0000 7facbe0a6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:56.282 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:56.284+0000 7facbe0a6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:44:56.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:56.187+0000 7f178960d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:44:56.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:56.322+0000 7f178960d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:44:56.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:56.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:56.594 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-26T01:44:56.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:56.640 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list-id 2024-04-26T01:44:56.651 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:56.201+0000 7f69be3c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:44:56.734 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:44:56.734 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-26T01:44:56.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T01:44:56.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T01:44:56.734 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-26T01:44:56.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:56.856 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:56.605+0000 7f178960d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T01:44:56.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:56.913 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T01:44:56.914 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-26T01:44:57.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:57.075 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:56.793+0000 7f69be3c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:44:57.075 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z[34608]: 2024-04-26T01:44:57.005+0000 7f69be3c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:44:57.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: Standby manager daemon z restarted 2024-04-26T01:44:57.110 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: Standby manager daemon z started 2024-04-26T01:44:57.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:44:57.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:57.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:44:57.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: Standby manager daemon z restarted 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: Standby manager daemon z started 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:57.174 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:56.886+0000 7facbe0a6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:44:57.175 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 01:44:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y[34138]: 2024-04-26T01:44:57.107+0000 7facbe0a6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:44:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:57.207 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T01:44:57.207 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-04-26T01:44:57.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set brx.0 up 2024-04-26T01:44:57.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T01:44:57.207 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-04-26T01:44:57.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:57.366 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:57.305+0000 7f178960d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:44:57.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:57.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:44:57.400 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T01:44:57.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: Standby manager daemon z restarted 2024-04-26T01:44:57.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: Standby manager daemon z started 2024-04-26T01:44:57.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T01:44:57.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:57.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T01:44:57.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:57 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.191:0/1148240193' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:57.443 INFO:teuthology.orchestra.run.smithi052.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T01:44:57.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T01:44:57.443 DEBUG:teuthology.orchestra.run.smithi052:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T01:44:57.532 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T01:44:57.532 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T01:44:57.532 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/95eafff3-c868-445a-b0d3-cf6c55ba5d86 2024-04-26T01:44:57.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:57.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/95eafff3-c868-445a-b0d3-cf6c55ba5d86 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:44:57.636 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:57.448+0000 7f178960d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:44:57.717 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:44:57.717 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-26T01:44:57.717 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-04-26T01:44:57.961 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:57.813+0000 7f178960d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:44:58.311 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:44:58.311 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: Standby manager daemon y restarted 2024-04-26T01:44:58.311 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: Standby manager daemon y started 2024-04-26T01:44:58.311 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:44:58.311 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:58.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:44:58.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-mon[27851]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:58.312 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:58.010+0000 7f178960d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:44:58.312 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:58.153+0000 7f178960d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:44:58.312 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:58.313+0000 7f178960d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:44:58.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: Standby manager daemon y restarted 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: Standby manager daemon y started 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:44:58.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:58 smithi191 ceph-mon[32745]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: Standby manager daemon y restarted 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: Standby manager daemon y started 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T01:44:58.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:58 smithi183 ceph-mon[32752]: from='mgr.? 172.21.15.183:0/692737382' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.52:3300,172.21.15.183:3300,172.21.15.191:3300" 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.52:3300,172.21.15.183:3300,172.21.15.191:3300:/volumes/qa/sv_1/95eafff3-c868-445a-b0d3-cf6c55ba5d86 2024-04-26T01:44:58.533 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=8b150f36-036c-11ef-bc93-c7b262605968" will pass to kernel 2024-04-26T01:44:58.536 INFO:teuthology.orchestra.run.smithi052.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T01:44:58.536 INFO:teuthology.orchestra.run.smithi052.stdout: You just mounted a file system that supports labels which does not 2024-04-26T01:44:58.536 INFO:teuthology.orchestra.run.smithi052.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T01:44:58.536 INFO:teuthology.orchestra.run.smithi052.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T01:44:58.536 INFO:teuthology.orchestra.run.smithi052.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T01:44:58.541 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T01:44:58.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T01:44:58.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T01:44:58.621 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:44:58.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:44:58.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:58.741 INFO:teuthology.orchestra.run.smithi052.stdout:client14949 2024-04-26T01:44:58.741 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:44:58.741 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T01:44:58.787 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:44:58.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:44:58.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:58.949 INFO:teuthology.orchestra.run.smithi052.stdout:client14949 2024-04-26T01:44:58.949 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T01:44:58.960 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:58.699+0000 7f178960d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:44:59.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T01:44:59.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:59.057 INFO:teuthology.orchestra.run.smithi052.stdout:client14949 2024-04-26T01:44:59.057 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8b150f36-036c-11ef-bc93-c7b262605968.client14949/status') 2024-04-26T01:44:59.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T01:44:59.229 INFO:teuthology.orchestra.run.smithi052.stdout:instance: client.14949 (3)192.168.0.1:0/190953378 2024-04-26T01:44:59.229 INFO:teuthology.orchestra.run.smithi052.stdout:blocklisted: false 2024-04-26T01:44:59.229 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records in 2024-04-26T01:44:59.229 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records out 2024-04-26T01:44:59.229 INFO:teuthology.orchestra.run.smithi052.stderr:69 bytes copied, 7.3741e-05 s, 936 kB/s 2024-04-26T01:44:59.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T01:44:59.242 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:44:59.252 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:44:59.252 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:44:59.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T01:44:59.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:44:59 smithi191 ceph-mon[32745]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:44:59.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:44:59 smithi183 ceph-mon[32752]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:44:59.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 -- 172.21.15.52:0/1130575326 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b50075f90 msgr2=0x7f6b500743e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 --2- 172.21.15.52:0/1130575326 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b50075f90 0x7f6b500743e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009900 tx=0x7f6b4002a410 comp rx=0 tx=0).stop 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 -- 172.21.15.52:0/1130575326 shutdown_connections 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 --2- 172.21.15.52:0/1130575326 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b5010c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 --2- 172.21.15.52:0/1130575326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b500749b0 0x7f6b50074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 --2- 172.21.15.52:0/1130575326 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b50075f90 0x7f6b500743e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 -- 172.21.15.52:0/1130575326 >> 172.21.15.52:0/1130575326 conn(0x7f6b500fd350 msgr2=0x7f6b500ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 -- 172.21.15.52:0/1130575326 shutdown_connections 2024-04-26T01:44:59.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.449+0000 7f6b55a00640 1 -- 172.21.15.52:0/1130575326 wait complete. 2024-04-26T01:44:59.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 Processor -- start 2024-04-26T01:44:59.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 -- start start 2024-04-26T01:44:59.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 0x7f6b5019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b50075f90 0x7f6b5019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5010fad0 con 0x7f6b500749b0 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.450+0000 7f6b55a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5010f950 con 0x7f6b50075f90 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5010fc50 con 0x7f6b50109ca0 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 0x7f6b5019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50372/0 (socket says 172.21.15.52:50372) 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 -- 172.21.15.52:0/3629157429 learned_addr learned my addr 172.21.15.52:0/3629157429 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:44:59.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b4f400640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b50075f90 0x7f6b5019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:44:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b50075f90 msgr2=0x7f6b5019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b50075f90 0x7f6b5019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 msgr2=0x7f6b5019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:44:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 0x7f6b5019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:44:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b55000640 1 -- 172.21.15.52:0/3629157429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b501a1e20 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.451+0000 7f6b4fe00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 0x7f6b5019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b55000640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800eb10 tx=0x7f6b4800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48004b20 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b48004cc0 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b501a2110 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010670 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.452+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b501a2610 con 0x7f6b50109ca0 2024-04-26T01:44:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.453+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b501089d0 con 0x7f6b50109ca0 2024-04-26T01:44:59.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.455+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7f6b48005020 con 0x7f6b50109ca0 2024-04-26T01:44:59.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.457+0000 7f6b4cc00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 0x7f6b340a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:44:59.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.457+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8219+0+0 (secure 0 0 0) 0x7f6b480d0a60 con 0x7f6b50109ca0 2024-04-26T01:44:59.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.457+0000 7f6b4fe00640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 msgr2=0x7f6b340a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:59.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.457+0000 7f6b4fe00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 0x7f6b340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T01:44:59.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.461+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b4809d3f0 con 0x7f6b50109ca0 2024-04-26T01:44:59.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:44:59 smithi052 ceph-mon[27851]: mgrmap e21: x(active, since 12m), standbys: y, z 2024-04-26T01:44:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.658+0000 7f6b4fe00640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 msgr2=0x7f6b340a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/2415060450 2024-04-26T01:44:59.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.658+0000 7f6b4fe00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 0x7f6b340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T01:44:59.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.708+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f6b50062b50 con 0x7f6b50109ca0 2024-04-26T01:44:59.811 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:59.510+0000 7f178960d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:44:59.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.838+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f6b48099360 con 0x7f6b50109ca0 2024-04-26T01:44:59.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.839+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 msgr2=0x7f6b340a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T01:44:59.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:44:59.839+0000 7f6b4cc00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 0x7f6b340a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.091 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:44:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:44:59.813+0000 7f178960d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:45:00.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.103+0000 7f6b4cc00640 1 -- 172.21.15.52:0/3629157429 <== mon.1 v2:172.21.15.191:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f6b48058d40 con 0x7f6b50109ca0 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 msgr2=0x7f6b501a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800eb10 tx=0x7f6b4800c2d0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 shutdown_connections 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:6800/2415060450,v1:172.21.15.52:6801/2415060450] conn(0x7f6b340a3bf0 0x7f6b340a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b50109ca0 0x7f6b501a17b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.106+0000 7f6b55a00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b50075f90 0x7f6b5019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.107+0000 7f6b55a00640 1 --2- 172.21.15.52:0/3629157429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b500749b0 0x7f6b5019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.107+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 >> 172.21.15.52:0/3629157429 conn(0x7f6b500fd350 msgr2=0x7f6b50105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:00.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.107+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 shutdown_connections 2024-04-26T01:45:00.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.107+0000 7f6b55a00640 1 -- 172.21.15.52:0/3629157429 wait complete. 2024-04-26T01:45:00.127 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T01:45:00.137 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T01:45:00.149 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T01:45:00.149 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:45:00.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T01:45:00.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 -- 172.21.15.52:0/3667044138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64105180 msgr2=0x7faf64107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 --2- 172.21.15.52:0/3667044138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64105180 0x7faf64107570 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf50009900 tx=0x7faf5002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 -- 172.21.15.52:0/3667044138 shutdown_connections 2024-04-26T01:45:00.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 --2- 172.21.15.52:0/3667044138 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64107ab0 0x7faf64109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 --2- 172.21.15.52:0/3667044138 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64105180 0x7faf64107570 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 --2- 172.21.15.52:0/3667044138 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 0x7faf64104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.345+0000 7faf6a800640 1 -- 172.21.15.52:0/3667044138 >> 172.21.15.52:0/3667044138 conn(0x7faf640fc240 msgr2=0x7faf640fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:00.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.346+0000 7faf6a800640 1 -- 172.21.15.52:0/3667044138 shutdown_connections 2024-04-26T01:45:00.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.346+0000 7faf6a800640 1 -- 172.21.15.52:0/3667044138 wait complete. 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.346+0000 7faf6a800640 1 Processor -- start 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 -- start start 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 0x7faf6419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64105180 0x7faf6419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410d950 con 0x7faf64102850 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410d7d0 con 0x7faf64105180 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf6a800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410dad0 con 0x7faf64107ab0 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf69400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 0x7faf6419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf69e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf68a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64105180 0x7faf6419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf69e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50506/0 (socket says 172.21.15.52:50506) 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.347+0000 7faf69e00640 1 -- 172.21.15.52:0/231199454 learned_addr learned my addr 172.21.15.52:0/231199454 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:00.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 -- 172.21.15.52:0/231199454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64105180 msgr2=0x7faf6419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64105180 0x7faf6419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 -- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 msgr2=0x7faf6419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 0x7faf6419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 -- 172.21.15.52:0/231199454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641a1e40 con 0x7faf64107ab0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf69e00640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00eb10 tx=0x7faf5c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c004b20 con 0x7faf64107ab0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a2130 con 0x7faf64107ab0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5c004cc0 con 0x7faf64107ab0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c010690 con 0x7faf64107ab0 2024-04-26T01:45:00.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.348+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a26a0 con 0x7faf64107ab0 2024-04-26T01:45:00.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.350+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf64109600 con 0x7faf64107ab0 2024-04-26T01:45:00.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.352+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7faf5c005020 con 0x7faf64107ab0 2024-04-26T01:45:00.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.354+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7faf5c0d0570 con 0x7faf64107ab0 2024-04-26T01:45:00.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.358+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf5c09d200 con 0x7faf64107ab0 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3629157429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: Active manager daemon x restarted 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: Activating manager daemon x 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: mgrmap e22: x(active, starting, since 0.0134433s), standbys: y, z 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:00.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:00.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:00.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: Manager daemon x is now available 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.405 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:00 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3629157429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: Active manager daemon x restarted 2024-04-26T01:45:00.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: Activating manager daemon x 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: mgrmap e22: x(active, starting, since 0.0134433s), standbys: y, z 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:00.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:00.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: Manager daemon x is now available 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.427 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:00 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.152+0000 7f16dfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.152+0000 7f16dfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.152+0000 7f16dfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.152+0000 7f16dfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.152+0000 7f16dfa00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.153+0000 7f16da000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.153+0000 7f16da000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.153+0000 7f16da000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.153+0000 7f16da000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.461 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:45:00.153+0000 7f16da000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3629157429' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: Active manager daemon x restarted 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: Activating manager daemon x 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: mgrmap e22: x(active, starting, since 0.0134433s), standbys: y, z 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T01:45:00.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T01:45:00.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T01:45:00.464 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: Manager daemon x is now available 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.465 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:00 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T01:45:00.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.850+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7faf5c098850 con 0x7faf64107ab0 2024-04-26T01:45:00.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.853+0000 7faf62000640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 0x7faf4c0acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:00.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.853+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7faf4c01dcc0 con 0x7faf4c0aa8a0 2024-04-26T01:45:00.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.860+0000 7faf69400640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 0x7faf4c0acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:00.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.868+0000 7faf69400640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 0x7faf4c0acc90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf580095f0 tx=0x7faf58005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:00.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.960+0000 7faf62000640 1 -- 172.21.15.52:0/231199454 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7faf4c01dcc0 con 0x7faf4c0aa8a0 2024-04-26T01:45:00.957 INFO:teuthology.orchestra.run.smithi052.stdout:Schedule set for path / 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 msgr2=0x7faf4c0acc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 0x7faf4c0acc90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf580095f0 tx=0x7faf58005f90 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 msgr2=0x7faf641a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00eb10 tx=0x7faf5c00c2d0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 shutdown_connections 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faf4c0aa8a0 0x7faf4c0acc90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf64107ab0 0x7faf641a17d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf64105180 0x7faf6419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 --2- 172.21.15.52:0/231199454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf64102850 0x7faf6419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.962+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 >> 172.21.15.52:0/231199454 conn(0x7faf640fc240 msgr2=0x7faf64102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:00.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.963+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 shutdown_connections 2024-04-26T01:45:00.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:00.963+0000 7faf6a800640 1 -- 172.21.15.52:0/231199454 wait complete. 2024-04-26T01:45:00.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T01:45:01.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.160+0000 7efecf800640 1 -- 172.21.15.52:0/200365283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8105b40 msgr2=0x7efec8105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 --2- 172.21.15.52:0/200365283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8105b40 0x7efec8105fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efec0009be0 tx=0x7efec002a6c0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 -- 172.21.15.52:0/200365283 shutdown_connections 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 --2- 172.21.15.52:0/200365283 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8105b40 0x7efec8105fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 --2- 172.21.15.52:0/200365283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8103190 0x7efec8105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 --2- 172.21.15.52:0/200365283 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 0x7efec8102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 -- 172.21.15.52:0/200365283 >> 172.21.15.52:0/200365283 conn(0x7efec80fc420 msgr2=0x7efec80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 -- 172.21.15.52:0/200365283 shutdown_connections 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.161+0000 7efecf800640 1 -- 172.21.15.52:0/200365283 wait complete. 2024-04-26T01:45:01.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.162+0000 7efecf800640 1 Processor -- start 2024-04-26T01:45:01.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.162+0000 7efecf800640 1 -- start start 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 0x7efec819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8105b40 0x7efec81a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efec81140b0 con 0x7efec8100820 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efec8113f30 con 0x7efec8103190 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecf800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efec8114230 con 0x7efec8105b40 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecce00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 0x7efec819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:50000/0 (socket says 172.21.15.52:50000) 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 -- 172.21.15.52:0/3520669533 learned_addr learned my addr 172.21.15.52:0/3520669533 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efecd800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8105b40 0x7efec81a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 -- 172.21.15.52:0/3520669533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8105b40 msgr2=0x7efec81a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8105b40 0x7efec81a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 -- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 msgr2=0x7efec819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 0x7efec819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.163+0000 7efec7e00640 1 -- 172.21.15.52:0/3520669533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81a6320 con 0x7efec8103190 2024-04-26T01:45:01.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.164+0000 7efec7e00640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efeb400e9f0 tx=0x7efeb400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:01.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.164+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb400cd80 con 0x7efec8103190 2024-04-26T01:45:01.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.164+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeb40108e0 con 0x7efec8103190 2024-04-26T01:45:01.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.165+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec81a6610 con 0x7efec8103190 2024-04-26T01:45:01.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.165+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4018980 con 0x7efec8103190 2024-04-26T01:45:01.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.165+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec81a6b60 con 0x7efec8103190 2024-04-26T01:45:01.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.166+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec80fdf10 con 0x7efec8103190 2024-04-26T01:45:01.166 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.169+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7efeb4010430 con 0x7efec8103190 2024-04-26T01:45:01.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.170+0000 7efec5600640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 0x7efeb00a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.170+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7efeb40d0720 con 0x7efec8103190 2024-04-26T01:45:01.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.170+0000 7efecce00640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 0x7efeb00a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.171+0000 7efecce00640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 0x7efeb00a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efebc009530 tx=0x7efebc009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:01.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.174+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efeb409d2c0 con 0x7efec8103190 2024-04-26T01:45:01.382 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:01 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:01.382 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:01 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:01.382 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:01 smithi191 ceph-mon[32745]: mgrmap e23: x(active, since 1.03911s), standbys: y, z 2024-04-26T01:45:01.393 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:01 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:01.393 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:01 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:01.393 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:01 smithi183 ceph-mon[32752]: mgrmap e23: x(active, since 1.03911s), standbys: y, z 2024-04-26T01:45:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.452+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7efec80624f0 con 0x7efeb00a3a60 2024-04-26T01:45:01.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:01 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T01:45:01.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:01 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:45:01.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:01 smithi052 ceph-mon[27851]: mgrmap e23: x(active, since 1.03911s), standbys: y, z 2024-04-26T01:45:01.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.540+0000 7efec5600640 1 -- 172.21.15.52:0/3520669533 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7efec80624f0 con 0x7efeb00a3a60 2024-04-26T01:45:01.538 INFO:teuthology.orchestra.run.smithi052.stdout:Retention added to path / 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 msgr2=0x7efeb00a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 0x7efeb00a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efebc009530 tx=0x7efebc009260 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 msgr2=0x7efec819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efeb400e9f0 tx=0x7efeb400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 shutdown_connections 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efeb00a3a60 0x7efeb00a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efec8105b40 0x7efec81a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec8103190 0x7efec819ec90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 --2- 172.21.15.52:0/3520669533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efec8100820 0x7efec819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 >> 172.21.15.52:0/3520669533 conn(0x7efec80fc420 msgr2=0x7efec8101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:01.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 shutdown_connections 2024-04-26T01:45:01.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.543+0000 7efecf800640 1 -- 172.21.15.52:0/3520669533 wait complete. 2024-04-26T01:45:01.558 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 -- 172.21.15.52:0/417819408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 msgr2=0x7f0d6c102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 --2- 172.21.15.52:0/417819408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c102200 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009900 tx=0x7f0d5c02a410 comp rx=0 tx=0).stop 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 -- 172.21.15.52:0/417819408 shutdown_connections 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 --2- 172.21.15.52:0/417819408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c103990 0x7f0d6c10a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 --2- 172.21.15.52:0/417819408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 --2- 172.21.15.52:0/417819408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c102200 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 -- 172.21.15.52:0/417819408 >> 172.21.15.52:0/417819408 conn(0x7f0d6c0fd350 msgr2=0x7f0d6c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 -- 172.21.15.52:0/417819408 shutdown_connections 2024-04-26T01:45:01.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.739+0000 7f0d6be00640 1 -- 172.21.15.52:0/417819408 wait complete. 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.740+0000 7f0d6be00640 1 Processor -- start 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.740+0000 7f0d6be00640 1 -- start start 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c103990 0x7f0d6c1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10d960 con 0x7f0d6c103990 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10d7e0 con 0x7f0d6c102fd0 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c10dae0 con 0x7f0d6c101de0 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50556/0 (socket says 172.21.15.52:50556) 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 -- 172.21.15.52:0/1386356875 learned_addr learned my addr 172.21.15.52:0/1386356875 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:01.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6b400640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c103990 0x7f0d6c1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6a000640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 -- 172.21.15.52:0/1386356875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 msgr2=0x7f0d6c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 -- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c103990 msgr2=0x7f0d6c1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c103990 0x7f0d6c1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6aa00640 1 -- 172.21.15.52:0/1386356875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6c1a1e30 con 0x7f0d6c101de0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.741+0000 7f0d6a000640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d6aa00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c031500 tx=0x7f0d5c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c03f070 con 0x7f0d6c101de0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6c1a20c0 con 0x7f0d6c101de0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d5c031c50 con 0x7f0d6c101de0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c035030 con 0x7f0d6c101de0 2024-04-26T01:45:01.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.742+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6c1a25b0 con 0x7f0d6c101de0 2024-04-26T01:45:01.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.743+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d6c1a2810 con 0x7f0d6c101de0 2024-04-26T01:45:01.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.745+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f0d5c042070 con 0x7f0d6c101de0 2024-04-26T01:45:01.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.747+0000 7f0d63400640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 0x7f0d500a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:01.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.747+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0d5c0f0e30 con 0x7f0d6c101de0 2024-04-26T01:45:01.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.747+0000 7f0d6a000640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 0x7f0d500a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:01.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.748+0000 7f0d6a000640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 0x7f0d500a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c19b720 tx=0x7f0d58005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:01.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:01.753+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d5c0bd9d0 con 0x7f0d6c101de0 2024-04-26T01:45:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.004+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f0d6c0758a0 con 0x7f0d500a3990 2024-04-26T01:45:02.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.022+0000 7f0d63400640 1 -- 172.21.15.52:0/1386356875 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f0d6c0758a0 con 0x7f0d500a3990 2024-04-26T01:45:02.020 INFO:teuthology.orchestra.run.smithi052.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:45:00", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T01:45:02.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 msgr2=0x7f0d500a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 0x7f0d500a5e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c19b720 tx=0x7f0d58005f90 comp rx=0 tx=0).stop 2024-04-26T01:45:02.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 msgr2=0x7f0d6c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c031500 tx=0x7f0d5c003c10 comp rx=0 tx=0).stop 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 shutdown_connections 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0d500a3990 0x7f0d500a5e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0d6c103990 0x7f0d6c1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d6c102fd0 0x7f0d6c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 --2- 172.21.15.52:0/1386356875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d6c101de0 0x7f0d6c19a210 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 >> 172.21.15.52:0/1386356875 conn(0x7f0d6c0fd350 msgr2=0x7f0d6c074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 shutdown_connections 2024-04-26T01:45:02.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.025+0000 7f0d6be00640 1 -- 172.21.15.52:0/1386356875 wait complete. 2024-04-26T01:45:02.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T01:45:02.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 -- 172.21.15.52:0/1846405056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65581041e0 msgr2=0x7f6558104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 --2- 172.21.15.52:0/1846405056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65581041e0 0x7f6558104600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6550009ca0 tx=0x7f655002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 -- 172.21.15.52:0/1846405056 shutdown_connections 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 --2- 172.21.15.52:0/1846405056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6558105d90 0x7f655810c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 --2- 172.21.15.52:0/1846405056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f6558105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 --2- 172.21.15.52:0/1846405056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65581041e0 0x7f6558104600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.234+0000 7f655cc00640 1 -- 172.21.15.52:0/1846405056 >> 172.21.15.52:0/1846405056 conn(0x7f65580ff770 msgr2=0x7f6558101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.235+0000 7f655cc00640 1 -- 172.21.15.52:0/1846405056 shutdown_connections 2024-04-26T01:45:02.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.235+0000 7f655cc00640 1 -- 172.21.15.52:0/1846405056 wait complete. 2024-04-26T01:45:02.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.235+0000 7f655cc00640 1 Processor -- start 2024-04-26T01:45:02.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 -- start start 2024-04-26T01:45:02.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65581041e0 0x7f655819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f655819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f655810fd70 con 0x7f65581041e0 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f655810fbf0 con 0x7f65581053d0 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f655cc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f655810fef0 con 0x7f6558105d90 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f6555e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65581041e0 0x7f655819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f6556800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f6556800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50580/0 (socket says 172.21.15.52:50580) 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f6556800640 1 -- 172.21.15.52:0/3667443454 learned_addr learned my addr 172.21.15.52:0/3667443454 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.236+0000 7f6555400640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f655819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 -- 172.21.15.52:0/3667443454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 msgr2=0x7f655819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f655819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 -- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65581041e0 msgr2=0x7f655819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65581041e0 0x7f655819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 -- 172.21.15.52:0/3667443454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65581a1f60 con 0x7f6558105d90 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6555400640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f655819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f6556800640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f654800e970 tx=0x7f654800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654800ccc0 con 0x7f6558105d90 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65581a2250 con 0x7f6558105d90 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654800ce60 con 0x7f6558105d90 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548010430 con 0x7f6558105d90 2024-04-26T01:45:02.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.237+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65581a2750 con 0x7f6558105d90 2024-04-26T01:45:02.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.238+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655810b4a0 con 0x7f6558105d90 2024-04-26T01:45:02.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.240+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7f6548004ff0 con 0x7f6558105d90 2024-04-26T01:45:02.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.242+0000 7f654ea00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 0x7f653c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:02.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.242+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f65480d0d20 con 0x7f6558105d90 2024-04-26T01:45:02.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.242+0000 7f6555e00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 0x7f653c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:02.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.243+0000 7f6555e00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 0x7f653c0a6280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6550000f80 tx=0x7f6550005f30 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:02.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.244+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f654809d390 con 0x7f6558105d90 2024-04-26T01:45:02.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:02 smithi191 ceph-mon[32745]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:02.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:02 smithi191 ceph-mon[32745]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:02.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:02 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:02 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:02 smithi183 ceph-mon[32752]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:02.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:02 smithi183 ceph-mon[32752]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:02.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:02 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:02 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:02 smithi052 ceph-mon[27851]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:02.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:02 smithi052 ceph-mon[27851]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:02.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:02 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:02 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:02.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.477+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f6558062390 con 0x7f653c0a3dc0 2024-04-26T01:45:02.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.485+0000 7f654ea00640 1 -- 172.21.15.52:0/3667443454 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f6558062390 con 0x7f653c0a3dc0 2024-04-26T01:45:02.483 INFO:teuthology.orchestra.run.smithi052.stdout:/ 1m 6m3h 2024-04-26T01:45:02.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 msgr2=0x7f653c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 0x7f653c0a6280 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6550000f80 tx=0x7f6550005f30 comp rx=0 tx=0).stop 2024-04-26T01:45:02.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 msgr2=0x7f65581a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:02.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f654800e970 tx=0x7f654800ed70 comp rx=0 tx=0).stop 2024-04-26T01:45:02.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 shutdown_connections 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f653c0a3dc0 0x7f653c0a6280 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6558105d90 0x7f65581a18f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f65581053d0 0x7f655819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 --2- 172.21.15.52:0/3667443454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65581041e0 0x7f655819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.487+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 >> 172.21.15.52:0/3667443454 conn(0x7f65580ff770 msgr2=0x7f6558108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.488+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 shutdown_connections 2024-04-26T01:45:02.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:02.488+0000 7f655cc00640 1 -- 172.21.15.52:0/3667443454 wait complete. 2024-04-26T01:45:02.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T01:45:02.569 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T01:45:02.582 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T01:45:02.596 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T01:45:02.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T01:45:02.601 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-26T01:45:02.616 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T01:45:02.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:45:03.069 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: [26/Apr/2024:01:45:01] ENGINE Bus STARTING 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: [26/Apr/2024:01:45:01] ENGINE Serving on https://172.21.15.52:7150 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: [26/Apr/2024:01:45:01] ENGINE Serving on http://172.21.15.52:8765 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: [26/Apr/2024:01:45:01] ENGINE Bus STARTED 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:03.400 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:03 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: [26/Apr/2024:01:45:01] ENGINE Bus STARTING 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: [26/Apr/2024:01:45:01] ENGINE Serving on https://172.21.15.52:7150 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: [26/Apr/2024:01:45:01] ENGINE Serving on http://172.21.15.52:8765 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: [26/Apr/2024:01:45:01] ENGINE Bus STARTED 2024-04-26T01:45:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:03 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: [26/Apr/2024:01:45:01] ENGINE Bus STARTING 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: [26/Apr/2024:01:45:01] ENGINE Serving on https://172.21.15.52:7150 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: [26/Apr/2024:01:45:01] ENGINE Serving on http://172.21.15.52:8765 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: [26/Apr/2024:01:45:01] ENGINE Bus STARTED 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:03 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:45:04.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:04 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:45:04.406 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:45:04.406 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.406 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:04 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T01:45:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:04.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:04 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 -- 172.21.15.52:0/2852118191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 msgr2=0x7f1e18102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/2852118191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e18102df0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009990 tx=0x7f1e04009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 -- 172.21.15.52:0/2852118191 shutdown_connections 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/2852118191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/2852118191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 0x7f1e18108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/2852118191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e18102df0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 -- 172.21.15.52:0/2852118191 >> 172.21.15.52:0/2852118191 conn(0x7f1e180fe4c0 msgr2=0x7f1e181008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 -- 172.21.15.52:0/2852118191 shutdown_connections 2024-04-26T01:45:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.399+0000 7f1e1ca00700 1 -- 172.21.15.52:0/2852118191 wait complete. 2024-04-26T01:45:05.398 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:05 smithi052 ceph-mon[27851]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:05.398 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:05 smithi052 ceph-mon[27851]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:45:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.401+0000 7f1e1ca00700 1 Processor -- start 2024-04-26T01:45:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.401+0000 7f1e1ca00700 1 -- start start 2024-04-26T01:45:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.401+0000 7f1e1ca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.401+0000 7f1e1ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e1ca00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 0x7f1e1819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e1ca00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1e18112bb0 con 0x7f1e18103330 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e1ca00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1e18112a30 con 0x7f1e18102970 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e1ca00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1e18112d30 con 0x7f1e18108970 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42536/0 (socket says 172.21.15.52:42536) 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 -- 172.21.15.52:0/772306133 learned_addr learned my addr 172.21.15.52:0/772306133 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:05.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e17200700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 0x7f1e1819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e15e00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 -- 172.21.15.52:0/772306133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 msgr2=0x7f1e1819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 0x7f1e1819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 -- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 msgr2=0x7f1e1819b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1819b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e16800700 1 -- 172.21.15.52:0/772306133 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e181a0220 con 0x7f1e18102970 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.402+0000 7f1e15e00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e16800700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1e0800e9c0 tx=0x7f1e0800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:05 smithi191 ceph-mon[32745]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:05.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:05 smithi191 ceph-mon[32745]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800cbd0 con 0x7f1e18102970 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0800cd70 con 0x7f1e18102970 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08010430 con 0x7f1e18102970 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e181a0510 con 0x7f1e18102970 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.403+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e181a7d90 con 0x7f1e18102970 2024-04-26T01:45:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.405+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e18069730 con 0x7f1e18102970 2024-04-26T01:45:05.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.407+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f1e08004d70 con 0x7f1e18102970 2024-04-26T01:45:05.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.408+0000 7f1e0f400700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 0x7f1dfc0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:05.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.409+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f1e080d0d90 con 0x7f1e18102970 2024-04-26T01:45:05.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.409+0000 7f1e15e00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 0x7f1dfc0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:05.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.410+0000 7f1e15e00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 0x7f1dfc0a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b2e0 tx=0x7f1e04005e80 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:05.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.411+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e08053ff0 con 0x7f1e18102970 2024-04-26T01:45:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:05 smithi183 ceph-mon[32752]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:05 smithi183 ceph-mon[32752]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-26T01:45:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.635+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e180ffa10 con 0x7f1dfc0a3ec0 2024-04-26T01:45:05.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.645+0000 7f1e0f400700 1 -- 172.21.15.52:0/772306133 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257772 (secure 0 0 0) 0x7f1e180ffa10 con 0x7f1dfc0a3ec0 2024-04-26T01:45:05.645 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:05.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.651+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 msgr2=0x7f1dfc0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:05.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.651+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 0x7f1dfc0a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b2e0 tx=0x7f1e04005e80 comp rx=0 tx=0).stop 2024-04-26T01:45:05.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.651+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 msgr2=0x7f1e1819b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:05.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.651+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1e0800e9c0 tx=0x7f1e0800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 shutdown_connections 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1dfc0a3ec0 0x7f1dfc0a6380 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b2e0 tx=0x7f1e04005e80 comp rx=0 tx=0).stop 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e18108970 0x7f1e1819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e18102970 0x7f1e1819b210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 --2- 172.21.15.52:0/772306133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e18103330 0x7f1e1819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:05.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 >> 172.21.15.52:0/772306133 conn(0x7f1e180fe4c0 msgr2=0x7f1e1810e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:05.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 shutdown_connections 2024-04-26T01:45:05.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:05.652+0000 7f1e1ca00700 1 -- 172.21.15.52:0/772306133 wait complete. 2024-04-26T01:45:05.653 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:45:06.614 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-26T01:45:03.850095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192562,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1542,"num_read_kb":935,"num_write":482,"num_write_kb":4386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":398380,"kb_used_data":15852,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124543444,"statfs":{"total":1151940427776,"available":1151532486656,"internally_reserved":0,"allocated":16232448,"data_stored":9650646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.971643"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386883+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:45:00.386883+0000","last_peered":"2024-04-26T01:45:00.386883+0000","last_clean":"2024-04-26T01:45:00.386883+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:45:00.386883+0000","last_undegraded":"2024-04-26T01:45:00.386883+0000","last_fullsized":"2024-04-26T01:45:00.386883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145954+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:45:00.145954+0000","last_peered":"2024-04-26T01:45:00.145954+0000","last_clean":"2024-04-26T01:45:00.145954+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:45:00.145954+0000","last_undegraded":"2024-04-26T01:45:00.145954+0000","last_fullsized":"2024-04-26T01:45:00.145954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146356+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:45:00.146356+0000","last_peered":"2024-04-26T01:45:00.146356+0000","last_clean":"2024-04-26T01:45:00.146356+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:45:00.146356+0000","last_undegraded":"2024-04-26T01:45:00.146356+0000","last_fullsized":"2024-04-26T01:45:00.146356+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971689+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:44:59.971689+0000","last_peered":"2024-04-26T01:44:59.971689+0000","last_clean":"2024-04-26T01:44:59.971689+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:44:59.971689+0000","last_undegraded":"2024-04-26T01:44:59.971689+0000","last_fullsized":"2024-04-26T01:44:59.971689+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829979+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:44:59.829979+0000","last_peered":"2024-04-26T01:44:59.829979+0000","last_clean":"2024-04-26T01:44:59.829979+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:44:59.829979+0000","last_undegraded":"2024-04-26T01:44:59.829979+0000","last_fullsized":"2024-04-26T01:44:59.829979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983845+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:44:59.983845+0000","last_peered":"2024-04-26T01:44:59.983845+0000","last_clean":"2024-04-26T01:44:59.983845+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:44:59.983845+0000","last_undegraded":"2024-04-26T01:44:59.983845+0000","last_fullsized":"2024-04-26T01:44:59.983845+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832466+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:44:59.832466+0000","last_peered":"2024-04-26T01:44:59.832466+0000","last_clean":"2024-04-26T01:44:59.832466+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:44:59.832466+0000","last_undegraded":"2024-04-26T01:44:59.832466+0000","last_fullsized":"2024-04-26T01:44:59.832466+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942792+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:44:59.942792+0000","last_peered":"2024-04-26T01:44:59.942792+0000","last_clean":"2024-04-26T01:44:59.942792+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:44:59.942792+0000","last_undegraded":"2024-04-26T01:44:59.942792+0000","last_fullsized":"2024-04-26T01:44:59.942792+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145813+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:45:00.145813+0000","last_peered":"2024-04-26T01:45:00.145813+0000","last_clean":"2024-04-26T01:45:00.145813+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:45:00.145813+0000","last_undegraded":"2024-04-26T01:45:00.145813+0000","last_fullsized":"2024-04-26T01:45:00.145813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.984139+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:44:59.984139+0000","last_peered":"2024-04-26T01:44:59.984139+0000","last_clean":"2024-04-26T01:44:59.984139+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:44:59.984139+0000","last_undegraded":"2024-04-26T01:44:59.984139+0000","last_fullsized":"2024-04-26T01:44:59.984139+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833160+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:44:59.833160+0000","last_peered":"2024-04-26T01:44:59.833160+0000","last_clean":"2024-04-26T01:44:59.833160+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:44:59.833160+0000","last_undegraded":"2024-04-26T01:44:59.833160+0000","last_fullsized":"2024-04-26T01:44:59.833160+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942577+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:44:59.942577+0000","last_peered":"2024-04-26T01:44:59.942577+0000","last_clean":"2024-04-26T01:44:59.942577+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:44:59.942577+0000","last_undegraded":"2024-04-26T01:44:59.942577+0000","last_fullsized":"2024-04-26T01:44:59.942577+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836208+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:44:59.836208+0000","last_peered":"2024-04-26T01:44:59.836208+0000","last_clean":"2024-04-26T01:44:59.836208+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:44:59.836208+0000","last_undegraded":"2024-04-26T01:44:59.836208+0000","last_fullsized":"2024-04-26T01:44:59.836208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145890+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:45:00.145890+0000","last_peered":"2024-04-26T01:45:00.145890+0000","last_clean":"2024-04-26T01:45:00.145890+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:45:00.145890+0000","last_undegraded":"2024-04-26T01:45:00.145890+0000","last_fullsized":"2024-04-26T01:45:00.145890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145592+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:45:00.145592+0000","last_peered":"2024-04-26T01:45:00.145592+0000","last_clean":"2024-04-26T01:45:00.145592+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:45:00.145592+0000","last_undegraded":"2024-04-26T01:45:00.145592+0000","last_fullsized":"2024-04-26T01:45:00.145592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145452+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:45:00.145452+0000","last_peered":"2024-04-26T01:45:00.145452+0000","last_clean":"2024-04-26T01:45:00.145452+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:45:00.145452+0000","last_undegraded":"2024-04-26T01:45:00.145452+0000","last_fullsized":"2024-04-26T01:45:00.145452+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386985+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:45:00.386985+0000","last_peered":"2024-04-26T01:45:00.386985+0000","last_clean":"2024-04-26T01:45:00.386985+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:45:00.386985+0000","last_undegraded":"2024-04-26T01:45:00.386985+0000","last_fullsized":"2024-04-26T01:45:00.386985+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971739+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:44:59.971739+0000","last_peered":"2024-04-26T01:44:59.971739+0000","last_clean":"2024-04-26T01:44:59.971739+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:44:59.971739+0000","last_undegraded":"2024-04-26T01:44:59.971739+0000","last_fullsized":"2024-04-26T01:44:59.971739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146085+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:45:00.146085+0000","last_peered":"2024-04-26T01:45:00.146085+0000","last_clean":"2024-04-26T01:45:00.146085+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:45:00.146085+0000","last_undegraded":"2024-04-26T01:45:00.146085+0000","last_fullsized":"2024-04-26T01:45:00.146085+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.972070+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:44:59.972070+0000","last_peered":"2024-04-26T01:44:59.972070+0000","last_clean":"2024-04-26T01:44:59.972070+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:44:59.972070+0000","last_undegraded":"2024-04-26T01:44:59.972070+0000","last_fullsized":"2024-04-26T01:44:59.972070+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943348+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:44:59.943348+0000","last_peered":"2024-04-26T01:44:59.943348+0000","last_clean":"2024-04-26T01:44:59.943348+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:44:59.943348+0000","last_undegraded":"2024-04-26T01:44:59.943348+0000","last_fullsized":"2024-04-26T01:44:59.943348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943600+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:44:59.943600+0000","last_peered":"2024-04-26T01:44:59.943600+0000","last_clean":"2024-04-26T01:44:59.943600+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:44:59.943600+0000","last_undegraded":"2024-04-26T01:44:59.943600+0000","last_fullsized":"2024-04-26T01:44:59.943600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146367+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:45:00.146367+0000","last_peered":"2024-04-26T01:45:00.146367+0000","last_clean":"2024-04-26T01:45:00.146367+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:45:00.146367+0000","last_undegraded":"2024-04-26T01:45:00.146367+0000","last_fullsized":"2024-04-26T01:45:00.146367+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386813+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:45:00.386813+0000","last_peered":"2024-04-26T01:45:00.386813+0000","last_clean":"2024-04-26T01:45:00.386813+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:45:00.386813+0000","last_undegraded":"2024-04-26T01:45:00.386813+0000","last_fullsized":"2024-04-26T01:45:00.386813+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145523+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:45:00.145523+0000","last_peered":"2024-04-26T01:45:00.145523+0000","last_clean":"2024-04-26T01:45:00.145523+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:45:00.145523+0000","last_undegraded":"2024-04-26T01:45:00.145523+0000","last_fullsized":"2024-04-26T01:45:00.145523+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830007+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:44:59.830007+0000","last_peered":"2024-04-26T01:44:59.830007+0000","last_clean":"2024-04-26T01:44:59.830007+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:44:59.830007+0000","last_undegraded":"2024-04-26T01:44:59.830007+0000","last_fullsized":"2024-04-26T01:44:59.830007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980895+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:44:59.980895+0000","last_peered":"2024-04-26T01:44:59.980895+0000","last_clean":"2024-04-26T01:44:59.980895+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:44:59.980895+0000","last_undegraded":"2024-04-26T01:44:59.980895+0000","last_fullsized":"2024-04-26T01:44:59.980895+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"95'7","reported_seq":253,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.700747+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:45:00.700747+0000","last_peered":"2024-04-26T01:45:00.700747+0000","last_clean":"2024-04-26T01:45:00.700747+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:45:00.700747+0000","last_undegraded":"2024-04-26T01:45:00.700747+0000","last_fullsized":"2024-04-26T01:45:00.700747+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829791+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:44:59.829791+0000","last_peered":"2024-04-26T01:44:59.829791+0000","last_clean":"2024-04-26T01:44:59.829791+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:44:59.829791+0000","last_undegraded":"2024-04-26T01:44:59.829791+0000","last_fullsized":"2024-04-26T01:44:59.829791+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983749+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:44:59.983749+0000","last_peered":"2024-04-26T01:44:59.983749+0000","last_clean":"2024-04-26T01:44:59.983749+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:44:59.983749+0000","last_undegraded":"2024-04-26T01:44:59.983749+0000","last_fullsized":"2024-04-26T01:44:59.983749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980762+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:44:59.980762+0000","last_peered":"2024-04-26T01:44:59.980762+0000","last_clean":"2024-04-26T01:44:59.980762+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:44:59.980762+0000","last_undegraded":"2024-04-26T01:44:59.980762+0000","last_fullsized":"2024-04-26T01:44:59.980762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836546+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:44:59.836546+0000","last_peered":"2024-04-26T01:44:59.836546+0000","last_clean":"2024-04-26T01:44:59.836546+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:44:59.836546+0000","last_undegraded":"2024-04-26T01:44:59.836546+0000","last_fullsized":"2024-04-26T01:44:59.836546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942696+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:44:59.942696+0000","last_peered":"2024-04-26T01:44:59.942696+0000","last_clean":"2024-04-26T01:44:59.942696+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:44:59.942696+0000","last_undegraded":"2024-04-26T01:44:59.942696+0000","last_fullsized":"2024-04-26T01:44:59.942696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980603+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:44:59.980603+0000","last_peered":"2024-04-26T01:44:59.980603+0000","last_clean":"2024-04-26T01:44:59.980603+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:44:59.980603+0000","last_undegraded":"2024-04-26T01:44:59.980603+0000","last_fullsized":"2024-04-26T01:44:59.980603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971311+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:44:59.971311+0000","last_peered":"2024-04-26T01:44:59.971311+0000","last_clean":"2024-04-26T01:44:59.971311+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:44:59.971311+0000","last_undegraded":"2024-04-26T01:44:59.971311+0000","last_fullsized":"2024-04-26T01:44:59.971311+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":247,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.162860+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:45:00.162860+0000","last_peered":"2024-04-26T01:45:00.162860+0000","last_clean":"2024-04-26T01:45:00.162860+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:45:00.162860+0000","last_undegraded":"2024-04-26T01:45:00.162860+0000","last_fullsized":"2024-04-26T01:45:00.162860+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":214,"num_read_kb":214,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145864+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:45:00.145864+0000","last_peered":"2024-04-26T01:45:00.145864+0000","last_clean":"2024-04-26T01:45:00.145864+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:45:00.145864+0000","last_undegraded":"2024-04-26T01:45:00.145864+0000","last_fullsized":"2024-04-26T01:45:00.145864+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832945+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:44:59.832945+0000","last_peered":"2024-04-26T01:44:59.832945+0000","last_clean":"2024-04-26T01:44:59.832945+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:44:59.832945+0000","last_undegraded":"2024-04-26T01:44:59.832945+0000","last_fullsized":"2024-04-26T01:44:59.832945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146547+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:45:00.146547+0000","last_peered":"2024-04-26T01:45:00.146547+0000","last_clean":"2024-04-26T01:45:00.146547+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:45:00.146547+0000","last_undegraded":"2024-04-26T01:45:00.146547+0000","last_fullsized":"2024-04-26T01:45:00.146547+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.981042+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:44:59.981042+0000","last_peered":"2024-04-26T01:44:59.981042+0000","last_clean":"2024-04-26T01:44:59.981042+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:44:59.981042+0000","last_undegraded":"2024-04-26T01:44:59.981042+0000","last_fullsized":"2024-04-26T01:44:59.981042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145594+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:45:00.145594+0000","last_peered":"2024-04-26T01:45:00.145594+0000","last_clean":"2024-04-26T01:45:00.145594+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:45:00.145594+0000","last_undegraded":"2024-04-26T01:45:00.145594+0000","last_fullsized":"2024-04-26T01:45:00.145594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":508,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.700647+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:45:00.700647+0000","last_peered":"2024-04-26T01:45:00.700647+0000","last_clean":"2024-04-26T01:45:00.700647+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:45:00.700647+0000","last_undegraded":"2024-04-26T01:45:00.700647+0000","last_fullsized":"2024-04-26T01:45:00.700647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71438,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":440,"num_read_kb":290,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829990+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:44:59.829990+0000","last_peered":"2024-04-26T01:44:59.829990+0000","last_clean":"2024-04-26T01:44:59.829990+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:44:59.829990+0000","last_undegraded":"2024-04-26T01:44:59.829990+0000","last_fullsized":"2024-04-26T01:44:59.829990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145896+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:45:00.145896+0000","last_peered":"2024-04-26T01:45:00.145896+0000","last_clean":"2024-04-26T01:45:00.145896+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:45:00.145896+0000","last_undegraded":"2024-04-26T01:45:00.145896+0000","last_fullsized":"2024-04-26T01:45:00.145896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836995+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:44:59.836995+0000","last_peered":"2024-04-26T01:44:59.836995+0000","last_clean":"2024-04-26T01:44:59.836995+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:44:59.836995+0000","last_undegraded":"2024-04-26T01:44:59.836995+0000","last_fullsized":"2024-04-26T01:44:59.836995+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980687+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:44:59.980687+0000","last_peered":"2024-04-26T01:44:59.980687+0000","last_clean":"2024-04-26T01:44:59.980687+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:44:59.980687+0000","last_undegraded":"2024-04-26T01:44:59.980687+0000","last_fullsized":"2024-04-26T01:44:59.980687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832836+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:44:59.832836+0000","last_peered":"2024-04-26T01:44:59.832836+0000","last_clean":"2024-04-26T01:44:59.832836+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:44:59.832836+0000","last_undegraded":"2024-04-26T01:44:59.832836+0000","last_fullsized":"2024-04-26T01:44:59.832836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"95'7","reported_seq":253,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.166172+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:45:00.166172+0000","last_peered":"2024-04-26T01:45:00.166172+0000","last_clean":"2024-04-26T01:45:00.166172+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:45:00.166172+0000","last_undegraded":"2024-04-26T01:45:00.166172+0000","last_fullsized":"2024-04-26T01:45:00.166172+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":5,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832336+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:44:59.832336+0000","last_peered":"2024-04-26T01:44:59.832336+0000","last_clean":"2024-04-26T01:44:59.832336+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:44:59.832336+0000","last_undegraded":"2024-04-26T01:44:59.832336+0000","last_fullsized":"2024-04-26T01:44:59.832336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829747+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:44:59.829747+0000","last_peered":"2024-04-26T01:44:59.829747+0000","last_clean":"2024-04-26T01:44:59.829747+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:44:59.829747+0000","last_undegraded":"2024-04-26T01:44:59.829747+0000","last_fullsized":"2024-04-26T01:44:59.829747+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971521+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:44:59.971521+0000","last_peered":"2024-04-26T01:44:59.971521+0000","last_clean":"2024-04-26T01:44:59.971521+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:44:59.971521+0000","last_undegraded":"2024-04-26T01:44:59.971521+0000","last_fullsized":"2024-04-26T01:44:59.971521+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"95'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830365+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:44:59.830365+0000","last_peered":"2024-04-26T01:44:59.830365+0000","last_clean":"2024-04-26T01:44:59.830365+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:44:59.830365+0000","last_undegraded":"2024-04-26T01:44:59.830365+0000","last_fullsized":"2024-04-26T01:44:59.830365+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386745+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:45:00.386745+0000","last_peered":"2024-04-26T01:45:00.386745+0000","last_clean":"2024-04-26T01:45:00.386745+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:45:00.386745+0000","last_undegraded":"2024-04-26T01:45:00.386745+0000","last_fullsized":"2024-04-26T01:45:00.386745+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971578+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:44:59.971578+0000","last_peered":"2024-04-26T01:44:59.971578+0000","last_clean":"2024-04-26T01:44:59.971578+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:44:59.971578+0000","last_undegraded":"2024-04-26T01:44:59.971578+0000","last_fullsized":"2024-04-26T01:44:59.971578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983891+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:44:59.983891+0000","last_peered":"2024-04-26T01:44:59.983891+0000","last_clean":"2024-04-26T01:44:59.983891+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:44:59.983891+0000","last_undegraded":"2024-04-26T01:44:59.983891+0000","last_fullsized":"2024-04-26T01:44:59.983891+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832723+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:44:59.832723+0000","last_peered":"2024-04-26T01:44:59.832723+0000","last_clean":"2024-04-26T01:44:59.832723+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:44:59.832723+0000","last_undegraded":"2024-04-26T01:44:59.832723+0000","last_fullsized":"2024-04-26T01:44:59.832723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983790+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:44:59.983790+0000","last_peered":"2024-04-26T01:44:59.983790+0000","last_clean":"2024-04-26T01:44:59.983790+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:44:59.983790+0000","last_undegraded":"2024-04-26T01:44:59.983790+0000","last_fullsized":"2024-04-26T01:44:59.983790+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980893+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:44:59.980893+0000","last_peered":"2024-04-26T01:44:59.980893+0000","last_clean":"2024-04-26T01:44:59.980893+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:44:59.980893+0000","last_undegraded":"2024-04-26T01:44:59.980893+0000","last_fullsized":"2024-04-26T01:44:59.980893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":248,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.940486+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:44:59.940486+0000","last_peered":"2024-04-26T01:44:59.940486+0000","last_clean":"2024-04-26T01:44:59.940486+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:44:59.940486+0000","last_undegraded":"2024-04-26T01:44:59.940486+0000","last_fullsized":"2024-04-26T01:44:59.940486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":213,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943097+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:44:59.943097+0000","last_peered":"2024-04-26T01:44:59.943097+0000","last_clean":"2024-04-26T01:44:59.943097+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:44:59.943097+0000","last_undegraded":"2024-04-26T01:44:59.943097+0000","last_fullsized":"2024-04-26T01:44:59.943097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942566+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:44:59.942566+0000","last_peered":"2024-04-26T01:44:59.942566+0000","last_clean":"2024-04-26T01:44:59.942566+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:44:59.942566+0000","last_undegraded":"2024-04-26T01:44:59.942566+0000","last_fullsized":"2024-04-26T01:44:59.942566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832440+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:44:59.832440+0000","last_peered":"2024-04-26T01:44:59.832440+0000","last_clean":"2024-04-26T01:44:59.832440+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:44:59.832440+0000","last_undegraded":"2024-04-26T01:44:59.832440+0000","last_fullsized":"2024-04-26T01:44:59.832440+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.835940+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:44:59.835940+0000","last_peered":"2024-04-26T01:44:59.835940+0000","last_clean":"2024-04-26T01:44:59.835940+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:44:59.835940+0000","last_undegraded":"2024-04-26T01:44:59.835940+0000","last_fullsized":"2024-04-26T01:44:59.835940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832637+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:44:59.832637+0000","last_peered":"2024-04-26T01:44:59.832637+0000","last_clean":"2024-04-26T01:44:59.832637+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:44:59.832637+0000","last_undegraded":"2024-04-26T01:44:59.832637+0000","last_fullsized":"2024-04-26T01:44:59.832637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830163+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:44:59.830163+0000","last_peered":"2024-04-26T01:44:59.830163+0000","last_clean":"2024-04-26T01:44:59.830163+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:44:59.830163+0000","last_undegraded":"2024-04-26T01:44:59.830163+0000","last_fullsized":"2024-04-26T01:44:59.830163+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836146+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:44:59.836146+0000","last_peered":"2024-04-26T01:44:59.836146+0000","last_clean":"2024-04-26T01:44:59.836146+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:44:59.836146+0000","last_undegraded":"2024-04-26T01:44:59.836146+0000","last_fullsized":"2024-04-26T01:44:59.836146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"95'49","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.115388+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:45:00.115388+0000","last_peered":"2024-04-26T01:45:00.115388+0000","last_clean":"2024-04-26T01:45:00.115388+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:45:00.115388+0000","last_undegraded":"2024-04-26T01:45:00.115388+0000","last_fullsized":"2024-04-26T01:45:00.115388+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943583+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:44:59.943583+0000","last_peered":"2024-04-26T01:44:59.943583+0000","last_clean":"2024-04-26T01:44:59.943583+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:44:59.943583+0000","last_undegraded":"2024-04-26T01:44:59.943583+0000","last_fullsized":"2024-04-26T01:44:59.943583+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942746+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:44:59.942746+0000","last_peered":"2024-04-26T01:44:59.942746+0000","last_clean":"2024-04-26T01:44:59.942746+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:44:59.942746+0000","last_undegraded":"2024-04-26T01:44:59.942746+0000","last_fullsized":"2024-04-26T01:44:59.942746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146000+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:45:00.146000+0000","last_peered":"2024-04-26T01:45:00.146000+0000","last_clean":"2024-04-26T01:45:00.146000+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:45:00.146000+0000","last_undegraded":"2024-04-26T01:45:00.146000+0000","last_fullsized":"2024-04-26T01:45:00.146000+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145455+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:45:00.145455+0000","last_peered":"2024-04-26T01:45:00.145455+0000","last_clean":"2024-04-26T01:45:00.145455+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:45:00.145455+0000","last_undegraded":"2024-04-26T01:45:00.145455+0000","last_fullsized":"2024-04-26T01:45:00.145455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829725+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:44:59.829725+0000","last_peered":"2024-04-26T01:44:59.829725+0000","last_clean":"2024-04-26T01:44:59.829725+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:44:59.829725+0000","last_undegraded":"2024-04-26T01:44:59.829725+0000","last_fullsized":"2024-04-26T01:44:59.829725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942730+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:44:59.942730+0000","last_peered":"2024-04-26T01:44:59.942730+0000","last_clean":"2024-04-26T01:44:59.942730+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:44:59.942730+0000","last_undegraded":"2024-04-26T01:44:59.942730+0000","last_fullsized":"2024-04-26T01:44:59.942730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386613+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:45:00.386613+0000","last_peered":"2024-04-26T01:45:00.386613+0000","last_clean":"2024-04-26T01:45:00.386613+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:45:00.386613+0000","last_undegraded":"2024-04-26T01:45:00.386613+0000","last_fullsized":"2024-04-26T01:45:00.386613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145777+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:45:00.145777+0000","last_peered":"2024-04-26T01:45:00.145777+0000","last_clean":"2024-04-26T01:45:00.145777+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:45:00.145777+0000","last_undegraded":"2024-04-26T01:45:00.145777+0000","last_fullsized":"2024-04-26T01:45:00.145777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943095+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:44:59.943095+0000","last_peered":"2024-04-26T01:44:59.943095+0000","last_clean":"2024-04-26T01:44:59.943095+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:44:59.943095+0000","last_undegraded":"2024-04-26T01:44:59.943095+0000","last_fullsized":"2024-04-26T01:44:59.943095+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830189+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:44:59.830189+0000","last_peered":"2024-04-26T01:44:59.830189+0000","last_clean":"2024-04-26T01:44:59.830189+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:44:59.830189+0000","last_undegraded":"2024-04-26T01:44:59.830189+0000","last_fullsized":"2024-04-26T01:44:59.830189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832778+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:44:59.832778+0000","last_peered":"2024-04-26T01:44:59.832778+0000","last_clean":"2024-04-26T01:44:59.832778+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:44:59.832778+0000","last_undegraded":"2024-04-26T01:44:59.832778+0000","last_fullsized":"2024-04-26T01:44:59.832778+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942645+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:44:59.942645+0000","last_peered":"2024-04-26T01:44:59.942645+0000","last_clean":"2024-04-26T01:44:59.942645+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:44:59.942645+0000","last_undegraded":"2024-04-26T01:44:59.942645+0000","last_fullsized":"2024-04-26T01:44:59.942645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833327+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:44:59.833327+0000","last_peered":"2024-04-26T01:44:59.833327+0000","last_clean":"2024-04-26T01:44:59.833327+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:44:59.833327+0000","last_undegraded":"2024-04-26T01:44:59.833327+0000","last_fullsized":"2024-04-26T01:44:59.833327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.106275+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:45:00.106275+0000","last_peered":"2024-04-26T01:45:00.106275+0000","last_clean":"2024-04-26T01:45:00.106275+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:45:00.106275+0000","last_undegraded":"2024-04-26T01:45:00.106275+0000","last_fullsized":"2024-04-26T01:45:00.106275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832937+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:44:59.832937+0000","last_peered":"2024-04-26T01:44:59.832937+0000","last_clean":"2024-04-26T01:44:59.832937+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:44:59.832937+0000","last_undegraded":"2024-04-26T01:44:59.832937+0000","last_fullsized":"2024-04-26T01:44:59.832937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836360+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:44:59.836360+0000","last_peered":"2024-04-26T01:44:59.836360+0000","last_clean":"2024-04-26T01:44:59.836360+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:44:59.836360+0000","last_undegraded":"2024-04-26T01:44:59.836360+0000","last_fullsized":"2024-04-26T01:44:59.836360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833557+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:44:59.833557+0000","last_peered":"2024-04-26T01:44:59.833557+0000","last_clean":"2024-04-26T01:44:59.833557+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:44:59.833557+0000","last_undegraded":"2024-04-26T01:44:59.833557+0000","last_fullsized":"2024-04-26T01:44:59.833557+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386692+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:45:00.386692+0000","last_peered":"2024-04-26T01:45:00.386692+0000","last_clean":"2024-04-26T01:45:00.386692+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:45:00.386692+0000","last_undegraded":"2024-04-26T01:45:00.386692+0000","last_fullsized":"2024-04-26T01:45:00.386692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145926+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:45:00.145926+0000","last_peered":"2024-04-26T01:45:00.145926+0000","last_clean":"2024-04-26T01:45:00.145926+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:45:00.145926+0000","last_undegraded":"2024-04-26T01:45:00.145926+0000","last_fullsized":"2024-04-26T01:45:00.145926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145769+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:45:00.145769+0000","last_peered":"2024-04-26T01:45:00.145769+0000","last_clean":"2024-04-26T01:45:00.145769+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:45:00.145769+0000","last_undegraded":"2024-04-26T01:45:00.145769+0000","last_fullsized":"2024-04-26T01:45:00.145769+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145984+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:45:00.145984+0000","last_peered":"2024-04-26T01:45:00.145984+0000","last_clean":"2024-04-26T01:45:00.145984+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:45:00.145984+0000","last_undegraded":"2024-04-26T01:45:00.145984+0000","last_fullsized":"2024-04-26T01:45:00.145984+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.835967+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:44:59.835967+0000","last_peered":"2024-04-26T01:44:59.835967+0000","last_clean":"2024-04-26T01:44:59.835967+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:44:59.835967+0000","last_undegraded":"2024-04-26T01:44:59.835967+0000","last_fullsized":"2024-04-26T01:44:59.835967+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386872+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:45:00.386872+0000","last_peered":"2024-04-26T01:45:00.386872+0000","last_clean":"2024-04-26T01:45:00.386872+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:45:00.386872+0000","last_undegraded":"2024-04-26T01:45:00.386872+0000","last_fullsized":"2024-04-26T01:45:00.386872+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146069+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:45:00.146069+0000","last_peered":"2024-04-26T01:45:00.146069+0000","last_clean":"2024-04-26T01:45:00.146069+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:45:00.146069+0000","last_undegraded":"2024-04-26T01:45:00.146069+0000","last_fullsized":"2024-04-26T01:45:00.146069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943455+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:44:59.943455+0000","last_peered":"2024-04-26T01:44:59.943455+0000","last_clean":"2024-04-26T01:44:59.943455+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:44:59.943455+0000","last_undegraded":"2024-04-26T01:44:59.943455+0000","last_fullsized":"2024-04-26T01:44:59.943455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980685+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:44:59.980685+0000","last_peered":"2024-04-26T01:44:59.980685+0000","last_clean":"2024-04-26T01:44:59.980685+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:44:59.980685+0000","last_undegraded":"2024-04-26T01:44:59.980685+0000","last_fullsized":"2024-04-26T01:44:59.980685+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942739+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:44:59.942739+0000","last_peered":"2024-04-26T01:44:59.942739+0000","last_clean":"2024-04-26T01:44:59.942739+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:44:59.942739+0000","last_undegraded":"2024-04-26T01:44:59.942739+0000","last_fullsized":"2024-04-26T01:44:59.942739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145556+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:45:00.145556+0000","last_peered":"2024-04-26T01:45:00.145556+0000","last_clean":"2024-04-26T01:45:00.145556+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:45:00.145556+0000","last_undegraded":"2024-04-26T01:45:00.145556+0000","last_fullsized":"2024-04-26T01:45:00.145556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836713+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:44:59.836713+0000","last_peered":"2024-04-26T01:44:59.836713+0000","last_clean":"2024-04-26T01:44:59.836713+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:44:59.836713+0000","last_undegraded":"2024-04-26T01:44:59.836713+0000","last_fullsized":"2024-04-26T01:44:59.836713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386719+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:45:00.386719+0000","last_peered":"2024-04-26T01:45:00.386719+0000","last_clean":"2024-04-26T01:45:00.386719+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:45:00.386719+0000","last_undegraded":"2024-04-26T01:45:00.386719+0000","last_fullsized":"2024-04-26T01:45:00.386719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601960,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1406,"num_read_kb":816,"num_write":299,"num_write_kb":1446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2039808,"data_stored":1805880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252929,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":506662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514567,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33072,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712080,"statfs":{"total":95995035648,"available":95961169920,"internally_reserved":0,"allocated":1880064,"data_stored":1302495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36752,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708400,"statfs":{"total":95995035648,"available":95957401600,"internally_reserved":0,"allocated":1445888,"data_stored":900114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972434,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":900034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234070,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1646592,"data_stored":1100453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397596,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":506287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":577518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":507558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117100,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1585152,"data_stored":1031414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855465,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":508620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659234,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37088,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708064,"statfs":{"total":95995035648,"available":95957057536,"internally_reserved":0,"allocated":1794048,"data_stored":1230423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":72988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":205930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:45:06.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:45:07.104 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:45:07.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:07 smithi191 ceph-mon[32745]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:45:07.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:07 smithi052 ceph-mon[27851]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='client.25118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: Updating smithi052:/etc/ceph/ceph.conf 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: Updating smithi183:/etc/ceph/ceph.conf 2024-04-26T01:45:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:07 smithi183 ceph-mon[32752]: Updating smithi191:/etc/ceph/ceph.conf 2024-04-26T01:45:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:08 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:08 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: Updating smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:08.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:08.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:08 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 -- 172.21.15.52:0/4137897669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754102a90 msgr2=0x7f1754102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 --2- 172.21.15.52:0/4137897669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754102a90 0x7f1754102f10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17400098f0 tx=0x7f1740009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 -- 172.21.15.52:0/4137897669 shutdown_connections 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 --2- 172.21.15.52:0/4137897669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754103450 0x7f175410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 --2- 172.21.15.52:0/4137897669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 0x7f1754108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 --2- 172.21.15.52:0/4137897669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754102a90 0x7f1754102f10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 -- 172.21.15.52:0/4137897669 >> 172.21.15.52:0/4137897669 conn(0x7f1754079bf0 msgr2=0x7f175407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:09.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 -- 172.21.15.52:0/4137897669 shutdown_connections 2024-04-26T01:45:09.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.392+0000 7f175c200700 1 -- 172.21.15.52:0/4137897669 wait complete. 2024-04-26T01:45:09.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.393+0000 7f175c200700 1 Processor -- start 2024-04-26T01:45:09.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 -- start start 2024-04-26T01:45:09.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 0x7f1754199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 0x7f175419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1754112c80 con 0x7f1754103450 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1754112b00 con 0x7f1754108a80 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.394+0000 7f175c200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1754112e00 con 0x7f1754102a90 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1758e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 0x7f1754199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:09.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58336/0 (socket says 172.21.15.52:58336) 2024-04-26T01:45:09.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1758e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 0x7f1754199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38104/0 (socket says 172.21.15.52:38104) 2024-04-26T01:45:09.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 -- 172.21.15.52:0/3318392057 learned_addr learned my addr 172.21.15.52:0/3318392057 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:09.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f175a200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 0x7f175419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:09.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 -- 172.21.15.52:0/3318392057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 msgr2=0x7f175419daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 0x7f175419daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 -- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 msgr2=0x7f1754199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 0x7f1754199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.395+0000 7f1759800700 1 -- 172.21.15.52:0/3318392057 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175419e1e0 con 0x7f1754102a90 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f1759800700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f174800ec10 tx=0x7f174800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174800cc90 con 0x7f1754102a90 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174800ce30 con 0x7f1754102a90 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f175419e4d0 con 0x7f1754102a90 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748010430 con 0x7f1754102a90 2024-04-26T01:45:09.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.396+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1754074c70 con 0x7f1754102a90 2024-04-26T01:45:09.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.397+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1754069730 con 0x7f1754102a90 2024-04-26T01:45:09.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.399+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f1748004000 con 0x7f1754102a90 2024-04-26T01:45:09.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.401+0000 7f1752000700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 0x7f173c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:09.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.401+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f17480d0b40 con 0x7f1754102a90 2024-04-26T01:45:09.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.401+0000 7f1758e00700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 0x7f173c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:09.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.402+0000 7f1758e00700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 0x7f173c0a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17400098c0 tx=0x7f1740019040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:09.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.405+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f174809d040 con 0x7f1754102a90 2024-04-26T01:45:09.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.636+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1754062e80 con 0x7f173c0a3ec0 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:09.635 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:09.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:09 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:09 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.660+0000 7f1752000700 1 -- 172.21.15.52:0/3318392057 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257772 (secure 0 0 0) 0x7f1740065f00 con 0x7f173c0a3ec0 2024-04-26T01:45:09.659 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 msgr2=0x7f173c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 0x7f173c0a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17400098c0 tx=0x7f1740019040 comp rx=0 tx=0).stop 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 msgr2=0x7f17541991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f174800ec10 tx=0x7f174800ef20 comp rx=0 tx=0).stop 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 shutdown_connections 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f173c0a3ec0 0x7f173c0a6380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1754108a80 0x7f175419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1754102a90 0x7f17541991d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 --2- 172.21.15.52:0/3318392057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1754103450 0x7f1754199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:09.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.666+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 >> 172.21.15.52:0/3318392057 conn(0x7f1754079bf0 msgr2=0x7f17541017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:09.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.667+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 shutdown_connections 2024-04-26T01:45:09.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:09.667+0000 7f175c200700 1 -- 172.21.15.52:0/3318392057 wait complete. 2024-04-26T01:45:09.666 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: Updating smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:45:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:45:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:45:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:09 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:45:10.626 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-04-26T01:45:07.851298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389194,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1927,"num_read_kb":1392,"num_write":628,"num_write_kb":5300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":963,"ondisk_log_size":963,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399132,"kb_used_data":16428,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124542692,"statfs":{"total":1151940427776,"available":1151531716608,"internally_reserved":0,"allocated":16822272,"data_stored":10240542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":385,"num_read_kb":457,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.972847"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386883+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:45:00.386883+0000","last_peered":"2024-04-26T01:45:00.386883+0000","last_clean":"2024-04-26T01:45:00.386883+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:45:00.386883+0000","last_undegraded":"2024-04-26T01:45:00.386883+0000","last_fullsized":"2024-04-26T01:45:00.386883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145954+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:45:00.145954+0000","last_peered":"2024-04-26T01:45:00.145954+0000","last_clean":"2024-04-26T01:45:00.145954+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:45:00.145954+0000","last_undegraded":"2024-04-26T01:45:00.145954+0000","last_fullsized":"2024-04-26T01:45:00.145954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146356+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:45:00.146356+0000","last_peered":"2024-04-26T01:45:00.146356+0000","last_clean":"2024-04-26T01:45:00.146356+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:45:00.146356+0000","last_undegraded":"2024-04-26T01:45:00.146356+0000","last_fullsized":"2024-04-26T01:45:00.146356+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971689+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:44:59.971689+0000","last_peered":"2024-04-26T01:44:59.971689+0000","last_clean":"2024-04-26T01:44:59.971689+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:44:59.971689+0000","last_undegraded":"2024-04-26T01:44:59.971689+0000","last_fullsized":"2024-04-26T01:44:59.971689+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829979+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:44:59.829979+0000","last_peered":"2024-04-26T01:44:59.829979+0000","last_clean":"2024-04-26T01:44:59.829979+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:44:59.829979+0000","last_undegraded":"2024-04-26T01:44:59.829979+0000","last_fullsized":"2024-04-26T01:44:59.829979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983845+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:44:59.983845+0000","last_peered":"2024-04-26T01:44:59.983845+0000","last_clean":"2024-04-26T01:44:59.983845+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:44:59.983845+0000","last_undegraded":"2024-04-26T01:44:59.983845+0000","last_fullsized":"2024-04-26T01:44:59.983845+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832466+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:44:59.832466+0000","last_peered":"2024-04-26T01:44:59.832466+0000","last_clean":"2024-04-26T01:44:59.832466+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:44:59.832466+0000","last_undegraded":"2024-04-26T01:44:59.832466+0000","last_fullsized":"2024-04-26T01:44:59.832466+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942792+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:44:59.942792+0000","last_peered":"2024-04-26T01:44:59.942792+0000","last_clean":"2024-04-26T01:44:59.942792+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:44:59.942792+0000","last_undegraded":"2024-04-26T01:44:59.942792+0000","last_fullsized":"2024-04-26T01:44:59.942792+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145813+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:45:00.145813+0000","last_peered":"2024-04-26T01:45:00.145813+0000","last_clean":"2024-04-26T01:45:00.145813+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:45:00.145813+0000","last_undegraded":"2024-04-26T01:45:00.145813+0000","last_fullsized":"2024-04-26T01:45:00.145813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.984139+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:44:59.984139+0000","last_peered":"2024-04-26T01:44:59.984139+0000","last_clean":"2024-04-26T01:44:59.984139+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:44:59.984139+0000","last_undegraded":"2024-04-26T01:44:59.984139+0000","last_fullsized":"2024-04-26T01:44:59.984139+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833160+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:44:59.833160+0000","last_peered":"2024-04-26T01:44:59.833160+0000","last_clean":"2024-04-26T01:44:59.833160+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:44:59.833160+0000","last_undegraded":"2024-04-26T01:44:59.833160+0000","last_fullsized":"2024-04-26T01:44:59.833160+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942577+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:44:59.942577+0000","last_peered":"2024-04-26T01:44:59.942577+0000","last_clean":"2024-04-26T01:44:59.942577+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:44:59.942577+0000","last_undegraded":"2024-04-26T01:44:59.942577+0000","last_fullsized":"2024-04-26T01:44:59.942577+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836208+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:44:59.836208+0000","last_peered":"2024-04-26T01:44:59.836208+0000","last_clean":"2024-04-26T01:44:59.836208+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:44:59.836208+0000","last_undegraded":"2024-04-26T01:44:59.836208+0000","last_fullsized":"2024-04-26T01:44:59.836208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145890+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:45:00.145890+0000","last_peered":"2024-04-26T01:45:00.145890+0000","last_clean":"2024-04-26T01:45:00.145890+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:45:00.145890+0000","last_undegraded":"2024-04-26T01:45:00.145890+0000","last_fullsized":"2024-04-26T01:45:00.145890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145592+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:45:00.145592+0000","last_peered":"2024-04-26T01:45:00.145592+0000","last_clean":"2024-04-26T01:45:00.145592+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:45:00.145592+0000","last_undegraded":"2024-04-26T01:45:00.145592+0000","last_fullsized":"2024-04-26T01:45:00.145592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145452+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:45:00.145452+0000","last_peered":"2024-04-26T01:45:00.145452+0000","last_clean":"2024-04-26T01:45:00.145452+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:45:00.145452+0000","last_undegraded":"2024-04-26T01:45:00.145452+0000","last_fullsized":"2024-04-26T01:45:00.145452+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386985+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:45:00.386985+0000","last_peered":"2024-04-26T01:45:00.386985+0000","last_clean":"2024-04-26T01:45:00.386985+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:45:00.386985+0000","last_undegraded":"2024-04-26T01:45:00.386985+0000","last_fullsized":"2024-04-26T01:45:00.386985+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971739+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:44:59.971739+0000","last_peered":"2024-04-26T01:44:59.971739+0000","last_clean":"2024-04-26T01:44:59.971739+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:44:59.971739+0000","last_undegraded":"2024-04-26T01:44:59.971739+0000","last_fullsized":"2024-04-26T01:44:59.971739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146085+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:45:00.146085+0000","last_peered":"2024-04-26T01:45:00.146085+0000","last_clean":"2024-04-26T01:45:00.146085+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:45:00.146085+0000","last_undegraded":"2024-04-26T01:45:00.146085+0000","last_fullsized":"2024-04-26T01:45:00.146085+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.972070+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:44:59.972070+0000","last_peered":"2024-04-26T01:44:59.972070+0000","last_clean":"2024-04-26T01:44:59.972070+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:44:59.972070+0000","last_undegraded":"2024-04-26T01:44:59.972070+0000","last_fullsized":"2024-04-26T01:44:59.972070+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943401+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:44:59.943401+0000","last_peered":"2024-04-26T01:44:59.943401+0000","last_clean":"2024-04-26T01:44:59.943401+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:44:59.943401+0000","last_undegraded":"2024-04-26T01:44:59.943401+0000","last_fullsized":"2024-04-26T01:44:59.943401+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943600+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:44:59.943600+0000","last_peered":"2024-04-26T01:44:59.943600+0000","last_clean":"2024-04-26T01:44:59.943600+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:44:59.943600+0000","last_undegraded":"2024-04-26T01:44:59.943600+0000","last_fullsized":"2024-04-26T01:44:59.943600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146367+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:45:00.146367+0000","last_peered":"2024-04-26T01:45:00.146367+0000","last_clean":"2024-04-26T01:45:00.146367+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:45:00.146367+0000","last_undegraded":"2024-04-26T01:45:00.146367+0000","last_fullsized":"2024-04-26T01:45:00.146367+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386813+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:45:00.386813+0000","last_peered":"2024-04-26T01:45:00.386813+0000","last_clean":"2024-04-26T01:45:00.386813+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:45:00.386813+0000","last_undegraded":"2024-04-26T01:45:00.386813+0000","last_fullsized":"2024-04-26T01:45:00.386813+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145523+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:45:00.145523+0000","last_peered":"2024-04-26T01:45:00.145523+0000","last_clean":"2024-04-26T01:45:00.145523+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:45:00.145523+0000","last_undegraded":"2024-04-26T01:45:00.145523+0000","last_fullsized":"2024-04-26T01:45:00.145523+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830007+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:44:59.830007+0000","last_peered":"2024-04-26T01:44:59.830007+0000","last_clean":"2024-04-26T01:44:59.830007+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:44:59.830007+0000","last_undegraded":"2024-04-26T01:44:59.830007+0000","last_fullsized":"2024-04-26T01:44:59.830007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980895+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:44:59.980895+0000","last_peered":"2024-04-26T01:44:59.980895+0000","last_clean":"2024-04-26T01:44:59.980895+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:44:59.980895+0000","last_undegraded":"2024-04-26T01:44:59.980895+0000","last_fullsized":"2024-04-26T01:44:59.980895+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"95'7","reported_seq":258,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:04.706462+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:45:04.706462+0000","last_peered":"2024-04-26T01:45:04.706462+0000","last_clean":"2024-04-26T01:45:04.706462+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:45:04.706462+0000","last_undegraded":"2024-04-26T01:45:04.706462+0000","last_fullsized":"2024-04-26T01:45:04.706462+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829791+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:44:59.829791+0000","last_peered":"2024-04-26T01:44:59.829791+0000","last_clean":"2024-04-26T01:44:59.829791+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:44:59.829791+0000","last_undegraded":"2024-04-26T01:44:59.829791+0000","last_fullsized":"2024-04-26T01:44:59.829791+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983749+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:44:59.983749+0000","last_peered":"2024-04-26T01:44:59.983749+0000","last_clean":"2024-04-26T01:44:59.983749+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:44:59.983749+0000","last_undegraded":"2024-04-26T01:44:59.983749+0000","last_fullsized":"2024-04-26T01:44:59.983749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980762+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:44:59.980762+0000","last_peered":"2024-04-26T01:44:59.980762+0000","last_clean":"2024-04-26T01:44:59.980762+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:44:59.980762+0000","last_undegraded":"2024-04-26T01:44:59.980762+0000","last_fullsized":"2024-04-26T01:44:59.980762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836546+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:44:59.836546+0000","last_peered":"2024-04-26T01:44:59.836546+0000","last_clean":"2024-04-26T01:44:59.836546+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:44:59.836546+0000","last_undegraded":"2024-04-26T01:44:59.836546+0000","last_fullsized":"2024-04-26T01:44:59.836546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942696+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:44:59.942696+0000","last_peered":"2024-04-26T01:44:59.942696+0000","last_clean":"2024-04-26T01:44:59.942696+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:44:59.942696+0000","last_undegraded":"2024-04-26T01:44:59.942696+0000","last_fullsized":"2024-04-26T01:44:59.942696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980603+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:44:59.980603+0000","last_peered":"2024-04-26T01:44:59.980603+0000","last_clean":"2024-04-26T01:44:59.980603+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:44:59.980603+0000","last_undegraded":"2024-04-26T01:44:59.980603+0000","last_fullsized":"2024-04-26T01:44:59.980603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971311+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:44:59.971311+0000","last_peered":"2024-04-26T01:44:59.971311+0000","last_clean":"2024-04-26T01:44:59.971311+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:44:59.971311+0000","last_undegraded":"2024-04-26T01:44:59.971311+0000","last_fullsized":"2024-04-26T01:44:59.971311+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":252,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:04.705874+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:45:04.705874+0000","last_peered":"2024-04-26T01:45:04.705874+0000","last_clean":"2024-04-26T01:45:04.705874+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:45:04.705874+0000","last_undegraded":"2024-04-26T01:45:04.705874+0000","last_fullsized":"2024-04-26T01:45:04.705874+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":219,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145864+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:45:00.145864+0000","last_peered":"2024-04-26T01:45:00.145864+0000","last_clean":"2024-04-26T01:45:00.145864+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:45:00.145864+0000","last_undegraded":"2024-04-26T01:45:00.145864+0000","last_fullsized":"2024-04-26T01:45:00.145864+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832945+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:44:59.832945+0000","last_peered":"2024-04-26T01:44:59.832945+0000","last_clean":"2024-04-26T01:44:59.832945+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:44:59.832945+0000","last_undegraded":"2024-04-26T01:44:59.832945+0000","last_fullsized":"2024-04-26T01:44:59.832945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146547+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:45:00.146547+0000","last_peered":"2024-04-26T01:45:00.146547+0000","last_clean":"2024-04-26T01:45:00.146547+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:45:00.146547+0000","last_undegraded":"2024-04-26T01:45:00.146547+0000","last_fullsized":"2024-04-26T01:45:00.146547+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.981042+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:44:59.981042+0000","last_peered":"2024-04-26T01:44:59.981042+0000","last_clean":"2024-04-26T01:44:59.981042+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:44:59.981042+0000","last_undegraded":"2024-04-26T01:44:59.981042+0000","last_fullsized":"2024-04-26T01:44:59.981042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145594+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:45:00.145594+0000","last_peered":"2024-04-26T01:45:00.145594+0000","last_clean":"2024-04-26T01:45:00.145594+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:45:00.145594+0000","last_undegraded":"2024-04-26T01:45:00.145594+0000","last_fullsized":"2024-04-26T01:45:00.145594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":518,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:05.706828+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:45:05.706828+0000","last_peered":"2024-04-26T01:45:05.706828+0000","last_clean":"2024-04-26T01:45:05.706828+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:45:05.706828+0000","last_undegraded":"2024-04-26T01:45:05.706828+0000","last_fullsized":"2024-04-26T01:45:05.706828+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71438,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":450,"num_read_kb":295,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829990+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:44:59.829990+0000","last_peered":"2024-04-26T01:44:59.829990+0000","last_clean":"2024-04-26T01:44:59.829990+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:44:59.829990+0000","last_undegraded":"2024-04-26T01:44:59.829990+0000","last_fullsized":"2024-04-26T01:44:59.829990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145896+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:45:00.145896+0000","last_peered":"2024-04-26T01:45:00.145896+0000","last_clean":"2024-04-26T01:45:00.145896+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:45:00.145896+0000","last_undegraded":"2024-04-26T01:45:00.145896+0000","last_fullsized":"2024-04-26T01:45:00.145896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836995+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:44:59.836995+0000","last_peered":"2024-04-26T01:44:59.836995+0000","last_clean":"2024-04-26T01:44:59.836995+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:44:59.836995+0000","last_undegraded":"2024-04-26T01:44:59.836995+0000","last_fullsized":"2024-04-26T01:44:59.836995+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980687+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:44:59.980687+0000","last_peered":"2024-04-26T01:44:59.980687+0000","last_clean":"2024-04-26T01:44:59.980687+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:44:59.980687+0000","last_undegraded":"2024-04-26T01:44:59.980687+0000","last_fullsized":"2024-04-26T01:44:59.980687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832836+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:44:59.832836+0000","last_peered":"2024-04-26T01:44:59.832836+0000","last_clean":"2024-04-26T01:44:59.832836+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:44:59.832836+0000","last_undegraded":"2024-04-26T01:44:59.832836+0000","last_fullsized":"2024-04-26T01:44:59.832836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"95'7","reported_seq":259,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:03.951772+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:45:03.951772+0000","last_peered":"2024-04-26T01:45:03.951772+0000","last_clean":"2024-04-26T01:45:03.951772+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:45:03.951772+0000","last_undegraded":"2024-04-26T01:45:03.951772+0000","last_fullsized":"2024-04-26T01:45:03.951772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832336+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:44:59.832336+0000","last_peered":"2024-04-26T01:44:59.832336+0000","last_clean":"2024-04-26T01:44:59.832336+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:44:59.832336+0000","last_undegraded":"2024-04-26T01:44:59.832336+0000","last_fullsized":"2024-04-26T01:44:59.832336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829747+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:44:59.829747+0000","last_peered":"2024-04-26T01:44:59.829747+0000","last_clean":"2024-04-26T01:44:59.829747+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:44:59.829747+0000","last_undegraded":"2024-04-26T01:44:59.829747+0000","last_fullsized":"2024-04-26T01:44:59.829747+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971521+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:44:59.971521+0000","last_peered":"2024-04-26T01:44:59.971521+0000","last_clean":"2024-04-26T01:44:59.971521+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:44:59.971521+0000","last_undegraded":"2024-04-26T01:44:59.971521+0000","last_fullsized":"2024-04-26T01:44:59.971521+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"95'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830365+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:44:59.830365+0000","last_peered":"2024-04-26T01:44:59.830365+0000","last_clean":"2024-04-26T01:44:59.830365+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:44:59.830365+0000","last_undegraded":"2024-04-26T01:44:59.830365+0000","last_fullsized":"2024-04-26T01:44:59.830365+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386745+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:45:00.386745+0000","last_peered":"2024-04-26T01:45:00.386745+0000","last_clean":"2024-04-26T01:45:00.386745+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:45:00.386745+0000","last_undegraded":"2024-04-26T01:45:00.386745+0000","last_fullsized":"2024-04-26T01:45:00.386745+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971578+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:44:59.971578+0000","last_peered":"2024-04-26T01:44:59.971578+0000","last_clean":"2024-04-26T01:44:59.971578+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:44:59.971578+0000","last_undegraded":"2024-04-26T01:44:59.971578+0000","last_fullsized":"2024-04-26T01:44:59.971578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983891+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:44:59.983891+0000","last_peered":"2024-04-26T01:44:59.983891+0000","last_clean":"2024-04-26T01:44:59.983891+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:44:59.983891+0000","last_undegraded":"2024-04-26T01:44:59.983891+0000","last_fullsized":"2024-04-26T01:44:59.983891+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832756+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:44:59.832756+0000","last_peered":"2024-04-26T01:44:59.832756+0000","last_clean":"2024-04-26T01:44:59.832756+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:44:59.832756+0000","last_undegraded":"2024-04-26T01:44:59.832756+0000","last_fullsized":"2024-04-26T01:44:59.832756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983790+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:44:59.983790+0000","last_peered":"2024-04-26T01:44:59.983790+0000","last_clean":"2024-04-26T01:44:59.983790+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:44:59.983790+0000","last_undegraded":"2024-04-26T01:44:59.983790+0000","last_fullsized":"2024-04-26T01:44:59.983790+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980893+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:44:59.980893+0000","last_peered":"2024-04-26T01:44:59.980893+0000","last_clean":"2024-04-26T01:44:59.980893+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:44:59.980893+0000","last_undegraded":"2024-04-26T01:44:59.980893+0000","last_fullsized":"2024-04-26T01:44:59.980893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":254,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:04.953569+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:45:04.953569+0000","last_peered":"2024-04-26T01:45:04.953569+0000","last_clean":"2024-04-26T01:45:04.953569+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:45:04.953569+0000","last_undegraded":"2024-04-26T01:45:04.953569+0000","last_fullsized":"2024-04-26T01:45:04.953569+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943097+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:44:59.943097+0000","last_peered":"2024-04-26T01:44:59.943097+0000","last_clean":"2024-04-26T01:44:59.943097+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:44:59.943097+0000","last_undegraded":"2024-04-26T01:44:59.943097+0000","last_fullsized":"2024-04-26T01:44:59.943097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942566+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:44:59.942566+0000","last_peered":"2024-04-26T01:44:59.942566+0000","last_clean":"2024-04-26T01:44:59.942566+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:44:59.942566+0000","last_undegraded":"2024-04-26T01:44:59.942566+0000","last_fullsized":"2024-04-26T01:44:59.942566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"95'148","reported_seq":408,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.003957+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:45:02.003957+0000","last_peered":"2024-04-26T01:45:02.003957+0000","last_clean":"2024-04-26T01:45:02.003957+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:45:02.003957+0000","last_undegraded":"2024-04-26T01:45:02.003957+0000","last_fullsized":"2024-04-26T01:45:02.003957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.835940+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:44:59.835940+0000","last_peered":"2024-04-26T01:44:59.835940+0000","last_clean":"2024-04-26T01:44:59.835940+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:44:59.835940+0000","last_undegraded":"2024-04-26T01:44:59.835940+0000","last_fullsized":"2024-04-26T01:44:59.835940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832637+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:44:59.832637+0000","last_peered":"2024-04-26T01:44:59.832637+0000","last_clean":"2024-04-26T01:44:59.832637+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:44:59.832637+0000","last_undegraded":"2024-04-26T01:44:59.832637+0000","last_fullsized":"2024-04-26T01:44:59.832637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830163+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:44:59.830163+0000","last_peered":"2024-04-26T01:44:59.830163+0000","last_clean":"2024-04-26T01:44:59.830163+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:44:59.830163+0000","last_undegraded":"2024-04-26T01:44:59.830163+0000","last_fullsized":"2024-04-26T01:44:59.830163+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836146+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:44:59.836146+0000","last_peered":"2024-04-26T01:44:59.836146+0000","last_clean":"2024-04-26T01:44:59.836146+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:44:59.836146+0000","last_undegraded":"2024-04-26T01:44:59.836146+0000","last_fullsized":"2024-04-26T01:44:59.836146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"95'102","reported_seq":223,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.483945+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:45:02.483945+0000","last_peered":"2024-04-26T01:45:02.483945+0000","last_clean":"2024-04-26T01:45:02.483945+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:45:02.483945+0000","last_undegraded":"2024-04-26T01:45:02.483945+0000","last_fullsized":"2024-04-26T01:45:02.483945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943583+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:44:59.943583+0000","last_peered":"2024-04-26T01:44:59.943583+0000","last_clean":"2024-04-26T01:44:59.943583+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:44:59.943583+0000","last_undegraded":"2024-04-26T01:44:59.943583+0000","last_fullsized":"2024-04-26T01:44:59.943583+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942746+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:44:59.942746+0000","last_peered":"2024-04-26T01:44:59.942746+0000","last_clean":"2024-04-26T01:44:59.942746+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:44:59.942746+0000","last_undegraded":"2024-04-26T01:44:59.942746+0000","last_fullsized":"2024-04-26T01:44:59.942746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146000+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:45:00.146000+0000","last_peered":"2024-04-26T01:45:00.146000+0000","last_clean":"2024-04-26T01:45:00.146000+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:45:00.146000+0000","last_undegraded":"2024-04-26T01:45:00.146000+0000","last_fullsized":"2024-04-26T01:45:00.146000+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145455+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:45:00.145455+0000","last_peered":"2024-04-26T01:45:00.145455+0000","last_clean":"2024-04-26T01:45:00.145455+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:45:00.145455+0000","last_undegraded":"2024-04-26T01:45:00.145455+0000","last_fullsized":"2024-04-26T01:45:00.145455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829725+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:44:59.829725+0000","last_peered":"2024-04-26T01:44:59.829725+0000","last_clean":"2024-04-26T01:44:59.829725+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:44:59.829725+0000","last_undegraded":"2024-04-26T01:44:59.829725+0000","last_fullsized":"2024-04-26T01:44:59.829725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942730+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:44:59.942730+0000","last_peered":"2024-04-26T01:44:59.942730+0000","last_clean":"2024-04-26T01:44:59.942730+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:44:59.942730+0000","last_undegraded":"2024-04-26T01:44:59.942730+0000","last_fullsized":"2024-04-26T01:44:59.942730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386613+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:45:00.386613+0000","last_peered":"2024-04-26T01:45:00.386613+0000","last_clean":"2024-04-26T01:45:00.386613+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:45:00.386613+0000","last_undegraded":"2024-04-26T01:45:00.386613+0000","last_fullsized":"2024-04-26T01:45:00.386613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145777+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:45:00.145777+0000","last_peered":"2024-04-26T01:45:00.145777+0000","last_clean":"2024-04-26T01:45:00.145777+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:45:00.145777+0000","last_undegraded":"2024-04-26T01:45:00.145777+0000","last_fullsized":"2024-04-26T01:45:00.145777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943095+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:44:59.943095+0000","last_peered":"2024-04-26T01:44:59.943095+0000","last_clean":"2024-04-26T01:44:59.943095+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:44:59.943095+0000","last_undegraded":"2024-04-26T01:44:59.943095+0000","last_fullsized":"2024-04-26T01:44:59.943095+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830189+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:44:59.830189+0000","last_peered":"2024-04-26T01:44:59.830189+0000","last_clean":"2024-04-26T01:44:59.830189+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:44:59.830189+0000","last_undegraded":"2024-04-26T01:44:59.830189+0000","last_fullsized":"2024-04-26T01:44:59.830189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832778+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:44:59.832778+0000","last_peered":"2024-04-26T01:44:59.832778+0000","last_clean":"2024-04-26T01:44:59.832778+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:44:59.832778+0000","last_undegraded":"2024-04-26T01:44:59.832778+0000","last_fullsized":"2024-04-26T01:44:59.832778+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942645+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:44:59.942645+0000","last_peered":"2024-04-26T01:44:59.942645+0000","last_clean":"2024-04-26T01:44:59.942645+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:44:59.942645+0000","last_undegraded":"2024-04-26T01:44:59.942645+0000","last_fullsized":"2024-04-26T01:44:59.942645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833327+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:44:59.833327+0000","last_peered":"2024-04-26T01:44:59.833327+0000","last_clean":"2024-04-26T01:44:59.833327+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:44:59.833327+0000","last_undegraded":"2024-04-26T01:44:59.833327+0000","last_fullsized":"2024-04-26T01:44:59.833327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"95'65","reported_seq":125,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.017193+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:45:02.017193+0000","last_peered":"2024-04-26T01:45:02.017193+0000","last_clean":"2024-04-26T01:45:02.017193+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:45:02.017193+0000","last_undegraded":"2024-04-26T01:45:02.017193+0000","last_fullsized":"2024-04-26T01:45:02.017193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832937+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:44:59.832937+0000","last_peered":"2024-04-26T01:44:59.832937+0000","last_clean":"2024-04-26T01:44:59.832937+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:44:59.832937+0000","last_undegraded":"2024-04-26T01:44:59.832937+0000","last_fullsized":"2024-04-26T01:44:59.832937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836360+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:44:59.836360+0000","last_peered":"2024-04-26T01:44:59.836360+0000","last_clean":"2024-04-26T01:44:59.836360+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:44:59.836360+0000","last_undegraded":"2024-04-26T01:44:59.836360+0000","last_fullsized":"2024-04-26T01:44:59.836360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833557+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:44:59.833557+0000","last_peered":"2024-04-26T01:44:59.833557+0000","last_clean":"2024-04-26T01:44:59.833557+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:44:59.833557+0000","last_undegraded":"2024-04-26T01:44:59.833557+0000","last_fullsized":"2024-04-26T01:44:59.833557+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386692+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:45:00.386692+0000","last_peered":"2024-04-26T01:45:00.386692+0000","last_clean":"2024-04-26T01:45:00.386692+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:45:00.386692+0000","last_undegraded":"2024-04-26T01:45:00.386692+0000","last_fullsized":"2024-04-26T01:45:00.386692+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145926+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:45:00.145926+0000","last_peered":"2024-04-26T01:45:00.145926+0000","last_clean":"2024-04-26T01:45:00.145926+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:45:00.145926+0000","last_undegraded":"2024-04-26T01:45:00.145926+0000","last_fullsized":"2024-04-26T01:45:00.145926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145769+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:45:00.145769+0000","last_peered":"2024-04-26T01:45:00.145769+0000","last_clean":"2024-04-26T01:45:00.145769+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:45:00.145769+0000","last_undegraded":"2024-04-26T01:45:00.145769+0000","last_fullsized":"2024-04-26T01:45:00.145769+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145984+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:45:00.145984+0000","last_peered":"2024-04-26T01:45:00.145984+0000","last_clean":"2024-04-26T01:45:00.145984+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:45:00.145984+0000","last_undegraded":"2024-04-26T01:45:00.145984+0000","last_fullsized":"2024-04-26T01:45:00.145984+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.835967+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:44:59.835967+0000","last_peered":"2024-04-26T01:44:59.835967+0000","last_clean":"2024-04-26T01:44:59.835967+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:44:59.835967+0000","last_undegraded":"2024-04-26T01:44:59.835967+0000","last_fullsized":"2024-04-26T01:44:59.835967+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386872+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:45:00.386872+0000","last_peered":"2024-04-26T01:45:00.386872+0000","last_clean":"2024-04-26T01:45:00.386872+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:45:00.386872+0000","last_undegraded":"2024-04-26T01:45:00.386872+0000","last_fullsized":"2024-04-26T01:45:00.386872+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146069+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:45:00.146069+0000","last_peered":"2024-04-26T01:45:00.146069+0000","last_clean":"2024-04-26T01:45:00.146069+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:45:00.146069+0000","last_undegraded":"2024-04-26T01:45:00.146069+0000","last_fullsized":"2024-04-26T01:45:00.146069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943455+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:44:59.943455+0000","last_peered":"2024-04-26T01:44:59.943455+0000","last_clean":"2024-04-26T01:44:59.943455+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:44:59.943455+0000","last_undegraded":"2024-04-26T01:44:59.943455+0000","last_fullsized":"2024-04-26T01:44:59.943455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980685+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:44:59.980685+0000","last_peered":"2024-04-26T01:44:59.980685+0000","last_clean":"2024-04-26T01:44:59.980685+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:44:59.980685+0000","last_undegraded":"2024-04-26T01:44:59.980685+0000","last_fullsized":"2024-04-26T01:44:59.980685+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942739+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:44:59.942739+0000","last_peered":"2024-04-26T01:44:59.942739+0000","last_clean":"2024-04-26T01:44:59.942739+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:44:59.942739+0000","last_undegraded":"2024-04-26T01:44:59.942739+0000","last_fullsized":"2024-04-26T01:44:59.942739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145556+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:45:00.145556+0000","last_peered":"2024-04-26T01:45:00.145556+0000","last_clean":"2024-04-26T01:45:00.145556+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:45:00.145556+0000","last_undegraded":"2024-04-26T01:45:00.145556+0000","last_fullsized":"2024-04-26T01:45:00.145556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836713+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:44:59.836713+0000","last_peered":"2024-04-26T01:44:59.836713+0000","last_clean":"2024-04-26T01:44:59.836713+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:44:59.836713+0000","last_undegraded":"2024-04-26T01:44:59.836713+0000","last_fullsized":"2024-04-26T01:44:59.836713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386719+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:45:00.386719+0000","last_peered":"2024-04-26T01:45:00.386719+0000","last_clean":"2024-04-26T01:45:00.386719+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:45:00.386719+0000","last_undegraded":"2024-04-26T01:45:00.386719+0000","last_fullsized":"2024-04-26T01:45:00.386719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":798592,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1690,"num_read_kb":1062,"num_write":425,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2629632,"data_stored":2395776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252930,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":506662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514568,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33204,"kb_used_data":2028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711948,"statfs":{"total":95995035648,"available":95961034752,"internally_reserved":0,"allocated":2076672,"data_stored":1499127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710798,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36752,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708400,"statfs":{"total":95995035648,"available":95957401600,"internally_reserved":0,"allocated":1445888,"data_stored":900114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972435,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":900034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234071,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1100453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397597,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":506287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":577518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":507558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117101,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37196,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707956,"statfs":{"total":95995035648,"available":95956946944,"internally_reserved":0,"allocated":1781760,"data_stored":1228046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855466,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":508620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659235,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37400,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707752,"statfs":{"total":95995035648,"available":95956738048,"internally_reserved":0,"allocated":1990656,"data_stored":1427055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":72988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":402562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:45:10.630 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T01:45:10.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T01:45:11.134 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:11 smithi052 ceph-mon[27851]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:11 smithi052 ceph-mon[27851]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T01:45:11.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:11 smithi191 ceph-mon[32745]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:11 smithi191 ceph-mon[32745]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T01:45:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:11 smithi183 ceph-mon[32752]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:11 smithi183 ceph-mon[32752]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-26T01:45:13.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 -- 172.21.15.52:0/3940113391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb368109c80 msgr2=0x7fb36810dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:13.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 --2- 172.21.15.52:0/3940113391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb368109c80 0x7fb36810dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb354009920 tx=0x7fb354009c30 comp rx=0 tx=0).stop 2024-04-26T01:45:13.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 -- 172.21.15.52:0/3940113391 shutdown_connections 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 --2- 172.21.15.52:0/3940113391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 0x7fb368115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 --2- 172.21.15.52:0/3940113391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3681092d0 0x7fb3681096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 --2- 172.21.15.52:0/3940113391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb368109c80 0x7fb36810dcd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 -- 172.21.15.52:0/3940113391 >> 172.21.15.52:0/3940113391 conn(0x7fb368076210 msgr2=0x7fb368107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 -- 172.21.15.52:0/3940113391 shutdown_connections 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.391+0000 7fb36ec00700 1 -- 172.21.15.52:0/3940113391 wait complete. 2024-04-26T01:45:13.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 Processor -- start 2024-04-26T01:45:13.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 -- start start 2024-04-26T01:45:13.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 0x7fb3681a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:13.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:13.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 0x7fb36819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:13.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb368119220 con 0x7fb3681092d0 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681190a0 con 0x7fb368109c80 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.393+0000 7fb36ec00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3681193a0 con 0x7fb36810e3b0 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 0x7fb3681a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42556/0 (socket says 172.21.15.52:42556) 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 0x7fb3681a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38116/0 (socket says 172.21.15.52:38116) 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 -- 172.21.15.52:0/4203570607 learned_addr learned my addr 172.21.15.52:0/4203570607 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:13.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb36cc00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 0x7fb36819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 -- 172.21.15.52:0/4203570607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 msgr2=0x7fb36819f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 0x7fb36819f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 -- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 msgr2=0x7fb3681a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 0x7fb3681a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.394+0000 7fb367400700 1 -- 172.21.15.52:0/4203570607 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36819ff40 con 0x7fb368109c80 2024-04-26T01:45:13.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.395+0000 7fb367400700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb354009230 tx=0x7fb35400f950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:13.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.399+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354027070 con 0x7fb368109c80 2024-04-26T01:45:13.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.399+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3540054f0 con 0x7fb368109c80 2024-04-26T01:45:13.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.399+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354004930 con 0x7fb368109c80 2024-04-26T01:45:13.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.399+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3681a01d0 con 0x7fb368109c80 2024-04-26T01:45:13.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.399+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3681ac510 con 0x7fb368109c80 2024-04-26T01:45:13.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.400+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3681a0530 con 0x7fb368109c80 2024-04-26T01:45:13.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.402+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fb35402a070 con 0x7fb368109c80 2024-04-26T01:45:13.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.403+0000 7fb364c00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 0x7fb34c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:13.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.403+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fb3540d1210 con 0x7fb368109c80 2024-04-26T01:45:13.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.403+0000 7fb367e00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 0x7fb34c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:13.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.404+0000 7fb367e00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 0x7fb34c0a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb358009750 tx=0x7fb358009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:13.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.408+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb35405eff0 con 0x7fb368109c80 2024-04-26T01:45:13.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.634+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb368069730 con 0x7fb34c0a3ec0 2024-04-26T01:45:13.633 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:13 smithi052 ceph-mon[27851]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T01:45:13.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.644+0000 7fb364c00700 1 -- 172.21.15.52:0/4203570607 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7fb368069730 con 0x7fb34c0a3ec0 2024-04-26T01:45:13.643 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 msgr2=0x7fb34c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 0x7fb34c0a6380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb358009750 tx=0x7fb358009310 comp rx=0 tx=0).stop 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 msgr2=0x7fb3681a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb354009230 tx=0x7fb35400f950 comp rx=0 tx=0).stop 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 shutdown_connections 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb34c0a3ec0 0x7fb34c0a6380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb36810e3b0 0x7fb36819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3681092d0 0x7fb3681a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 --2- 172.21.15.52:0/4203570607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb368109c80 0x7fb3681a5ad0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:13.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 >> 172.21.15.52:0/4203570607 conn(0x7fb368076210 msgr2=0x7fb3681086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:13.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 shutdown_connections 2024-04-26T01:45:13.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:13.651+0000 7fb36ec00700 1 -- 172.21.15.52:0/4203570607 wait complete. 2024-04-26T01:45:13.651 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-04-26T01:45:13.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:13 smithi191 ceph-mon[32745]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T01:45:13.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:13 smithi183 ceph-mon[32752]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-26T01:45:14.565 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-26T01:45:11.853112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389194,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1957,"num_read_kb":1407,"num_write":628,"num_write_kb":5300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":963,"ondisk_log_size":963,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":399132,"kb_used_data":16428,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124542692,"statfs":{"total":1151940427776,"available":1151531716608,"internally_reserved":0,"allocated":16822272,"data_stored":10240542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":415,"num_read_kb":472,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.974660"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386916+0000","last_change":"2024-04-26T01:41:03.295264+0000","last_active":"2024-04-26T01:45:00.386916+0000","last_peered":"2024-04-26T01:45:00.386916+0000","last_clean":"2024-04-26T01:45:00.386916+0000","last_became_active":"2024-04-26T01:41:03.294841+0000","last_became_peered":"2024-04-26T01:41:03.294841+0000","last_unstale":"2024-04-26T01:45:00.386916+0000","last_undegraded":"2024-04-26T01:45:00.386916+0000","last_fullsized":"2024-04-26T01:45:00.386916+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:51.035908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145954+0000","last_change":"2024-04-26T01:41:03.285860+0000","last_active":"2024-04-26T01:45:00.145954+0000","last_peered":"2024-04-26T01:45:00.145954+0000","last_clean":"2024-04-26T01:45:00.145954+0000","last_became_active":"2024-04-26T01:41:03.285455+0000","last_became_peered":"2024-04-26T01:41:03.285455+0000","last_unstale":"2024-04-26T01:45:00.145954+0000","last_undegraded":"2024-04-26T01:45:00.145954+0000","last_fullsized":"2024-04-26T01:45:00.145954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:30.920650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146356+0000","last_change":"2024-04-26T01:41:03.291090+0000","last_active":"2024-04-26T01:45:00.146356+0000","last_peered":"2024-04-26T01:45:00.146356+0000","last_clean":"2024-04-26T01:45:00.146356+0000","last_became_active":"2024-04-26T01:41:03.290523+0000","last_became_peered":"2024-04-26T01:41:03.290523+0000","last_unstale":"2024-04-26T01:45:00.146356+0000","last_undegraded":"2024-04-26T01:45:00.146356+0000","last_fullsized":"2024-04-26T01:45:00.146356+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:35:02.046245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971689+0000","last_change":"2024-04-26T01:41:47.501539+0000","last_active":"2024-04-26T01:44:59.971689+0000","last_peered":"2024-04-26T01:44:59.971689+0000","last_clean":"2024-04-26T01:44:59.971689+0000","last_became_active":"2024-04-26T01:41:47.500810+0000","last_became_peered":"2024-04-26T01:41:47.500810+0000","last_unstale":"2024-04-26T01:44:59.971689+0000","last_undegraded":"2024-04-26T01:44:59.971689+0000","last_fullsized":"2024-04-26T01:44:59.971689+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:18.265592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829979+0000","last_change":"2024-04-26T01:41:03.299080+0000","last_active":"2024-04-26T01:44:59.829979+0000","last_peered":"2024-04-26T01:44:59.829979+0000","last_clean":"2024-04-26T01:44:59.829979+0000","last_became_active":"2024-04-26T01:41:03.298720+0000","last_became_peered":"2024-04-26T01:41:03.298720+0000","last_unstale":"2024-04-26T01:44:59.829979+0000","last_undegraded":"2024-04-26T01:44:59.829979+0000","last_fullsized":"2024-04-26T01:44:59.829979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:39.606334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983845+0000","last_change":"2024-04-26T01:41:03.288766+0000","last_active":"2024-04-26T01:44:59.983845+0000","last_peered":"2024-04-26T01:44:59.983845+0000","last_clean":"2024-04-26T01:44:59.983845+0000","last_became_active":"2024-04-26T01:41:03.288441+0000","last_became_peered":"2024-04-26T01:41:03.288441+0000","last_unstale":"2024-04-26T01:44:59.983845+0000","last_undegraded":"2024-04-26T01:44:59.983845+0000","last_fullsized":"2024-04-26T01:44:59.983845+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:08:22.726096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832466+0000","last_change":"2024-04-26T01:41:03.282578+0000","last_active":"2024-04-26T01:44:59.832466+0000","last_peered":"2024-04-26T01:44:59.832466+0000","last_clean":"2024-04-26T01:44:59.832466+0000","last_became_active":"2024-04-26T01:41:03.282245+0000","last_became_peered":"2024-04-26T01:41:03.282245+0000","last_unstale":"2024-04-26T01:44:59.832466+0000","last_undegraded":"2024-04-26T01:44:59.832466+0000","last_fullsized":"2024-04-26T01:44:59.832466+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:12:23.618374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942792+0000","last_change":"2024-04-26T01:41:03.287588+0000","last_active":"2024-04-26T01:44:59.942792+0000","last_peered":"2024-04-26T01:44:59.942792+0000","last_clean":"2024-04-26T01:44:59.942792+0000","last_became_active":"2024-04-26T01:41:03.286959+0000","last_became_peered":"2024-04-26T01:41:03.286959+0000","last_unstale":"2024-04-26T01:44:59.942792+0000","last_undegraded":"2024-04-26T01:44:59.942792+0000","last_fullsized":"2024-04-26T01:44:59.942792+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:50:15.231813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145813+0000","last_change":"2024-04-26T01:41:03.277069+0000","last_active":"2024-04-26T01:45:00.145813+0000","last_peered":"2024-04-26T01:45:00.145813+0000","last_clean":"2024-04-26T01:45:00.145813+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:45:00.145813+0000","last_undegraded":"2024-04-26T01:45:00.145813+0000","last_fullsized":"2024-04-26T01:45:00.145813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:11:56.423746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.984139+0000","last_change":"2024-04-26T01:41:03.275167+0000","last_active":"2024-04-26T01:44:59.984139+0000","last_peered":"2024-04-26T01:44:59.984139+0000","last_clean":"2024-04-26T01:44:59.984139+0000","last_became_active":"2024-04-26T01:41:03.274798+0000","last_became_peered":"2024-04-26T01:41:03.274798+0000","last_unstale":"2024-04-26T01:44:59.984139+0000","last_undegraded":"2024-04-26T01:44:59.984139+0000","last_fullsized":"2024-04-26T01:44:59.984139+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:23:05.216052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833160+0000","last_change":"2024-04-26T01:41:03.281724+0000","last_active":"2024-04-26T01:44:59.833160+0000","last_peered":"2024-04-26T01:44:59.833160+0000","last_clean":"2024-04-26T01:44:59.833160+0000","last_became_active":"2024-04-26T01:41:03.281291+0000","last_became_peered":"2024-04-26T01:41:03.281291+0000","last_unstale":"2024-04-26T01:44:59.833160+0000","last_undegraded":"2024-04-26T01:44:59.833160+0000","last_fullsized":"2024-04-26T01:44:59.833160+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:25.888624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942577+0000","last_change":"2024-04-26T01:41:03.275539+0000","last_active":"2024-04-26T01:44:59.942577+0000","last_peered":"2024-04-26T01:44:59.942577+0000","last_clean":"2024-04-26T01:44:59.942577+0000","last_became_active":"2024-04-26T01:41:03.275099+0000","last_became_peered":"2024-04-26T01:41:03.275099+0000","last_unstale":"2024-04-26T01:44:59.942577+0000","last_undegraded":"2024-04-26T01:44:59.942577+0000","last_fullsized":"2024-04-26T01:44:59.942577+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:55.687124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836208+0000","last_change":"2024-04-26T01:41:03.302035+0000","last_active":"2024-04-26T01:44:59.836208+0000","last_peered":"2024-04-26T01:44:59.836208+0000","last_clean":"2024-04-26T01:44:59.836208+0000","last_became_active":"2024-04-26T01:41:03.301580+0000","last_became_peered":"2024-04-26T01:41:03.301580+0000","last_unstale":"2024-04-26T01:44:59.836208+0000","last_undegraded":"2024-04-26T01:44:59.836208+0000","last_fullsized":"2024-04-26T01:44:59.836208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:27.846775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145890+0000","last_change":"2024-04-26T01:41:03.285164+0000","last_active":"2024-04-26T01:45:00.145890+0000","last_peered":"2024-04-26T01:45:00.145890+0000","last_clean":"2024-04-26T01:45:00.145890+0000","last_became_active":"2024-04-26T01:41:03.283637+0000","last_became_peered":"2024-04-26T01:41:03.283637+0000","last_unstale":"2024-04-26T01:45:00.145890+0000","last_undegraded":"2024-04-26T01:45:00.145890+0000","last_fullsized":"2024-04-26T01:45:00.145890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:21:06.279054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145592+0000","last_change":"2024-04-26T01:41:03.291050+0000","last_active":"2024-04-26T01:45:00.145592+0000","last_peered":"2024-04-26T01:45:00.145592+0000","last_clean":"2024-04-26T01:45:00.145592+0000","last_became_active":"2024-04-26T01:41:03.290619+0000","last_became_peered":"2024-04-26T01:41:03.290619+0000","last_unstale":"2024-04-26T01:45:00.145592+0000","last_undegraded":"2024-04-26T01:45:00.145592+0000","last_fullsized":"2024-04-26T01:45:00.145592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:39:01.640824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145452+0000","last_change":"2024-04-26T01:41:03.284491+0000","last_active":"2024-04-26T01:45:00.145452+0000","last_peered":"2024-04-26T01:45:00.145452+0000","last_clean":"2024-04-26T01:45:00.145452+0000","last_became_active":"2024-04-26T01:41:03.284100+0000","last_became_peered":"2024-04-26T01:41:03.284100+0000","last_unstale":"2024-04-26T01:45:00.145452+0000","last_undegraded":"2024-04-26T01:45:00.145452+0000","last_fullsized":"2024-04-26T01:45:00.145452+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:47:59.262762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386985+0000","last_change":"2024-04-26T01:41:03.285716+0000","last_active":"2024-04-26T01:45:00.386985+0000","last_peered":"2024-04-26T01:45:00.386985+0000","last_clean":"2024-04-26T01:45:00.386985+0000","last_became_active":"2024-04-26T01:41:03.285498+0000","last_became_peered":"2024-04-26T01:41:03.285498+0000","last_unstale":"2024-04-26T01:45:00.386985+0000","last_undegraded":"2024-04-26T01:45:00.386985+0000","last_fullsized":"2024-04-26T01:45:00.386985+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:27.655241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971739+0000","last_change":"2024-04-26T01:41:03.284161+0000","last_active":"2024-04-26T01:44:59.971739+0000","last_peered":"2024-04-26T01:44:59.971739+0000","last_clean":"2024-04-26T01:44:59.971739+0000","last_became_active":"2024-04-26T01:41:03.283720+0000","last_became_peered":"2024-04-26T01:41:03.283720+0000","last_unstale":"2024-04-26T01:44:59.971739+0000","last_undegraded":"2024-04-26T01:44:59.971739+0000","last_fullsized":"2024-04-26T01:44:59.971739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:34:11.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146085+0000","last_change":"2024-04-26T01:41:03.293496+0000","last_active":"2024-04-26T01:45:00.146085+0000","last_peered":"2024-04-26T01:45:00.146085+0000","last_clean":"2024-04-26T01:45:00.146085+0000","last_became_active":"2024-04-26T01:41:03.293159+0000","last_became_peered":"2024-04-26T01:41:03.293159+0000","last_unstale":"2024-04-26T01:45:00.146085+0000","last_undegraded":"2024-04-26T01:45:00.146085+0000","last_fullsized":"2024-04-26T01:45:00.146085+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:51:59.158966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.972123+0000","last_change":"2024-04-26T01:40:59.225340+0000","last_active":"2024-04-26T01:44:59.972123+0000","last_peered":"2024-04-26T01:44:59.972123+0000","last_clean":"2024-04-26T01:44:59.972123+0000","last_became_active":"2024-04-26T01:40:59.223210+0000","last_became_peered":"2024-04-26T01:40:59.223210+0000","last_unstale":"2024-04-26T01:44:59.972123+0000","last_undegraded":"2024-04-26T01:44:59.972123+0000","last_fullsized":"2024-04-26T01:44:59.972123+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:57.037836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943401+0000","last_change":"2024-04-26T01:41:03.274373+0000","last_active":"2024-04-26T01:44:59.943401+0000","last_peered":"2024-04-26T01:44:59.943401+0000","last_clean":"2024-04-26T01:44:59.943401+0000","last_became_active":"2024-04-26T01:41:03.274030+0000","last_became_peered":"2024-04-26T01:41:03.274030+0000","last_unstale":"2024-04-26T01:44:59.943401+0000","last_undegraded":"2024-04-26T01:44:59.943401+0000","last_fullsized":"2024-04-26T01:44:59.943401+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:30.670763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943600+0000","last_change":"2024-04-26T01:40:59.276101+0000","last_active":"2024-04-26T01:44:59.943600+0000","last_peered":"2024-04-26T01:44:59.943600+0000","last_clean":"2024-04-26T01:44:59.943600+0000","last_became_active":"2024-04-26T01:40:59.275248+0000","last_became_peered":"2024-04-26T01:40:59.275248+0000","last_unstale":"2024-04-26T01:44:59.943600+0000","last_undegraded":"2024-04-26T01:44:59.943600+0000","last_fullsized":"2024-04-26T01:44:59.943600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:19.333491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146367+0000","last_change":"2024-04-26T01:41:03.292532+0000","last_active":"2024-04-26T01:45:00.146367+0000","last_peered":"2024-04-26T01:45:00.146367+0000","last_clean":"2024-04-26T01:45:00.146367+0000","last_became_active":"2024-04-26T01:41:03.292052+0000","last_became_peered":"2024-04-26T01:41:03.292052+0000","last_unstale":"2024-04-26T01:45:00.146367+0000","last_undegraded":"2024-04-26T01:45:00.146367+0000","last_fullsized":"2024-04-26T01:45:00.146367+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:36.871756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386813+0000","last_change":"2024-04-26T01:41:47.505878+0000","last_active":"2024-04-26T01:45:00.386813+0000","last_peered":"2024-04-26T01:45:00.386813+0000","last_clean":"2024-04-26T01:45:00.386813+0000","last_became_active":"2024-04-26T01:41:47.500067+0000","last_became_peered":"2024-04-26T01:41:47.500067+0000","last_unstale":"2024-04-26T01:45:00.386813+0000","last_undegraded":"2024-04-26T01:45:00.386813+0000","last_fullsized":"2024-04-26T01:45:00.386813+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:51:16.183619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145523+0000","last_change":"2024-04-26T01:41:03.297648+0000","last_active":"2024-04-26T01:45:00.145523+0000","last_peered":"2024-04-26T01:45:00.145523+0000","last_clean":"2024-04-26T01:45:00.145523+0000","last_became_active":"2024-04-26T01:41:03.297239+0000","last_became_peered":"2024-04-26T01:41:03.297239+0000","last_unstale":"2024-04-26T01:45:00.145523+0000","last_undegraded":"2024-04-26T01:45:00.145523+0000","last_fullsized":"2024-04-26T01:45:00.145523+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:56:45.096467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830007+0000","last_change":"2024-04-26T01:40:59.223006+0000","last_active":"2024-04-26T01:44:59.830007+0000","last_peered":"2024-04-26T01:44:59.830007+0000","last_clean":"2024-04-26T01:44:59.830007+0000","last_became_active":"2024-04-26T01:40:59.222023+0000","last_became_peered":"2024-04-26T01:40:59.222023+0000","last_unstale":"2024-04-26T01:44:59.830007+0000","last_undegraded":"2024-04-26T01:44:59.830007+0000","last_fullsized":"2024-04-26T01:44:59.830007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:51.834985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980895+0000","last_change":"2024-04-26T01:41:03.281384+0000","last_active":"2024-04-26T01:44:59.980895+0000","last_peered":"2024-04-26T01:44:59.980895+0000","last_clean":"2024-04-26T01:44:59.980895+0000","last_became_active":"2024-04-26T01:41:03.281097+0000","last_became_peered":"2024-04-26T01:41:03.281097+0000","last_unstale":"2024-04-26T01:44:59.980895+0000","last_undegraded":"2024-04-26T01:44:59.980895+0000","last_fullsized":"2024-04-26T01:44:59.980895+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:37:07.495483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"95'7","reported_seq":263,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:09.712707+0000","last_change":"2024-04-26T01:40:59.224953+0000","last_active":"2024-04-26T01:45:09.712707+0000","last_peered":"2024-04-26T01:45:09.712707+0000","last_clean":"2024-04-26T01:45:09.712707+0000","last_became_active":"2024-04-26T01:40:59.224563+0000","last_became_peered":"2024-04-26T01:40:59.224563+0000","last_unstale":"2024-04-26T01:45:09.712707+0000","last_undegraded":"2024-04-26T01:45:09.712707+0000","last_fullsized":"2024-04-26T01:45:09.712707+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:13.649325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":227,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829791+0000","last_change":"2024-04-26T01:41:03.288896+0000","last_active":"2024-04-26T01:44:59.829791+0000","last_peered":"2024-04-26T01:44:59.829791+0000","last_clean":"2024-04-26T01:44:59.829791+0000","last_became_active":"2024-04-26T01:41:03.288518+0000","last_became_peered":"2024-04-26T01:41:03.288518+0000","last_unstale":"2024-04-26T01:44:59.829791+0000","last_undegraded":"2024-04-26T01:44:59.829791+0000","last_fullsized":"2024-04-26T01:44:59.829791+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:47.060131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983749+0000","last_change":"2024-04-26T01:40:59.228834+0000","last_active":"2024-04-26T01:44:59.983749+0000","last_peered":"2024-04-26T01:44:59.983749+0000","last_clean":"2024-04-26T01:44:59.983749+0000","last_became_active":"2024-04-26T01:40:59.228445+0000","last_became_peered":"2024-04-26T01:40:59.228445+0000","last_unstale":"2024-04-26T01:44:59.983749+0000","last_undegraded":"2024-04-26T01:44:59.983749+0000","last_fullsized":"2024-04-26T01:44:59.983749+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:35:06.841549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980762+0000","last_change":"2024-04-26T01:41:03.292932+0000","last_active":"2024-04-26T01:44:59.980762+0000","last_peered":"2024-04-26T01:44:59.980762+0000","last_clean":"2024-04-26T01:44:59.980762+0000","last_became_active":"2024-04-26T01:41:03.292568+0000","last_became_peered":"2024-04-26T01:41:03.292568+0000","last_unstale":"2024-04-26T01:44:59.980762+0000","last_undegraded":"2024-04-26T01:44:59.980762+0000","last_fullsized":"2024-04-26T01:44:59.980762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:03.298439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836546+0000","last_change":"2024-04-26T01:40:59.227702+0000","last_active":"2024-04-26T01:44:59.836546+0000","last_peered":"2024-04-26T01:44:59.836546+0000","last_clean":"2024-04-26T01:44:59.836546+0000","last_became_active":"2024-04-26T01:40:59.227043+0000","last_became_peered":"2024-04-26T01:40:59.227043+0000","last_unstale":"2024-04-26T01:44:59.836546+0000","last_undegraded":"2024-04-26T01:44:59.836546+0000","last_fullsized":"2024-04-26T01:44:59.836546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:58:25.035464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942696+0000","last_change":"2024-04-26T01:41:03.279780+0000","last_active":"2024-04-26T01:44:59.942696+0000","last_peered":"2024-04-26T01:44:59.942696+0000","last_clean":"2024-04-26T01:44:59.942696+0000","last_became_active":"2024-04-26T01:41:03.271902+0000","last_became_peered":"2024-04-26T01:41:03.271902+0000","last_unstale":"2024-04-26T01:44:59.942696+0000","last_undegraded":"2024-04-26T01:44:59.942696+0000","last_fullsized":"2024-04-26T01:44:59.942696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:43:12.036763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980603+0000","last_change":"2024-04-26T01:40:59.225049+0000","last_active":"2024-04-26T01:44:59.980603+0000","last_peered":"2024-04-26T01:44:59.980603+0000","last_clean":"2024-04-26T01:44:59.980603+0000","last_became_active":"2024-04-26T01:40:59.224658+0000","last_became_peered":"2024-04-26T01:40:59.224658+0000","last_unstale":"2024-04-26T01:44:59.980603+0000","last_undegraded":"2024-04-26T01:44:59.980603+0000","last_fullsized":"2024-04-26T01:44:59.980603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:59:35.664760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971311+0000","last_change":"2024-04-26T01:41:47.510772+0000","last_active":"2024-04-26T01:44:59.971311+0000","last_peered":"2024-04-26T01:44:59.971311+0000","last_clean":"2024-04-26T01:44:59.971311+0000","last_became_active":"2024-04-26T01:41:47.510422+0000","last_became_peered":"2024-04-26T01:41:47.510422+0000","last_unstale":"2024-04-26T01:44:59.971311+0000","last_undegraded":"2024-04-26T01:44:59.971311+0000","last_fullsized":"2024-04-26T01:44:59.971311+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:06:44.088015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"95'4","reported_seq":257,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:09.712255+0000","last_change":"2024-04-26T01:40:59.224187+0000","last_active":"2024-04-26T01:45:09.712255+0000","last_peered":"2024-04-26T01:45:09.712255+0000","last_clean":"2024-04-26T01:45:09.712255+0000","last_became_active":"2024-04-26T01:40:59.223822+0000","last_became_peered":"2024-04-26T01:40:59.223822+0000","last_unstale":"2024-04-26T01:45:09.712255+0000","last_undegraded":"2024-04-26T01:45:09.712255+0000","last_fullsized":"2024-04-26T01:45:09.712255+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:06:09.270415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145864+0000","last_change":"2024-04-26T01:41:03.279135+0000","last_active":"2024-04-26T01:45:00.145864+0000","last_peered":"2024-04-26T01:45:00.145864+0000","last_clean":"2024-04-26T01:45:00.145864+0000","last_became_active":"2024-04-26T01:41:03.278685+0000","last_became_peered":"2024-04-26T01:41:03.278685+0000","last_unstale":"2024-04-26T01:45:00.145864+0000","last_undegraded":"2024-04-26T01:45:00.145864+0000","last_fullsized":"2024-04-26T01:45:00.145864+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:48:36.817034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832945+0000","last_change":"2024-04-26T01:40:59.225064+0000","last_active":"2024-04-26T01:44:59.832945+0000","last_peered":"2024-04-26T01:44:59.832945+0000","last_clean":"2024-04-26T01:44:59.832945+0000","last_became_active":"2024-04-26T01:40:59.224520+0000","last_became_peered":"2024-04-26T01:40:59.224520+0000","last_unstale":"2024-04-26T01:44:59.832945+0000","last_undegraded":"2024-04-26T01:44:59.832945+0000","last_fullsized":"2024-04-26T01:44:59.832945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:09:55.123854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146547+0000","last_change":"2024-04-26T01:41:47.505535+0000","last_active":"2024-04-26T01:45:00.146547+0000","last_peered":"2024-04-26T01:45:00.146547+0000","last_clean":"2024-04-26T01:45:00.146547+0000","last_became_active":"2024-04-26T01:41:47.505107+0000","last_became_peered":"2024-04-26T01:41:47.505107+0000","last_unstale":"2024-04-26T01:45:00.146547+0000","last_undegraded":"2024-04-26T01:45:00.146547+0000","last_fullsized":"2024-04-26T01:45:00.146547+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:43:13.425004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.981042+0000","last_change":"2024-04-26T01:40:59.271897+0000","last_active":"2024-04-26T01:44:59.981042+0000","last_peered":"2024-04-26T01:44:59.981042+0000","last_clean":"2024-04-26T01:44:59.981042+0000","last_became_active":"2024-04-26T01:40:59.271560+0000","last_became_peered":"2024-04-26T01:40:59.271560+0000","last_unstale":"2024-04-26T01:44:59.981042+0000","last_undegraded":"2024-04-26T01:44:59.981042+0000","last_fullsized":"2024-04-26T01:44:59.981042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:50:23.573627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145594+0000","last_change":"2024-04-26T01:41:03.275611+0000","last_active":"2024-04-26T01:45:00.145594+0000","last_peered":"2024-04-26T01:45:00.145594+0000","last_clean":"2024-04-26T01:45:00.145594+0000","last_became_active":"2024-04-26T01:41:03.274097+0000","last_became_peered":"2024-04-26T01:41:03.274097+0000","last_unstale":"2024-04-26T01:45:00.145594+0000","last_undegraded":"2024-04-26T01:45:00.145594+0000","last_fullsized":"2024-04-26T01:45:00.145594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:24:35.219663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"95'39","reported_seq":528,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:10.713044+0000","last_change":"2024-04-26T01:40:59.223740+0000","last_active":"2024-04-26T01:45:10.713044+0000","last_peered":"2024-04-26T01:45:10.713044+0000","last_clean":"2024-04-26T01:45:10.713044+0000","last_became_active":"2024-04-26T01:40:59.223377+0000","last_became_peered":"2024-04-26T01:40:59.223377+0000","last_unstale":"2024-04-26T01:45:10.713044+0000","last_undegraded":"2024-04-26T01:45:10.713044+0000","last_fullsized":"2024-04-26T01:45:10.713044+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:30.240798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71438,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":460,"num_read_kb":300,"num_write":40,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829990+0000","last_change":"2024-04-26T01:41:03.273307+0000","last_active":"2024-04-26T01:44:59.829990+0000","last_peered":"2024-04-26T01:44:59.829990+0000","last_clean":"2024-04-26T01:44:59.829990+0000","last_became_active":"2024-04-26T01:41:03.272968+0000","last_became_peered":"2024-04-26T01:41:03.272968+0000","last_unstale":"2024-04-26T01:44:59.829990+0000","last_undegraded":"2024-04-26T01:44:59.829990+0000","last_fullsized":"2024-04-26T01:44:59.829990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:04:33.427316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145896+0000","last_change":"2024-04-26T01:40:59.231455+0000","last_active":"2024-04-26T01:45:00.145896+0000","last_peered":"2024-04-26T01:45:00.145896+0000","last_clean":"2024-04-26T01:45:00.145896+0000","last_became_active":"2024-04-26T01:40:59.231058+0000","last_became_peered":"2024-04-26T01:40:59.231058+0000","last_unstale":"2024-04-26T01:45:00.145896+0000","last_undegraded":"2024-04-26T01:45:00.145896+0000","last_fullsized":"2024-04-26T01:45:00.145896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:41:50.100469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836995+0000","last_change":"2024-04-26T01:41:03.291718+0000","last_active":"2024-04-26T01:44:59.836995+0000","last_peered":"2024-04-26T01:44:59.836995+0000","last_clean":"2024-04-26T01:44:59.836995+0000","last_became_active":"2024-04-26T01:41:03.291329+0000","last_became_peered":"2024-04-26T01:41:03.291329+0000","last_unstale":"2024-04-26T01:44:59.836995+0000","last_undegraded":"2024-04-26T01:44:59.836995+0000","last_fullsized":"2024-04-26T01:44:59.836995+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:47:22.264790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980687+0000","last_change":"2024-04-26T01:40:59.231460+0000","last_active":"2024-04-26T01:44:59.980687+0000","last_peered":"2024-04-26T01:44:59.980687+0000","last_clean":"2024-04-26T01:44:59.980687+0000","last_became_active":"2024-04-26T01:40:59.231074+0000","last_became_peered":"2024-04-26T01:40:59.231074+0000","last_unstale":"2024-04-26T01:44:59.980687+0000","last_undegraded":"2024-04-26T01:44:59.980687+0000","last_fullsized":"2024-04-26T01:44:59.980687+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:00:36.705088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832836+0000","last_change":"2024-04-26T01:41:03.299202+0000","last_active":"2024-04-26T01:44:59.832836+0000","last_peered":"2024-04-26T01:44:59.832836+0000","last_clean":"2024-04-26T01:44:59.832836+0000","last_became_active":"2024-04-26T01:41:03.298854+0000","last_became_peered":"2024-04-26T01:41:03.298854+0000","last_unstale":"2024-04-26T01:44:59.832836+0000","last_undegraded":"2024-04-26T01:44:59.832836+0000","last_fullsized":"2024-04-26T01:44:59.832836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:51:59.615202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"95'7","reported_seq":264,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:08.962917+0000","last_change":"2024-04-26T01:40:59.222201+0000","last_active":"2024-04-26T01:45:08.962917+0000","last_peered":"2024-04-26T01:45:08.962917+0000","last_clean":"2024-04-26T01:45:08.962917+0000","last_became_active":"2024-04-26T01:40:59.221805+0000","last_became_peered":"2024-04-26T01:40:59.221805+0000","last_unstale":"2024-04-26T01:45:08.962917+0000","last_undegraded":"2024-04-26T01:45:08.962917+0000","last_fullsized":"2024-04-26T01:45:08.962917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:34:32.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832336+0000","last_change":"2024-04-26T01:41:03.278436+0000","last_active":"2024-04-26T01:44:59.832336+0000","last_peered":"2024-04-26T01:44:59.832336+0000","last_clean":"2024-04-26T01:44:59.832336+0000","last_became_active":"2024-04-26T01:41:03.278108+0000","last_became_peered":"2024-04-26T01:41:03.278108+0000","last_unstale":"2024-04-26T01:44:59.832336+0000","last_undegraded":"2024-04-26T01:44:59.832336+0000","last_fullsized":"2024-04-26T01:44:59.832336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:56:30.798023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829747+0000","last_change":"2024-04-26T01:40:59.273427+0000","last_active":"2024-04-26T01:44:59.829747+0000","last_peered":"2024-04-26T01:44:59.829747+0000","last_clean":"2024-04-26T01:44:59.829747+0000","last_became_active":"2024-04-26T01:40:59.272924+0000","last_became_peered":"2024-04-26T01:40:59.272924+0000","last_unstale":"2024-04-26T01:44:59.829747+0000","last_undegraded":"2024-04-26T01:44:59.829747+0000","last_fullsized":"2024-04-26T01:44:59.829747+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:15.227538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971521+0000","last_change":"2024-04-26T01:41:47.510428+0000","last_active":"2024-04-26T01:44:59.971521+0000","last_peered":"2024-04-26T01:44:59.971521+0000","last_clean":"2024-04-26T01:44:59.971521+0000","last_became_active":"2024-04-26T01:41:47.510048+0000","last_became_peered":"2024-04-26T01:41:47.510048+0000","last_unstale":"2024-04-26T01:44:59.971521+0000","last_undegraded":"2024-04-26T01:44:59.971521+0000","last_fullsized":"2024-04-26T01:44:59.971521+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:55:58.929900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"95'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830365+0000","last_change":"2024-04-26T01:40:59.223665+0000","last_active":"2024-04-26T01:44:59.830365+0000","last_peered":"2024-04-26T01:44:59.830365+0000","last_clean":"2024-04-26T01:44:59.830365+0000","last_became_active":"2024-04-26T01:40:59.223006+0000","last_became_peered":"2024-04-26T01:40:59.223006+0000","last_unstale":"2024-04-26T01:44:59.830365+0000","last_undegraded":"2024-04-26T01:44:59.830365+0000","last_fullsized":"2024-04-26T01:44:59.830365+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:43:07.791467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386745+0000","last_change":"2024-04-26T01:41:03.275524+0000","last_active":"2024-04-26T01:45:00.386745+0000","last_peered":"2024-04-26T01:45:00.386745+0000","last_clean":"2024-04-26T01:45:00.386745+0000","last_became_active":"2024-04-26T01:41:03.275181+0000","last_became_peered":"2024-04-26T01:41:03.275181+0000","last_unstale":"2024-04-26T01:45:00.386745+0000","last_undegraded":"2024-04-26T01:45:00.386745+0000","last_fullsized":"2024-04-26T01:45:00.386745+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:49:42.855711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.971630+0000","last_change":"2024-04-26T01:41:47.508155+0000","last_active":"2024-04-26T01:44:59.971630+0000","last_peered":"2024-04-26T01:44:59.971630+0000","last_clean":"2024-04-26T01:44:59.971630+0000","last_became_active":"2024-04-26T01:41:47.506668+0000","last_became_peered":"2024-04-26T01:41:47.506668+0000","last_unstale":"2024-04-26T01:44:59.971630+0000","last_undegraded":"2024-04-26T01:44:59.971630+0000","last_fullsized":"2024-04-26T01:44:59.971630+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:12:40.124270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983891+0000","last_change":"2024-04-26T01:41:03.293550+0000","last_active":"2024-04-26T01:44:59.983891+0000","last_peered":"2024-04-26T01:44:59.983891+0000","last_clean":"2024-04-26T01:44:59.983891+0000","last_became_active":"2024-04-26T01:41:03.293068+0000","last_became_peered":"2024-04-26T01:41:03.293068+0000","last_unstale":"2024-04-26T01:44:59.983891+0000","last_undegraded":"2024-04-26T01:44:59.983891+0000","last_fullsized":"2024-04-26T01:44:59.983891+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:30.760010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832756+0000","last_change":"2024-04-26T01:40:59.228632+0000","last_active":"2024-04-26T01:44:59.832756+0000","last_peered":"2024-04-26T01:44:59.832756+0000","last_clean":"2024-04-26T01:44:59.832756+0000","last_became_active":"2024-04-26T01:40:59.228284+0000","last_became_peered":"2024-04-26T01:40:59.228284+0000","last_unstale":"2024-04-26T01:44:59.832756+0000","last_undegraded":"2024-04-26T01:44:59.832756+0000","last_fullsized":"2024-04-26T01:44:59.832756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:09:14.820993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.983849+0000","last_change":"2024-04-26T01:41:03.289883+0000","last_active":"2024-04-26T01:44:59.983849+0000","last_peered":"2024-04-26T01:44:59.983849+0000","last_clean":"2024-04-26T01:44:59.983849+0000","last_became_active":"2024-04-26T01:41:03.289427+0000","last_became_peered":"2024-04-26T01:41:03.289427+0000","last_unstale":"2024-04-26T01:44:59.983849+0000","last_undegraded":"2024-04-26T01:44:59.983849+0000","last_fullsized":"2024-04-26T01:44:59.983849+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:46:21.083862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980952+0000","last_change":"2024-04-26T01:41:03.280016+0000","last_active":"2024-04-26T01:44:59.980952+0000","last_peered":"2024-04-26T01:44:59.980952+0000","last_clean":"2024-04-26T01:44:59.980952+0000","last_became_active":"2024-04-26T01:41:03.278805+0000","last_became_peered":"2024-04-26T01:41:03.278805+0000","last_unstale":"2024-04-26T01:44:59.980952+0000","last_undegraded":"2024-04-26T01:44:59.980952+0000","last_fullsized":"2024-04-26T01:44:59.980952+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:33:12.393968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":259,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:08.961947+0000","last_change":"2024-04-26T01:40:59.274719+0000","last_active":"2024-04-26T01:45:08.961947+0000","last_peered":"2024-04-26T01:45:08.961947+0000","last_clean":"2024-04-26T01:45:08.961947+0000","last_became_active":"2024-04-26T01:40:59.274185+0000","last_became_peered":"2024-04-26T01:40:59.274185+0000","last_unstale":"2024-04-26T01:45:08.961947+0000","last_undegraded":"2024-04-26T01:45:08.961947+0000","last_fullsized":"2024-04-26T01:45:08.961947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:02.978427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":223,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943097+0000","last_change":"2024-04-26T01:41:03.275590+0000","last_active":"2024-04-26T01:44:59.943097+0000","last_peered":"2024-04-26T01:44:59.943097+0000","last_clean":"2024-04-26T01:44:59.943097+0000","last_became_active":"2024-04-26T01:41:03.274804+0000","last_became_peered":"2024-04-26T01:41:03.274804+0000","last_unstale":"2024-04-26T01:44:59.943097+0000","last_undegraded":"2024-04-26T01:44:59.943097+0000","last_fullsized":"2024-04-26T01:44:59.943097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:36:56.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942650+0000","last_change":"2024-04-26T01:40:59.226114+0000","last_active":"2024-04-26T01:44:59.942650+0000","last_peered":"2024-04-26T01:44:59.942650+0000","last_clean":"2024-04-26T01:44:59.942650+0000","last_became_active":"2024-04-26T01:40:59.225741+0000","last_became_peered":"2024-04-26T01:40:59.225741+0000","last_unstale":"2024-04-26T01:44:59.942650+0000","last_undegraded":"2024-04-26T01:44:59.942650+0000","last_fullsized":"2024-04-26T01:44:59.942650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:49:56.281243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"95'148","reported_seq":408,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.003957+0000","last_change":"2024-04-26T01:39:25.657530+0000","last_active":"2024-04-26T01:45:02.003957+0000","last_peered":"2024-04-26T01:45:02.003957+0000","last_clean":"2024-04-26T01:45:02.003957+0000","last_became_active":"2024-04-26T01:39:25.656930+0000","last_became_peered":"2024-04-26T01:39:25.656930+0000","last_unstale":"2024-04-26T01:45:02.003957+0000","last_undegraded":"2024-04-26T01:45:02.003957+0000","last_fullsized":"2024-04-26T01:45:02.003957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:35:34.121702+0000","last_clean_scrub_stamp":"2024-04-26T01:35:34.121702+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:38.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836149+0000","last_change":"2024-04-26T01:41:03.281142+0000","last_active":"2024-04-26T01:44:59.836149+0000","last_peered":"2024-04-26T01:44:59.836149+0000","last_clean":"2024-04-26T01:44:59.836149+0000","last_became_active":"2024-04-26T01:41:03.280721+0000","last_became_peered":"2024-04-26T01:41:03.280721+0000","last_unstale":"2024-04-26T01:44:59.836149+0000","last_undegraded":"2024-04-26T01:44:59.836149+0000","last_fullsized":"2024-04-26T01:44:59.836149+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:47.725267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832637+0000","last_change":"2024-04-26T01:40:59.222528+0000","last_active":"2024-04-26T01:44:59.832637+0000","last_peered":"2024-04-26T01:44:59.832637+0000","last_clean":"2024-04-26T01:44:59.832637+0000","last_became_active":"2024-04-26T01:40:59.222134+0000","last_became_peered":"2024-04-26T01:40:59.222134+0000","last_unstale":"2024-04-26T01:44:59.832637+0000","last_undegraded":"2024-04-26T01:44:59.832637+0000","last_fullsized":"2024-04-26T01:44:59.832637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:24:10.057841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830239+0000","last_change":"2024-04-26T01:40:59.273323+0000","last_active":"2024-04-26T01:44:59.830239+0000","last_peered":"2024-04-26T01:44:59.830239+0000","last_clean":"2024-04-26T01:44:59.830239+0000","last_became_active":"2024-04-26T01:40:59.272931+0000","last_became_peered":"2024-04-26T01:40:59.272931+0000","last_unstale":"2024-04-26T01:44:59.830239+0000","last_undegraded":"2024-04-26T01:44:59.830239+0000","last_fullsized":"2024-04-26T01:44:59.830239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:10.780245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836146+0000","last_change":"2024-04-26T01:41:03.294066+0000","last_active":"2024-04-26T01:44:59.836146+0000","last_peered":"2024-04-26T01:44:59.836146+0000","last_clean":"2024-04-26T01:44:59.836146+0000","last_became_active":"2024-04-26T01:41:03.293676+0000","last_became_peered":"2024-04-26T01:41:03.293676+0000","last_unstale":"2024-04-26T01:44:59.836146+0000","last_undegraded":"2024-04-26T01:44:59.836146+0000","last_fullsized":"2024-04-26T01:44:59.836146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:55:46.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"95'102","reported_seq":223,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.483945+0000","last_change":"2024-04-26T01:40:59.230442+0000","last_active":"2024-04-26T01:45:02.483945+0000","last_peered":"2024-04-26T01:45:02.483945+0000","last_clean":"2024-04-26T01:45:02.483945+0000","last_became_active":"2024-04-26T01:40:59.229797+0000","last_became_peered":"2024-04-26T01:40:59.229797+0000","last_unstale":"2024-04-26T01:45:02.483945+0000","last_undegraded":"2024-04-26T01:45:02.483945+0000","last_fullsized":"2024-04-26T01:45:02.483945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:52.043967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943583+0000","last_change":"2024-04-26T01:41:03.292697+0000","last_active":"2024-04-26T01:44:59.943583+0000","last_peered":"2024-04-26T01:44:59.943583+0000","last_clean":"2024-04-26T01:44:59.943583+0000","last_became_active":"2024-04-26T01:41:03.291453+0000","last_became_peered":"2024-04-26T01:41:03.291453+0000","last_unstale":"2024-04-26T01:44:59.943583+0000","last_undegraded":"2024-04-26T01:44:59.943583+0000","last_fullsized":"2024-04-26T01:44:59.943583+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:38:13.139576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942746+0000","last_change":"2024-04-26T01:40:59.646754+0000","last_active":"2024-04-26T01:44:59.942746+0000","last_peered":"2024-04-26T01:44:59.942746+0000","last_clean":"2024-04-26T01:44:59.942746+0000","last_became_active":"2024-04-26T01:40:59.646416+0000","last_became_peered":"2024-04-26T01:40:59.646416+0000","last_unstale":"2024-04-26T01:44:59.942746+0000","last_undegraded":"2024-04-26T01:44:59.942746+0000","last_fullsized":"2024-04-26T01:44:59.942746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:21:30.806586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146000+0000","last_change":"2024-04-26T01:41:03.296105+0000","last_active":"2024-04-26T01:45:00.146000+0000","last_peered":"2024-04-26T01:45:00.146000+0000","last_clean":"2024-04-26T01:45:00.146000+0000","last_became_active":"2024-04-26T01:41:03.295575+0000","last_became_peered":"2024-04-26T01:41:03.295575+0000","last_unstale":"2024-04-26T01:45:00.146000+0000","last_undegraded":"2024-04-26T01:45:00.146000+0000","last_fullsized":"2024-04-26T01:45:00.146000+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.105635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145455+0000","last_change":"2024-04-26T01:40:59.273402+0000","last_active":"2024-04-26T01:45:00.145455+0000","last_peered":"2024-04-26T01:45:00.145455+0000","last_clean":"2024-04-26T01:45:00.145455+0000","last_became_active":"2024-04-26T01:40:59.273081+0000","last_became_peered":"2024-04-26T01:40:59.273081+0000","last_unstale":"2024-04-26T01:45:00.145455+0000","last_undegraded":"2024-04-26T01:45:00.145455+0000","last_fullsized":"2024-04-26T01:45:00.145455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:45.750384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.829725+0000","last_change":"2024-04-26T01:41:03.275806+0000","last_active":"2024-04-26T01:44:59.829725+0000","last_peered":"2024-04-26T01:44:59.829725+0000","last_clean":"2024-04-26T01:44:59.829725+0000","last_became_active":"2024-04-26T01:41:03.275478+0000","last_became_peered":"2024-04-26T01:41:03.275478+0000","last_unstale":"2024-04-26T01:44:59.829725+0000","last_undegraded":"2024-04-26T01:44:59.829725+0000","last_fullsized":"2024-04-26T01:44:59.829725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:41.912295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942730+0000","last_change":"2024-04-26T01:40:59.233795+0000","last_active":"2024-04-26T01:44:59.942730+0000","last_peered":"2024-04-26T01:44:59.942730+0000","last_clean":"2024-04-26T01:44:59.942730+0000","last_became_active":"2024-04-26T01:40:59.230806+0000","last_became_peered":"2024-04-26T01:40:59.230806+0000","last_unstale":"2024-04-26T01:44:59.942730+0000","last_undegraded":"2024-04-26T01:44:59.942730+0000","last_fullsized":"2024-04-26T01:44:59.942730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:16.044425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386613+0000","last_change":"2024-04-26T01:41:03.283585+0000","last_active":"2024-04-26T01:45:00.386613+0000","last_peered":"2024-04-26T01:45:00.386613+0000","last_clean":"2024-04-26T01:45:00.386613+0000","last_became_active":"2024-04-26T01:41:03.283216+0000","last_became_peered":"2024-04-26T01:41:03.283216+0000","last_unstale":"2024-04-26T01:45:00.386613+0000","last_undegraded":"2024-04-26T01:45:00.386613+0000","last_fullsized":"2024-04-26T01:45:00.386613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:50:44.395277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145777+0000","last_change":"2024-04-26T01:40:59.231727+0000","last_active":"2024-04-26T01:45:00.145777+0000","last_peered":"2024-04-26T01:45:00.145777+0000","last_clean":"2024-04-26T01:45:00.145777+0000","last_became_active":"2024-04-26T01:40:59.231365+0000","last_became_peered":"2024-04-26T01:40:59.231365+0000","last_unstale":"2024-04-26T01:45:00.145777+0000","last_undegraded":"2024-04-26T01:45:00.145777+0000","last_fullsized":"2024-04-26T01:45:00.145777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:10:30.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943095+0000","last_change":"2024-04-26T01:41:03.296370+0000","last_active":"2024-04-26T01:44:59.943095+0000","last_peered":"2024-04-26T01:44:59.943095+0000","last_clean":"2024-04-26T01:44:59.943095+0000","last_became_active":"2024-04-26T01:41:03.296039+0000","last_became_peered":"2024-04-26T01:41:03.296039+0000","last_unstale":"2024-04-26T01:44:59.943095+0000","last_undegraded":"2024-04-26T01:44:59.943095+0000","last_fullsized":"2024-04-26T01:44:59.943095+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:28:42.006125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.830189+0000","last_change":"2024-04-26T01:40:59.222330+0000","last_active":"2024-04-26T01:44:59.830189+0000","last_peered":"2024-04-26T01:44:59.830189+0000","last_clean":"2024-04-26T01:44:59.830189+0000","last_became_active":"2024-04-26T01:40:59.221841+0000","last_became_peered":"2024-04-26T01:40:59.221841+0000","last_unstale":"2024-04-26T01:44:59.830189+0000","last_undegraded":"2024-04-26T01:44:59.830189+0000","last_fullsized":"2024-04-26T01:44:59.830189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:25.594164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832778+0000","last_change":"2024-04-26T01:41:03.293611+0000","last_active":"2024-04-26T01:44:59.832778+0000","last_peered":"2024-04-26T01:44:59.832778+0000","last_clean":"2024-04-26T01:44:59.832778+0000","last_became_active":"2024-04-26T01:41:03.293236+0000","last_became_peered":"2024-04-26T01:41:03.293236+0000","last_unstale":"2024-04-26T01:44:59.832778+0000","last_undegraded":"2024-04-26T01:44:59.832778+0000","last_fullsized":"2024-04-26T01:44:59.832778+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:28:14.427803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942645+0000","last_change":"2024-04-26T01:41:47.513085+0000","last_active":"2024-04-26T01:44:59.942645+0000","last_peered":"2024-04-26T01:44:59.942645+0000","last_clean":"2024-04-26T01:44:59.942645+0000","last_became_active":"2024-04-26T01:41:47.506336+0000","last_became_peered":"2024-04-26T01:41:47.506336+0000","last_unstale":"2024-04-26T01:44:59.942645+0000","last_undegraded":"2024-04-26T01:44:59.942645+0000","last_fullsized":"2024-04-26T01:44:59.942645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:34.684026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833327+0000","last_change":"2024-04-26T01:41:03.273635+0000","last_active":"2024-04-26T01:44:59.833327+0000","last_peered":"2024-04-26T01:44:59.833327+0000","last_clean":"2024-04-26T01:44:59.833327+0000","last_became_active":"2024-04-26T01:41:03.273247+0000","last_became_peered":"2024-04-26T01:41:03.273247+0000","last_unstale":"2024-04-26T01:44:59.833327+0000","last_undegraded":"2024-04-26T01:44:59.833327+0000","last_fullsized":"2024-04-26T01:44:59.833327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:18.798825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"95'65","reported_seq":125,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:02.017193+0000","last_change":"2024-04-26T01:40:59.231690+0000","last_active":"2024-04-26T01:45:02.017193+0000","last_peered":"2024-04-26T01:45:02.017193+0000","last_clean":"2024-04-26T01:45:02.017193+0000","last_became_active":"2024-04-26T01:40:59.231127+0000","last_became_peered":"2024-04-26T01:40:59.231127+0000","last_unstale":"2024-04-26T01:45:02.017193+0000","last_undegraded":"2024-04-26T01:45:02.017193+0000","last_fullsized":"2024-04-26T01:45:02.017193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:24.852055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.832937+0000","last_change":"2024-04-26T01:41:03.290835+0000","last_active":"2024-04-26T01:44:59.832937+0000","last_peered":"2024-04-26T01:44:59.832937+0000","last_clean":"2024-04-26T01:44:59.832937+0000","last_became_active":"2024-04-26T01:41:03.290504+0000","last_became_peered":"2024-04-26T01:41:03.290504+0000","last_unstale":"2024-04-26T01:44:59.832937+0000","last_undegraded":"2024-04-26T01:44:59.832937+0000","last_fullsized":"2024-04-26T01:44:59.832937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:21:34.517800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836360+0000","last_change":"2024-04-26T01:40:59.234627+0000","last_active":"2024-04-26T01:44:59.836360+0000","last_peered":"2024-04-26T01:44:59.836360+0000","last_clean":"2024-04-26T01:44:59.836360+0000","last_became_active":"2024-04-26T01:40:59.233146+0000","last_became_peered":"2024-04-26T01:40:59.233146+0000","last_unstale":"2024-04-26T01:44:59.836360+0000","last_undegraded":"2024-04-26T01:44:59.836360+0000","last_fullsized":"2024-04-26T01:44:59.836360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:40:58.196588+0000","last_clean_scrub_stamp":"2024-04-26T01:40:58.196588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:40.352690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.833557+0000","last_change":"2024-04-26T01:41:03.277077+0000","last_active":"2024-04-26T01:44:59.833557+0000","last_peered":"2024-04-26T01:44:59.833557+0000","last_clean":"2024-04-26T01:44:59.833557+0000","last_became_active":"2024-04-26T01:41:03.276723+0000","last_became_peered":"2024-04-26T01:41:03.276723+0000","last_unstale":"2024-04-26T01:44:59.833557+0000","last_undegraded":"2024-04-26T01:44:59.833557+0000","last_fullsized":"2024-04-26T01:44:59.833557+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:21:57.403135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386742+0000","last_change":"2024-04-26T01:41:03.273931+0000","last_active":"2024-04-26T01:45:00.386742+0000","last_peered":"2024-04-26T01:45:00.386742+0000","last_clean":"2024-04-26T01:45:00.386742+0000","last_became_active":"2024-04-26T01:41:03.273594+0000","last_became_peered":"2024-04-26T01:41:03.273594+0000","last_unstale":"2024-04-26T01:45:00.386742+0000","last_undegraded":"2024-04-26T01:45:00.386742+0000","last_fullsized":"2024-04-26T01:45:00.386742+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:14.464587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145926+0000","last_change":"2024-04-26T01:41:03.275929+0000","last_active":"2024-04-26T01:45:00.145926+0000","last_peered":"2024-04-26T01:45:00.145926+0000","last_clean":"2024-04-26T01:45:00.145926+0000","last_became_active":"2024-04-26T01:41:03.275400+0000","last_became_peered":"2024-04-26T01:41:03.275400+0000","last_unstale":"2024-04-26T01:45:00.145926+0000","last_undegraded":"2024-04-26T01:45:00.145926+0000","last_fullsized":"2024-04-26T01:45:00.145926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:13.294261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145769+0000","last_change":"2024-04-26T01:41:03.288291+0000","last_active":"2024-04-26T01:45:00.145769+0000","last_peered":"2024-04-26T01:45:00.145769+0000","last_clean":"2024-04-26T01:45:00.145769+0000","last_became_active":"2024-04-26T01:41:03.288006+0000","last_became_peered":"2024-04-26T01:41:03.288006+0000","last_unstale":"2024-04-26T01:45:00.145769+0000","last_undegraded":"2024-04-26T01:45:00.145769+0000","last_fullsized":"2024-04-26T01:45:00.145769+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:54:41.008152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145984+0000","last_change":"2024-04-26T01:41:03.301934+0000","last_active":"2024-04-26T01:45:00.145984+0000","last_peered":"2024-04-26T01:45:00.145984+0000","last_clean":"2024-04-26T01:45:00.145984+0000","last_became_active":"2024-04-26T01:41:03.301577+0000","last_became_peered":"2024-04-26T01:41:03.301577+0000","last_unstale":"2024-04-26T01:45:00.145984+0000","last_undegraded":"2024-04-26T01:45:00.145984+0000","last_fullsized":"2024-04-26T01:45:00.145984+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:22:09.213585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.835967+0000","last_change":"2024-04-26T01:41:03.295309+0000","last_active":"2024-04-26T01:44:59.835967+0000","last_peered":"2024-04-26T01:44:59.835967+0000","last_clean":"2024-04-26T01:44:59.835967+0000","last_became_active":"2024-04-26T01:41:03.295024+0000","last_became_peered":"2024-04-26T01:41:03.295024+0000","last_unstale":"2024-04-26T01:44:59.835967+0000","last_undegraded":"2024-04-26T01:44:59.835967+0000","last_fullsized":"2024-04-26T01:44:59.835967+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:57:46.449194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386872+0000","last_change":"2024-04-26T01:41:03.277828+0000","last_active":"2024-04-26T01:45:00.386872+0000","last_peered":"2024-04-26T01:45:00.386872+0000","last_clean":"2024-04-26T01:45:00.386872+0000","last_became_active":"2024-04-26T01:41:03.277433+0000","last_became_peered":"2024-04-26T01:41:03.277433+0000","last_unstale":"2024-04-26T01:45:00.386872+0000","last_undegraded":"2024-04-26T01:45:00.386872+0000","last_fullsized":"2024-04-26T01:45:00.386872+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:47:01.317877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.146069+0000","last_change":"2024-04-26T01:41:03.274693+0000","last_active":"2024-04-26T01:45:00.146069+0000","last_peered":"2024-04-26T01:45:00.146069+0000","last_clean":"2024-04-26T01:45:00.146069+0000","last_became_active":"2024-04-26T01:41:03.274312+0000","last_became_peered":"2024-04-26T01:41:03.274312+0000","last_unstale":"2024-04-26T01:45:00.146069+0000","last_undegraded":"2024-04-26T01:45:00.146069+0000","last_fullsized":"2024-04-26T01:45:00.146069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:45:51.676241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.943455+0000","last_change":"2024-04-26T01:41:03.270594+0000","last_active":"2024-04-26T01:44:59.943455+0000","last_peered":"2024-04-26T01:44:59.943455+0000","last_clean":"2024-04-26T01:44:59.943455+0000","last_became_active":"2024-04-26T01:41:03.270252+0000","last_became_peered":"2024-04-26T01:41:03.270252+0000","last_unstale":"2024-04-26T01:44:59.943455+0000","last_undegraded":"2024-04-26T01:44:59.943455+0000","last_fullsized":"2024-04-26T01:44:59.943455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:54.212833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.980685+0000","last_change":"2024-04-26T01:41:03.292829+0000","last_active":"2024-04-26T01:44:59.980685+0000","last_peered":"2024-04-26T01:44:59.980685+0000","last_clean":"2024-04-26T01:44:59.980685+0000","last_became_active":"2024-04-26T01:41:03.292579+0000","last_became_peered":"2024-04-26T01:41:03.292579+0000","last_unstale":"2024-04-26T01:44:59.980685+0000","last_undegraded":"2024-04-26T01:44:59.980685+0000","last_fullsized":"2024-04-26T01:44:59.980685+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:18:07.016055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.942739+0000","last_change":"2024-04-26T01:41:03.287865+0000","last_active":"2024-04-26T01:44:59.942739+0000","last_peered":"2024-04-26T01:44:59.942739+0000","last_clean":"2024-04-26T01:44:59.942739+0000","last_became_active":"2024-04-26T01:41:03.287503+0000","last_became_peered":"2024-04-26T01:41:03.287503+0000","last_unstale":"2024-04-26T01:44:59.942739+0000","last_undegraded":"2024-04-26T01:44:59.942739+0000","last_fullsized":"2024-04-26T01:44:59.942739+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:13:52.277269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.145556+0000","last_change":"2024-04-26T01:41:03.273102+0000","last_active":"2024-04-26T01:45:00.145556+0000","last_peered":"2024-04-26T01:45:00.145556+0000","last_clean":"2024-04-26T01:45:00.145556+0000","last_became_active":"2024-04-26T01:41:03.272773+0000","last_became_peered":"2024-04-26T01:41:03.272773+0000","last_unstale":"2024-04-26T01:45:00.145556+0000","last_undegraded":"2024-04-26T01:45:00.145556+0000","last_fullsized":"2024-04-26T01:45:00.145556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:23:55.881815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:44:59.836713+0000","last_change":"2024-04-26T01:41:03.300689+0000","last_active":"2024-04-26T01:44:59.836713+0000","last_peered":"2024-04-26T01:44:59.836713+0000","last_clean":"2024-04-26T01:44:59.836713+0000","last_became_active":"2024-04-26T01:41:03.298677+0000","last_became_peered":"2024-04-26T01:41:03.298677+0000","last_unstale":"2024-04-26T01:44:59.836713+0000","last_undegraded":"2024-04-26T01:44:59.836713+0000","last_fullsized":"2024-04-26T01:44:59.836713+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:21:06.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:45:00.386719+0000","last_change":"2024-04-26T01:41:47.502054+0000","last_active":"2024-04-26T01:45:00.386719+0000","last_peered":"2024-04-26T01:45:00.386719+0000","last_clean":"2024-04-26T01:45:00.386719+0000","last_became_active":"2024-04-26T01:41:47.501664+0000","last_became_peered":"2024-04-26T01:41:47.501664+0000","last_unstale":"2024-04-26T01:45:00.386719+0000","last_undegraded":"2024-04-26T01:45:00.386719+0000","last_fullsized":"2024-04-26T01:45:00.386719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:41:02.231097+0000","last_clean_scrub_stamp":"2024-04-26T01:41:02.231097+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:09.144911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":798592,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1720,"num_read_kb":1077,"num_write":425,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2629632,"data_stored":2395776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252931,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":506662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514569,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33204,"kb_used_data":2028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93711948,"statfs":{"total":95995035648,"available":95961034752,"internally_reserved":0,"allocated":2076672,"data_stored":1499127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710799,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36752,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708400,"statfs":{"total":95995035648,"available":95957401600,"internally_reserved":0,"allocated":1445888,"data_stored":900114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972436,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28432,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716720,"statfs":{"total":95995035648,"available":95965921280,"internally_reserved":0,"allocated":1445888,"data_stored":900034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234072,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1100453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397598,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":506287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":577518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738495,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":507558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37196,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707956,"statfs":{"total":95995035648,"available":95956946944,"internally_reserved":0,"allocated":1781760,"data_stored":1228046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855467,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":508620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659236,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37400,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707752,"statfs":{"total":95995035648,"available":95956738048,"internally_reserved":0,"allocated":1990656,"data_stored":1427055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":72988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":402562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:45:14.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:15.085 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:15.532 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:15 smithi052 ceph-mon[27851]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:15.533 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:15 smithi052 ceph-mon[27851]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:15.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:15 smithi191 ceph-mon[32745]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:15.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:15 smithi191 ceph-mon[32745]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:15 smithi183 ceph-mon[32752]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T01:45:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:15 smithi183 ceph-mon[32752]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 -- 172.21.15.52:0/865843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c104200 msgr2=0x7f7d3c1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 --2- 172.21.15.52:0/865843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c104200 0x7f7d3c1045e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009900 tx=0x7f7d2c02a410 comp rx=0 tx=0).stop 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 -- 172.21.15.52:0/865843237 shutdown_connections 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 --2- 172.21.15.52:0/865843237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 --2- 172.21.15.52:0/865843237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104bb0 0x7f7d3c108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 --2- 172.21.15.52:0/865843237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c104200 0x7f7d3c1045e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 -- 172.21.15.52:0/865843237 >> 172.21.15.52:0/865843237 conn(0x7f7d3c0ffa60 msgr2=0x7f7d3c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 -- 172.21.15.52:0/865843237 shutdown_connections 2024-04-26T01:45:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.836+0000 7f7d3b200640 1 -- 172.21.15.52:0/865843237 wait complete. 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.837+0000 7f7d3b200640 1 Processor -- start 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.837+0000 7f7d3b200640 1 -- start start 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104200 0x7f7d3c1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c19a620 0x7f7d3c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c114280 con 0x7f7d3c1092d0 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c114100 con 0x7f7d3c19a620 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c114400 con 0x7f7d3c104200 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d31400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104200 0x7f7d3c1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d31400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40764/0 (socket says 172.21.15.52:40764) 2024-04-26T01:45:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d31400640 1 -- 172.21.15.52:0/3189659620 learned_addr learned my addr 172.21.15.52:0/3189659620 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.838+0000 7f7d3a800640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c19a620 0x7f7d3c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 -- 172.21.15.52:0/3189659620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104200 msgr2=0x7f7d3c1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104200 0x7f7d3c1a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 -- 172.21.15.52:0/3189659620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c19a620 msgr2=0x7f7d3c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c19a620 0x7f7d3c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 -- 172.21.15.52:0/3189659620 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c19b140 con 0x7f7d3c1092d0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d31400640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9f0 tx=0x7f7d2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.839+0000 7f7d33400640 1 -- 172.21.15.52:0/3189659620 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400ccc0 con 0x7f7d3c1092d0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.840+0000 7f7d33400640 1 -- 172.21.15.52:0/3189659620 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d240054d0 con 0x7f7d3c1092d0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.840+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c19b430 con 0x7f7d3c1092d0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.840+0000 7f7d33400640 1 -- 172.21.15.52:0/3189659620 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24010430 con 0x7f7d3c1092d0 2024-04-26T01:45:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.840+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c1a7680 con 0x7f7d3c1092d0 2024-04-26T01:45:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.843+0000 7f7d43dd4740 1 Processor -- start 2024-04-26T01:45:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.843+0000 7f7d43dd4740 1 -- start start 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.843+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e49b0 0x56112c5e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56112c54a900 con 0x56112c5e52d0 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56112c4f0ef0 con 0x56112c5efb90 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56112c5468c0 con 0x56112c5e49b0 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40776/0 (socket says 172.21.15.52:40776) 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 -- 172.21.15.52:0/140682191 learned_addr learned my addr 172.21.15.52:0/140682191 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d3a800640 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e49b0 0x56112c5e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 -- 172.21.15.52:0/140682191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e49b0 msgr2=0x56112c5e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e49b0 0x56112c5e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 -- 172.21.15.52:0/140682191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 msgr2=0x56112c5f1f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:45:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5f1f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.844+0000 7f7d39e00640 1 -- 172.21.15.52:0/140682191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112c25d210 con 0x56112c5e52d0 2024-04-26T01:45:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.845+0000 7f7d39e00640 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c000c00 tx=0x7f7d2c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b9af15424869e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.845+0000 7f7d33400640 1 -- 172.21.15.52:0/3189659620 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f7d240046c0 con 0x7f7d3c1092d0 2024-04-26T01:45:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.845+0000 7f7d2b400640 1 -- 172.21.15.52:0/140682191 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035040 con 0x56112c5e52d0 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.845+0000 7f7d2b400640 1 -- 172.21.15.52:0/140682191 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c0054f0 con 0x56112c5e52d0 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.845+0000 7f7d2b400640 1 -- 172.21.15.52:0/140682191 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0035f0 con 0x56112c5e52d0 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 -- 172.21.15.52:0/140682191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 msgr2=0x56112c5ef650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c000c00 tx=0x7f7d2c004040 comp rx=0 tx=0).stop 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 -- 172.21.15.52:0/140682191 shutdown_connections 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e52d0 0x56112c5ef650 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/140682191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e49b0 0x56112c5e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 -- 172.21.15.52:0/140682191 >> 172.21.15.52:0/140682191 conn(0x56112c5e0ec0 msgr2=0x56112c5e12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 -- 172.21.15.52:0/140682191 shutdown_connections 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.846+0000 7f7d43dd4740 1 -- 172.21.15.52:0/140682191 wait complete. 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.847+0000 7f7d33400640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 0x7f7d200a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.847+0000 7f7d33400640 1 -- 172.21.15.52:0/3189659620 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7d240d0cd0 con 0x7f7d3c1092d0 2024-04-26T01:45:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.847+0000 7f7d43dd4740 1 -- start start 2024-04-26T01:45:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.847+0000 7f7d3a800640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 0x7f7d200a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.847+0000 7f7d43dd4740 1 Processor -- start 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5ecb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5ed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56112c5f2760 con 0x56112c5e49b0 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56112c504310 con 0x56112c5efb90 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56112c5f28e0 con 0x56112c5e52d0 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d43dd4740 10 client.0 _ll_register_callbacks cb 0x56112c4d2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d3a800640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 0x7f7d200a6200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d34005fd0 tx=0x7f7d34009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5ecb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5ecb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40778/0 (socket says 172.21.15.52:40778) 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d39e00640 1 -- 172.21.15.52:0/981851665 learned_addr learned my addr 172.21.15.52:0/981851665 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d31400640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.848+0000 7f7d3a800640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5ed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 -- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 msgr2=0x56112c5ed5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5ed5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 -- 172.21.15.52:0/981851665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 msgr2=0x56112c5ecb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5ecb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 -- 172.21.15.52:0/981851665 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112c329620 con 0x56112c5e52d0 2024-04-26T01:45:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d31400640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d240ce330 tx=0x7f7d2409c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68617d1c0f353168 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.849+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d240a0e80 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7d240bd0f0 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d2c0054f0 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7d240a5740 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d43dd4740 10 client.25150 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56112c5fea40 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c0054f0 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.850+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7d24005120 con 0x56112c5e52d0 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 10 client.25150 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 10 client.25150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 20 client.25150 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c5fff00 0x56112c602310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 4 client.25150 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56112c55e8b0 con 0x56112c5fff00 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.851+0000 7f7d39e00640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c5fff00 0x56112c602310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.852+0000 7f7d39e00640 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c5fff00 0x56112c602310 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84301240cf68a9a server_cookie=3a52cb863a156781 in_seq=0 out_seq=0 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.852+0000 7f7d1fe00640 10 client.25150 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d1fe00640 1 -- 172.21.15.52:0/981851665 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56112c55e8b0 con 0x56112c5fff00 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d1fe00640 10 client.25150 handle_command_reply: tid=0 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 client.25150 shutdown 2024-04-26T01:45:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 msgr2=0x56112c5ed070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d240ce330 tx=0x7f7d2409c4f0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 shutdown_connections 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c5fff00 0x56112c602310 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5efb90 0x56112c5ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/981851665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5ecb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d1fe00640 0 client.25150 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 >> 172.21.15.52:0/981851665 conn(0x56112c5e0ec0 msgr2=0x56112c5e6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 shutdown_connections 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 1 -- 172.21.15.52:0/981851665 wait complete. 2024-04-26T01:45:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.853+0000 7f7d43dd4740 20 client.25150 trim_cache size 0 max 16384 2024-04-26T01:45:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.871+0000 7f7d43dd4740 1 Processor -- start 2024-04-26T01:45:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 -- start start 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e49b0 0x56112c5e7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5efb90 0x56112c5eea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1b23e0 con 0x56112c5efb90 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1afcb0 con 0x56112c5e49b0 2024-04-26T01:45:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1adb20 con 0x56112c5e52d0 2024-04-26T01:45:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d31400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d3a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5efb90 0x56112c5eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e49b0 0x56112c5e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d31400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44680/0 (socket says 172.21.15.52:44680) 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.872+0000 7f7d31400640 1 -- 172.21.15.52:0/218142085 learned_addr learned my addr 172.21.15.52:0/218142085 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 -- 172.21.15.52:0/218142085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e49b0 msgr2=0x56112c5e7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e49b0 0x56112c5e7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 -- 172.21.15.52:0/218142085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5efb90 msgr2=0x56112c5eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5efb90 0x56112c5eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 -- 172.21.15.52:0/218142085 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112c5e5100 con 0x56112c5e52d0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d31400640 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d24098f30 tx=0x7f7d2409c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7044fd60419b5acd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d2aa00640 1 -- 172.21.15.52:0/218142085 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d240a9020 con 0x56112c5e52d0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d2aa00640 1 -- 172.21.15.52:0/218142085 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d240a0e80 con 0x56112c5e52d0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.873+0000 7f7d2aa00640 1 -- 172.21.15.52:0/218142085 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d240ce7f0 con 0x56112c5e52d0 2024-04-26T01:45:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 -- 172.21.15.52:0/218142085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 msgr2=0x56112c5ed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d24098f30 tx=0x7f7d2409c560 comp rx=0 tx=0).stop 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 -- 172.21.15.52:0/218142085 shutdown_connections 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5efb90 0x56112c5eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5e52d0 0x56112c5ed3f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/218142085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e49b0 0x56112c5e7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 -- 172.21.15.52:0/218142085 >> 172.21.15.52:0/218142085 conn(0x56112c5e0ec0 msgr2=0x56112c5ee360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 -- 172.21.15.52:0/218142085 shutdown_connections 2024-04-26T01:45:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.874+0000 7f7d43dd4740 1 -- 172.21.15.52:0/218142085 wait complete. 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 -- start start 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 Processor -- start 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e52d0 0x56112c5f5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5efb90 0x56112c5f5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1ae170 con 0x56112c5e49b0 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1af5e0 con 0x56112c5e52d0 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1ae840 con 0x56112c5efb90 2024-04-26T01:45:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.875+0000 7f7d43dd4740 10 client.0 _ll_register_callbacks cb 0x7f7d2409c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40798/0 (socket says 172.21.15.52:40798) 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 -- 172.21.15.52:0/3431361345 learned_addr learned my addr 172.21.15.52:0/3431361345 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d3a800640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5efb90 0x56112c5f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d31400640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e52d0 0x56112c5f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 -- 172.21.15.52:0/3431361345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5efb90 msgr2=0x56112c5f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5efb90 0x56112c5f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 -- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e52d0 msgr2=0x56112c5f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e52d0 0x56112c5f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 -- 172.21.15.52:0/3431361345 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112c5f5cb0 con 0x56112c5e49b0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d39e00640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009900 tx=0x7f7d2c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5eea33834a6bfe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.876+0000 7f7d1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c035050 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7d2c0522d0 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d240a0e80 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7d2c0524d0 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d43dd4740 10 client.15051 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56112c666340 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.877+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d240a0e80 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7d2c003ca0 con 0x56112c5e49b0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 10 client.15051 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 10 client.15051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 20 client.15051 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c666570 0x56112c668960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 4 client.15051 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56112c558320 con 0x56112c666570 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.878+0000 7f7d31400640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c666570 0x56112c668960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.879+0000 7f7d31400640 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c666570 0x56112c668960 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92935de1c7a62b39 server_cookie=4adbd9a023eb1ba2 in_seq=0 out_seq=0 2024-04-26T01:45:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.879+0000 7f7d1fe00640 10 client.15051 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d1fe00640 1 -- 172.21.15.52:0/3431361345 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56112c558320 con 0x56112c666570 2024-04-26T01:45:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d1fe00640 10 client.15051 handle_command_reply: tid=0 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 client.15051 shutdown 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 msgr2=0x56112c5f7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009900 tx=0x7f7d2c02a410 comp rx=0 tx=0).stop 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 shutdown_connections 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56112c666570 0x56112c668960 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56112c5efb90 0x56112c5f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56112c5e52d0 0x56112c5f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 --2- 172.21.15.52:0/3431361345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56112c5e49b0 0x56112c5f7d70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 >> 172.21.15.52:0/3431361345 conn(0x56112c5e0ec0 msgr2=0x56112c5eebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d1fe00640 0 client.15051 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.891+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 shutdown_connections 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d43dd4740 1 -- 172.21.15.52:0/3431361345 wait complete. 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d43dd4740 20 client.15051 trim_cache size 0 max 16384 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:45:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:45:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 msgr2=0x7f7d200a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 0x7f7d200a6200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d34005fd0 tx=0x7f7d34009500 comp rx=0 tx=0).stop 2024-04-26T01:45:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 msgr2=0x7f7d3c1a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.892+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9f0 tx=0x7f7d2400edf0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 shutdown_connections 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7d200a3d40 0x7f7d200a6200 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d3c19a620 0x7f7d3c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d3c1092d0 0x7f7d3c1a09b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 --2- 172.21.15.52:0/3189659620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d3c104200 0x7f7d3c1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 >> 172.21.15.52:0/3189659620 conn(0x7f7d3c0ffa60 msgr2=0x7f7d3c100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 shutdown_connections 2024-04-26T01:45:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:15.893+0000 7f7d3b200640 1 -- 172.21.15.52:0/3189659620 wait complete. 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 -- 172.21.15.52:0/3598487026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 msgr2=0x7fa278102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 --2- 172.21.15.52:0/3598487026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278102f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa2640098f0 tx=0x7fa264009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 -- 172.21.15.52:0/3598487026 shutdown_connections 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 --2- 172.21.15.52:0/3598487026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa27810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 --2- 172.21.15.52:0/3598487026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa278108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 --2- 172.21.15.52:0/3598487026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278102f10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.275+0000 7fa27e000700 1 -- 172.21.15.52:0/3598487026 >> 172.21.15.52:0/3598487026 conn(0x7fa278079ba0 msgr2=0x7fa278079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.276+0000 7fa27e000700 1 -- 172.21.15.52:0/3598487026 shutdown_connections 2024-04-26T01:45:17.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.276+0000 7fa27e000700 1 -- 172.21.15.52:0/3598487026 wait complete. 2024-04-26T01:45:17.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.277+0000 7fa27e000700 1 Processor -- start 2024-04-26T01:45:17.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 -- start start 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa2780757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa278112c50 con 0x7fa278108a50 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa278112ad0 con 0x7fa278102a90 2024-04-26T01:45:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.278+0000 7fa27e000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa278112dd0 con 0x7fa278103450 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa277400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa277e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa2780757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44714/0 (socket says 172.21.15.52:44714) 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa277e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa2780757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40822/0 (socket says 172.21.15.52:40822) 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa277400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37530/0 (socket says 172.21.15.52:37530) 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 -- 172.21.15.52:0/827443310 learned_addr learned my addr 172.21.15.52:0/827443310 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 -- 172.21.15.52:0/827443310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 msgr2=0x7fa278074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 -- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 msgr2=0x7fa2780757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa2780757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.279+0000 7fa276a00700 1 -- 172.21.15.52:0/827443310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2781a7730 con 0x7fa278103450 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa276a00700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa264009230 tx=0x7fa26400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640050c0 con 0x7fa278103450 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa26400b560 con 0x7fa278103450 2024-04-26T01:45:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264003a00 con 0x7fa278103450 2024-04-26T01:45:17.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2781a79c0 con 0x7fa278103450 2024-04-26T01:45:17.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.280+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2781a7e50 con 0x7fa278103450 2024-04-26T01:45:17.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.281+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa278069730 con 0x7fa278103450 2024-04-26T01:45:17.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.284+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fa26400b710 con 0x7fa278103450 2024-04-26T01:45:17.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.285+0000 7fa26fe00700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 0x7fa25c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:17.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.285+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa2640d1a40 con 0x7fa278103450 2024-04-26T01:45:17.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.285+0000 7fa277400700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 0x7fa25c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:17.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.286+0000 7fa277400700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 0x7fa25c0a6380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa268009750 tx=0x7fa268009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:17.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.287+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa26405eff0 con 0x7fa278103450 2024-04-26T01:45:17.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.571+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa27802e020 con 0x7fa278103450 2024-04-26T01:45:17.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:17 smithi052 ceph-mon[27851]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:17.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.572+0000 7fa26fe00700 1 -- 172.21.15.52:0/827443310 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fa264013030 con 0x7fa278103450 2024-04-26T01:45:17.570 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:17.570 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:44:59.710897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:45:17.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 msgr2=0x7fa25c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:17.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 0x7fa25c0a6380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa268009750 tx=0x7fa268009310 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 msgr2=0x7fa278075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa264009230 tx=0x7fa26400fa70 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 shutdown_connections 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa25c0a3ec0 0x7fa25c0a6380 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa278108a50 0x7fa2780757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa278102a90 0x7fa278074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 --2- 172.21.15.52:0/827443310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa278103450 0x7fa278075280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 >> 172.21.15.52:0/827443310 conn(0x7fa278079ba0 msgr2=0x7fa278101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 shutdown_connections 2024-04-26T01:45:17.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:17.577+0000 7fa27e000700 1 -- 172.21.15.52:0/827443310 wait complete. 2024-04-26T01:45:17.576 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 25 2024-04-26T01:45:17.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:17 smithi183 ceph-mon[32752]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:17.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:17 smithi191 ceph-mon[32745]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-26T01:45:18.454 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:18.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:18 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/827443310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:18.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:18 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/827443310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:18.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:18 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/827443310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:18.914 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:19.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:19 smithi183 ceph-mon[32752]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T01:45:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:19 smithi052 ceph-mon[27851]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T01:45:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:19 smithi191 ceph-mon[32745]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-26T01:45:21.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 -- 172.21.15.52:0/382815872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 msgr2=0x7fecb806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:21.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 --2- 172.21.15.52:0/382815872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb806abb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feca0009990 tx=0x7feca0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 -- 172.21.15.52:0/382815872 shutdown_connections 2024-04-26T01:45:21.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 --2- 172.21.15.52:0/382815872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 --2- 172.21.15.52:0/382815872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 0x7fecb8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 --2- 172.21.15.52:0/382815872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb806abb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 -- 172.21.15.52:0/382815872 >> 172.21.15.52:0/382815872 conn(0x7fecb80788f0 msgr2=0x7fecb8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:21.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 -- 172.21.15.52:0/382815872 shutdown_connections 2024-04-26T01:45:21.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.013+0000 7fecbe800700 1 -- 172.21.15.52:0/382815872 wait complete. 2024-04-26T01:45:21.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.014+0000 7fecbe800700 1 Processor -- start 2024-04-26T01:45:21.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.014+0000 7fecbe800700 1 -- start start 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb8103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 0x7fecb8102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fecb8112c60 con 0x7fecb806a750 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fecb8112ae0 con 0x7fecb8108a30 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecbe800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fecb8112de0 con 0x7fecb806b0f0 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecb7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb8103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecaea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecaea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44728/0 (socket says 172.21.15.52:44728) 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecaea00700 1 -- 172.21.15.52:0/1033075822 learned_addr learned my addr 172.21.15.52:0/1033075822 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.015+0000 7fecb7e00700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 0x7fecb8102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 -- 172.21.15.52:0/1033075822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 msgr2=0x7fecb8102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:21.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 0x7fecb8102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 -- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 msgr2=0x7fecb8103600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:21.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb8103600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 -- 172.21.15.52:0/1033075822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb81abb80 con 0x7fecb806b0f0 2024-04-26T01:45:21.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecb7400700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb8103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:21.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecaea00700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feca0005910 tx=0x7feca000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.016+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0027070 con 0x7fecb806b0f0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.017+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feca0022470 con 0x7fecb806b0f0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.017+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb81abdb0 con 0x7fecb806b0f0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.017+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0017650 con 0x7fecb806b0f0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.017+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb81ac1b0 con 0x7fecb806b0f0 2024-04-26T01:45:21.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.018+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb8069730 con 0x7fecb806b0f0 2024-04-26T01:45:21.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.020+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7feca002a070 con 0x7fecb806b0f0 2024-04-26T01:45:21.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.021+0000 7fecb4c00700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 0x7fec9c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:21.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.022+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7feca00d1770 con 0x7fecb806b0f0 2024-04-26T01:45:21.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.022+0000 7fecb7400700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 0x7fec9c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:21.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.022+0000 7fecb7400700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 0x7fec9c0a6380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feca8009680 tx=0x7feca8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:21.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.024+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feca0013070 con 0x7fecb806b0f0 2024-04-26T01:45:21.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.308+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecb802df80 con 0x7fecb806b0f0 2024-04-26T01:45:21.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.309+0000 7fecb4c00700 1 -- 172.21.15.52:0/1033075822 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7feca009dd00 con 0x7fecb806b0f0 2024-04-26T01:45:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:21.307 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:44:59.710897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 msgr2=0x7fec9c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 0x7fec9c0a6380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feca8009680 tx=0x7feca8009260 comp rx=0 tx=0).stop 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 msgr2=0x7fecb8101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feca0005910 tx=0x7feca000b670 comp rx=0 tx=0).stop 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 shutdown_connections 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec9c0a3ec0 0x7fec9c0a6380 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecb8108a30 0x7fecb8102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fecb806a750 0x7fecb8103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 --2- 172.21.15.52:0/1033075822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecb806b0f0 0x7fecb8101a10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:21.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 >> 172.21.15.52:0/1033075822 conn(0x7fecb80788f0 msgr2=0x7fecb80ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:21.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.312+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 shutdown_connections 2024-04-26T01:45:21.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:21.313+0000 7fecbe800700 1 -- 172.21.15.52:0/1033075822 wait complete. 2024-04-26T01:45:21.311 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 25 2024-04-26T01:45:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:21 smithi183 ceph-mon[32752]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:21 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1033075822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:21.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:21 smithi052 ceph-mon[27851]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:21.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:21 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1033075822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:21.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:21 smithi191 ceph-mon[32745]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:21 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1033075822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:22.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T01:45:22.637 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:23.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:23 smithi183 ceph-mon[32752]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:23.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:23 smithi052 ceph-mon[27851]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:23.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:23 smithi191 ceph-mon[32745]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:25.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.024+0000 7f05a2200700 1 -- 172.21.15.52:0/3295214238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 msgr2=0x7f059c10f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.024+0000 7f05a2200700 1 --2- 172.21.15.52:0/3295214238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c10f840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0594009990 tx=0x7f0594009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 -- 172.21.15.52:0/3295214238 shutdown_connections 2024-04-26T01:45:25.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 --2- 172.21.15.52:0/3295214238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c10f840 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 --2- 172.21.15.52:0/3295214238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 --2- 172.21.15.52:0/3295214238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 -- 172.21.15.52:0/3295214238 >> 172.21.15.52:0/3295214238 conn(0x7f059c0fe4c0 msgr2=0x7f059c1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:25.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 -- 172.21.15.52:0/3295214238 shutdown_connections 2024-04-26T01:45:25.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.025+0000 7f05a2200700 1 -- 172.21.15.52:0/3295214238 wait complete. 2024-04-26T01:45:25.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.026+0000 7f05a2200700 1 Processor -- start 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.026+0000 7f05a2200700 1 -- start start 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.026+0000 7f05a2200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a2200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a2200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c19d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a2200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f059c112c00 con 0x7f059c103330 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a2200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f059c112a80 con 0x7f059c102970 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a2200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f059c112d80 con 0x7f059c108970 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a0e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a0e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41434/0 (socket says 172.21.15.52:41434) 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a0e00700 1 -- 172.21.15.52:0/1442187901 learned_addr learned my addr 172.21.15.52:0/1442187901 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:25.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f05a1800700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c19d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 -- 172.21.15.52:0/1442187901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 msgr2=0x7f059c19d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c19d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 -- 172.21.15.52:0/1442187901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 msgr2=0x7f059c199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.027+0000 7f059be00700 1 -- 172.21.15.52:0/1442187901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059c19dff0 con 0x7f059c103330 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f05a0e00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f05a1800700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c19d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f059be00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c199570 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f058800e9f0 tx=0x7f058800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058800c9c0 con 0x7f059c103330 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059c19e2e0 con 0x7f059c103330 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f058800cb60 con 0x7f059c103330 2024-04-26T01:45:25.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588018680 con 0x7f059c103330 2024-04-26T01:45:25.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.028+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059c1a5b60 con 0x7f059c103330 2024-04-26T01:45:25.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.030+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059c069730 con 0x7f059c103330 2024-04-26T01:45:25.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.032+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f0588010430 con 0x7f059c103330 2024-04-26T01:45:25.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.033+0000 7f0599600700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 0x7f05840a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:25.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.033+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f05880d0ac0 con 0x7f059c103330 2024-04-26T01:45:25.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.033+0000 7f05a0e00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 0x7f05840a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:25.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.034+0000 7f05a0e00700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 0x7f05840a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0590009c10 tx=0x7f059000b040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:25.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.038+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f058809cfc0 con 0x7f059c103330 2024-04-26T01:45:25.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.269+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f059c19a340 con 0x7f059c103330 2024-04-26T01:45:25.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.274+0000 7f0599600700 1 -- 172.21.15.52:0/1442187901 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+23044 (secure 0 0 0) 0x7f058809c8c0 con 0x7f059c103330 2024-04-26T01:45:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:25.272 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":95,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T01:44:59.818918+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/1613611936":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/2463815619":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/243168308":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/2167000831":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:45:25.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 msgr2=0x7f05840a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 0x7f05840a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0590009c10 tx=0x7f059000b040 comp rx=0 tx=0).stop 2024-04-26T01:45:25.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 msgr2=0x7f059c199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:25.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c199570 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f058800e9f0 tx=0x7f058800ed00 comp rx=0 tx=0).stop 2024-04-26T01:45:25.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 shutdown_connections 2024-04-26T01:45:25.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f05840a3ec0 0x7f05840a6380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f059c108970 0x7f059c19d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f059c102970 0x7f059c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 --2- 172.21.15.52:0/1442187901 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f059c103330 0x7f059c199570 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:25.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.277+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 >> 172.21.15.52:0/1442187901 conn(0x7f059c0fe4c0 msgr2=0x7f059c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:25.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.278+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 shutdown_connections 2024-04-26T01:45:25.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:25.278+0000 7f05a2200700 1 -- 172.21.15.52:0/1442187901 wait complete. 2024-04-26T01:45:25.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:25 smithi052 ceph-mon[27851]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:25.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:25 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1442187901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:25.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:25 smithi191 ceph-mon[32745]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:25 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1442187901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:25.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:25 smithi183 ceph-mon[32752]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:25.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:25 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1442187901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T01:45:26.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-26T01:45:26.649 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:27.689 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:27 smithi052 ceph-mon[27851]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:27.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:27 smithi191 ceph-mon[32745]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:27.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:27 smithi183 ceph-mon[32752]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 -- 172.21.15.52:0/3056837785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 msgr2=0x7f4344074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 --2- 172.21.15.52:0/3056837785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f4344074630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4334009900 tx=0x7f433402a480 comp rx=0 tx=0).stop 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 -- 172.21.15.52:0/3056837785 shutdown_connections 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 --2- 172.21.15.52:0/3056837785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f4344110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 --2- 172.21.15.52:0/3056837785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f434410e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 --2- 172.21.15.52:0/3056837785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f4344074630 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 -- 172.21.15.52:0/3056837785 >> 172.21.15.52:0/3056837785 conn(0x7f43440793e0 msgr2=0x7f43440797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 -- 172.21.15.52:0/3056837785 shutdown_connections 2024-04-26T01:45:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.032+0000 7f4348c00640 1 -- 172.21.15.52:0/3056837785 wait complete. 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.033+0000 7f4348c00640 1 Processor -- start 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.033+0000 7f4348c00640 1 -- start start 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.033+0000 7f4348c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f43441a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.033+0000 7f4348c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4348c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f434419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4348c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4344114450 con 0x7f434410e840 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4348c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43441142d0 con 0x7f4344074b70 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4348c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43441145d0 con 0x7f4344074250 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4341e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f43441a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4342800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f434419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41454/0 (socket says 172.21.15.52:41454) 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 -- 172.21.15.52:0/308463386 learned_addr learned my addr 172.21.15.52:0/308463386 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 -- 172.21.15.52:0/308463386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 msgr2=0x7f43441a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f43441a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 -- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 msgr2=0x7f434419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f434419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f433be00640 1 -- 172.21.15.52:0/308463386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434419af30 con 0x7f4344074b70 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.034+0000 7f4342800640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f434419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f433be00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f432c00e9f0 tx=0x7f432c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f433aa00640 1 -- 172.21.15.52:0/308463386 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c00cd80 con 0x7f4344074b70 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f433aa00640 1 -- 172.21.15.52:0/308463386 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f432c0054d0 con 0x7f4344074b70 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f433aa00640 1 -- 172.21.15.52:0/308463386 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c010690 con 0x7f4344074b70 2024-04-26T01:45:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434419b220 con 0x7f4344074b70 2024-04-26T01:45:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.035+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43441a7530 con 0x7f4344074b70 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.038+0000 7f434ba74740 1 Processor -- start 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.038+0000 7f433aa00640 1 -- 172.21.15.52:0/308463386 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f432c004e90 con 0x7f4344074b70 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 -- start start 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2de140 0x56498e2e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e2eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56498e243900 con 0x56498e2de140 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56498e1e9ef0 con 0x56498e2e8a00 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f434ba74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56498e23f8c0 con 0x56498e2dd820 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f4341e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2de140 0x56498e2e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f4342800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f4342800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54292/0 (socket says 172.21.15.52:54292) 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f4342800640 1 -- 172.21.15.52:0/848118017 learned_addr learned my addr 172.21.15.52:0/848118017 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.039+0000 7f433be00640 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e2eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 -- 172.21.15.52:0/848118017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 msgr2=0x56498e2eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e2eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 -- 172.21.15.52:0/848118017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2de140 msgr2=0x56498e2e84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2de140 0x56498e2e84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 -- 172.21.15.52:0/848118017 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498df56210 con 0x56498e2dd820 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4342800640 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f433c011790 tx=0x7f433c011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228d11b5acd7a5b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f433aa00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 0x7f43280a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f4333000640 1 -- 172.21.15.52:0/848118017 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c004d20 con 0x56498e2dd820 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.040+0000 7f433aa00640 1 -- 172.21.15.52:0/308463386 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f432c0d0d70 con 0x7f4344074b70 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f4333000640 1 -- 172.21.15.52:0/848118017 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c00fe90 con 0x56498e2dd820 2024-04-26T01:45:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f4333000640 1 -- 172.21.15.52:0/848118017 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c0059a0 con 0x56498e2dd820 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f4341e00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 0x7f43280a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 -- 172.21.15.52:0/848118017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 msgr2=0x56498e2ddc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f433c011790 tx=0x7f433c011c50 comp rx=0 tx=0).stop 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 -- 172.21.15.52:0/848118017 shutdown_connections 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e2eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2de140 0x56498e2e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 --2- 172.21.15.52:0/848118017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2ddc00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 -- 172.21.15.52:0/848118017 >> 172.21.15.52:0/848118017 conn(0x56498e2d9d30 msgr2=0x56498e2da110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f4341e00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 0x7f43280a63a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4334005ab0 tx=0x7f4334005a20 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.041+0000 7f434ba74740 1 -- 172.21.15.52:0/848118017 shutdown_connections 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.042+0000 7f434ba74740 1 -- 172.21.15.52:0/848118017 wait complete. 2024-04-26T01:45:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.042+0000 7f434ba74740 1 -- start start 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.042+0000 7f434ba74740 1 Processor -- start 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2e5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56498e2eb630 con 0x56498e2dd820 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56498e1fd310 con 0x56498e2de140 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56498e2eb7b0 con 0x56498e2e8a00 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f434ba74740 10 client.0 _ll_register_callbacks cb 0x56498e1cbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4341e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4342800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4342800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54296/0 (socket says 172.21.15.52:54296) 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4342800640 1 -- 172.21.15.52:0/1986569715 learned_addr learned my addr 172.21.15.52:0/1986569715 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f433be00640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4342800640 1 -- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 msgr2=0x56498e2e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.043+0000 7f4342800640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4342800640 1 -- 172.21.15.52:0/1986569715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 msgr2=0x56498e2e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4342800640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4342800640 1 -- 172.21.15.52:0/1986569715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498e022620 con 0x56498e2e8a00 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4341e00640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4342800640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f433c005600 tx=0x7f433c00fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76a9a3cf1c18c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4327400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c028020 con 0x56498e2e8a00 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f433c00fe90 con 0x56498e2e8a00 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.044+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f433c0058b0 con 0x56498e2e8a00 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f433c033050 con 0x56498e2e8a00 2024-04-26T01:45:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 10 client.25186 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56498e2f7870 con 0x56498e2e8a00 2024-04-26T01:45:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c00fe90 con 0x56498e2e8a00 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f433c003bd0 con 0x56498e2e8a00 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 10 client.25186 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 10 client.25186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 20 client.25186 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e2f8ce0 0x56498e2fb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 4 client.25186 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.045+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56498e2578b0 con 0x56498e2f8ce0 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.046+0000 7f4341e00640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e2f8ce0 0x56498e2fb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.046+0000 7f4341e00640 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e2f8ce0 0x56498e2fb0d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f071094b588828e server_cookie=b5f2a0edc07f169f in_seq=0 out_seq=0 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.046+0000 7f4327400640 10 client.25186 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f4327400640 1 -- 172.21.15.52:0/1986569715 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56498e2578b0 con 0x56498e2f8ce0 2024-04-26T01:45:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f4327400640 10 client.25186 handle_command_reply: tid=0 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 client.25186 shutdown 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 msgr2=0x56498e2e6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f433c005600 tx=0x7f433c00fc90 comp rx=0 tx=0).stop 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 shutdown_connections 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e2f8ce0 0x56498e2fb0d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2e8a00 0x56498e2e6400 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 --2- 172.21.15.52:0/1986569715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.047+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 >> 172.21.15.52:0/1986569715 conn(0x56498e2d9d30 msgr2=0x56498e2dfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.048+0000 7f4327400640 0 client.25186 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.048+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 shutdown_connections 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.048+0000 7f434ba74740 1 -- 172.21.15.52:0/1986569715 wait complete. 2024-04-26T01:45:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.048+0000 7f434ba74740 20 client.25186 trim_cache size 0 max 16384 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.064+0000 7f434ba74740 1 Processor -- start 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.064+0000 7f434ba74740 1 -- start start 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f43441b2550 con 0x56498e2e8a00 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43441afe20 con 0x56498e2de140 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f434ba74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43441adc70 con 0x56498e2dd820 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57526/0 (socket says 172.21.15.52:57526) 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 -- 172.21.15.52:0/641355380 learned_addr learned my addr 172.21.15.52:0/641355380 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4341e00640 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f433be00640 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 -- 172.21.15.52:0/641355380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 msgr2=0x56498e2f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 -- 172.21.15.52:0/641355380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 msgr2=0x56498e2e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.065+0000 7f4342800640 1 -- 172.21.15.52:0/641355380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498e2e1510 con 0x56498e2e8a00 2024-04-26T01:45:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f4342800640 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f433c005670 tx=0x7f433c011710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc79ae8d5f8511d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f4332600640 1 -- 172.21.15.52:0/641355380 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c03c070 con 0x56498e2e8a00 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f4332600640 1 -- 172.21.15.52:0/641355380 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0058b0 con 0x56498e2e8a00 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f4332600640 1 -- 172.21.15.52:0/641355380 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c028040 con 0x56498e2e8a00 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f434ba74740 1 -- 172.21.15.52:0/641355380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 msgr2=0x56498e300840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.066+0000 7f434ba74740 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f433c005670 tx=0x7f433c011710 comp rx=0 tx=0).stop 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 -- 172.21.15.52:0/641355380 shutdown_connections 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2e8a00 0x56498e300840 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2de140 0x56498e2e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 --2- 172.21.15.52:0/641355380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2dd820 0x56498e2f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 -- 172.21.15.52:0/641355380 >> 172.21.15.52:0/641355380 conn(0x56498e2d9d30 msgr2=0x56498e2e4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 -- 172.21.15.52:0/641355380 shutdown_connections 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 -- 172.21.15.52:0/641355380 wait complete. 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 -- start start 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.067+0000 7f434ba74740 1 Processor -- start 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2de140 0x56498e2edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e35d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f43441ae2e0 con 0x56498e2dd820 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43441af750 con 0x56498e2e8a00 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43441ae9b0 con 0x56498e2de140 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f434ba74740 10 client.0 _ll_register_callbacks cb 0x7f433c011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57540/0 (socket says 172.21.15.52:57540) 2024-04-26T01:45:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 -- 172.21.15.52:0/1553809185 learned_addr learned my addr 172.21.15.52:0/1553809185 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4342800640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e35d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f433be00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2de140 0x56498e2edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 -- 172.21.15.52:0/1553809185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2de140 msgr2=0x56498e2edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2de140 0x56498e2edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 -- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 msgr2=0x56498e35d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e35d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.068+0000 7f4341e00640 1 -- 172.21.15.52:0/1553809185 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498e2ee7b0 con 0x56498e2dd820 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f4341e00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43340041d0 tx=0x7f4334009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6e30b146c8bad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f4327400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334035040 con 0x56498e2dd820 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f4334054500 con 0x56498e2dd820 2024-04-26T01:45:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f433c0058b0 con 0x56498e2dd820 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4334054700 con 0x56498e2dd820 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f434ba74740 10 client.15087 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.069+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56498e35e760 con 0x56498e2dd820 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c0058b0 con 0x56498e2dd820 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4334003850 con 0x56498e2dd820 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 10 client.15087 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 10 client.15087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 20 client.15087 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e35eec0 0x56498e3612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 4 client.15087 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56498e251320 con 0x56498e35eec0 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.070+0000 7f433be00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e35eec0 0x56498e3612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.071+0000 7f433be00640 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e35eec0 0x56498e3612e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0ca98490680c6c2 server_cookie=1546146c82b5c456 in_seq=0 out_seq=0 2024-04-26T01:45:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.071+0000 7f4327400640 10 client.15087 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f4327400640 1 -- 172.21.15.52:0/1553809185 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56498e251320 con 0x56498e35eec0 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f4327400640 10 client.15087 handle_command_reply: tid=0 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 client.15087 shutdown 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 msgr2=0x56498e2ed9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43340041d0 tx=0x7f4334009260 comp rx=0 tx=0).stop 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 shutdown_connections 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56498e35eec0 0x56498e3612e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56498e2e8a00 0x56498e35d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56498e2de140 0x56498e2edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 --2- 172.21.15.52:0/1553809185 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56498e2dd820 0x56498e2ed9a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 >> 172.21.15.52:0/1553809185 conn(0x56498e2d9d30 msgr2=0x56498e2e4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f4327400640 0 client.15087 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:29.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 shutdown_connections 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 1 -- 172.21.15.52:0/1553809185 wait complete. 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.078+0000 7f434ba74740 20 client.15087 trim_cache size 0 max 16384 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:45:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 msgr2=0x7f43280a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 0x7f43280a63a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4334005ab0 tx=0x7f4334005a20 comp rx=0 tx=0).stop 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 msgr2=0x7f43441a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f432c00e9f0 tx=0x7f432c00edf0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 shutdown_connections 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f43280a3ee0 0x7f43280a63a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f434410e840 0x7f434419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4344074b70 0x7f43441a0af0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 --2- 172.21.15.52:0/308463386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4344074250 0x7f43441a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 >> 172.21.15.52:0/308463386 conn(0x7f43440793e0 msgr2=0x7f43440fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 shutdown_connections 2024-04-26T01:45:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:29.079+0000 7f4348c00640 1 -- 172.21.15.52:0/308463386 wait complete. 2024-04-26T01:45:29.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:29 smithi052 ceph-mon[27851]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:29.833 INFO:teuthology.orchestra.run.smithi052.stdout:{"run_dir":"/var/run/ceph"} 2024-04-26T01:45:29.834 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-26T01:45:29.834 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3448 2024-04-26T01:45:29.834 DEBUG:tasks.quiescer:Yielding 2024-04-26T01:45:29.834 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T01:45:29.848 INFO:tasks.workunit:Pulling workunits from ref afa1933b0fdbb6c99c947d1eda34d661d23cd327 2024-04-26T01:45:29.849 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T01:45:29.849 INFO:tasks.workunit:timeout=3h 2024-04-26T01:45:29.849 INFO:tasks.workunit:cleanup=True 2024-04-26T01:45:29.849 DEBUG:teuthology.orchestra.run.smithi052:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:29.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:45:29.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:29 smithi191 ceph-mon[32745]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 69h/105d Inode: 1099511628283 Links: 2 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-04-26 01:41:46.256884954 +0000 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-04-26 01:41:46.256884954 +0000 2024-04-26T01:45:29.904 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-04-26 01:44:58.616550274 +0000 2024-04-26T01:45:29.905 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-04-26 01:41:46.256884954 +0000 2024-04-26T01:45:29.905 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-26T01:45:29.905 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-26T01:45:29.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:29 smithi183 ceph-mon[32752]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-26T01:45:30.036 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout afa1933b0fdbb6c99c947d1eda34d661d23cd327 2024-04-26T01:45:30.090 INFO:tasks.workunit.client.0.smithi052.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T01:45:30.294 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:31.704 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:31 smithi191 ceph-mon[32745]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:31.706 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:31 smithi052 ceph-mon[27851]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:31.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:31 smithi183 ceph-mon[32752]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:45:32.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 -- 172.21.15.52:0/3102509068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c06ac90 msgr2=0x7fe88c06b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:32.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 --2- 172.21.15.52:0/3102509068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c06ac90 0x7fe88c06b0f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe8780098f0 tx=0x7fe878009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:32.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 -- 172.21.15.52:0/3102509068 shutdown_connections 2024-04-26T01:45:32.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 --2- 172.21.15.52:0/3102509068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 0x7fe88c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 --2- 172.21.15.52:0/3102509068 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c107970 0x7fe88c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 --2- 172.21.15.52:0/3102509068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c06ac90 0x7fe88c06b0f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 -- 172.21.15.52:0/3102509068 >> 172.21.15.52:0/3102509068 conn(0x7fe88c0788f0 msgr2=0x7fe88c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:32.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.580+0000 7fe890a00700 1 -- 172.21.15.52:0/3102509068 shutdown_connections 2024-04-26T01:45:32.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.582+0000 7fe890a00700 1 -- 172.21.15.52:0/3102509068 wait complete. 2024-04-26T01:45:32.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 Processor -- start 2024-04-26T01:45:32.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 -- start start 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 0x7fe88c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 0x7fe88c1a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c119210 con 0x7fe88c06ac90 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c119090 con 0x7fe88c107970 2024-04-26T01:45:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.583+0000 7fe890a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c119390 con 0x7fe88c105480 2024-04-26T01:45:32.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe889e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:32.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 0x7fe88c1a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:32.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe889e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57552/0 (socket says 172.21.15.52:57552) 2024-04-26T01:45:32.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe889e00700 1 -- 172.21.15.52:0/3066481259 learned_addr learned my addr 172.21.15.52:0/3066481259 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe889400700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 0x7fe88c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 -- 172.21.15.52:0/3066481259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 msgr2=0x7fe88c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 0x7fe88c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 -- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 msgr2=0x7fe88c19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe88a800700 1 -- 172.21.15.52:0/3066481259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c1a4750 con 0x7fe88c107970 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.584+0000 7fe889e00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:32.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe88a800700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 0x7fe88c1a4010 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe88400efc0 tx=0x7fe88400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8840092e0 con 0x7fe88c107970 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c1a49e0 con 0x7fe88c107970 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe884004bb0 con 0x7fe88c107970 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884004020 con 0x7fe88c107970 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.585+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c1ac2c0 con 0x7fe88c107970 2024-04-26T01:45:32.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.587+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88c069730 con 0x7fe88c107970 2024-04-26T01:45:32.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.589+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fe88401b030 con 0x7fe88c107970 2024-04-26T01:45:32.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.590+0000 7fe882a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 0x7fe8700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:32.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.590+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fe8840d04a0 con 0x7fe88c107970 2024-04-26T01:45:32.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.591+0000 7fe889e00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 0x7fe8700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:32.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.591+0000 7fe889e00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 0x7fe8700a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009750 tx=0x7fe87c009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:32.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.595+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe884022020 con 0x7fe88c107970 2024-04-26T01:45:32.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:32 smithi052 ceph-mon[27851]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-26T01:45:32.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.891+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe88c1ac490 con 0x7fe88c107970 2024-04-26T01:45:32.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.892+0000 7fe882a00700 1 -- 172.21.15.52:0/3066481259 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fe884010040 con 0x7fe88c107970 2024-04-26T01:45:32.890 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:45:32.890 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:44:59.710897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:45:32.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 msgr2=0x7fe8700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:32.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 0x7fe8700a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009750 tx=0x7fe87c009310 comp rx=0 tx=0).stop 2024-04-26T01:45:32.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 msgr2=0x7fe88c1a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:32.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 0x7fe88c1a4010 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe88400efc0 tx=0x7fe88400c3a0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 shutdown_connections 2024-04-26T01:45:32.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8700a3e40 0x7fe8700a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe88c107970 0x7fe88c1a4010 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe88c06ac90 0x7fe88c19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 --2- 172.21.15.52:0/3066481259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe88c105480 0x7fe88c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:32.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 >> 172.21.15.52:0/3066481259 conn(0x7fe88c0788f0 msgr2=0x7fe88c104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:32.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.895+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 shutdown_connections 2024-04-26T01:45:32.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:32.896+0000 7fe890a00700 1 -- 172.21.15.52:0/3066481259 wait complete. 2024-04-26T01:45:32.895 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 25 2024-04-26T01:45:32.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:32 smithi191 ceph-mon[32745]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-26T01:45:32.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:32 smithi183 ceph-mon[32752]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-26T01:45:33.859 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:33 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3066481259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:33.886 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:33 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3066481259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:33.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:33 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3066481259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:45:33.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T01:41:06.424614+0000', 'modified': '2024-04-26T01:44:59.710897+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24524, 'mds_1': 24512, 'mds_2': 24496}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24472': {'gid': 24472, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/234906181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 234906181}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 234906181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6837/2929119914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2929119914}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2929119914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.183:6837/1263803776', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6836', 'nonce': 1263803776}, {'type': 'v1', 'addr': '172.21.15.183:6837', 'nonce': 1263803776}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.191:6839/1544004799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 1544004799}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 1544004799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'a', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.52:6841/1834450967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6840', 'nonce': 1834450967}, {'type': 'v1', 'addr': '172.21.15.52:6841', 'nonce': 1834450967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.183:6839/1410734884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6838', 'nonce': 1410734884}, {'type': 'v1', 'addr': '172.21.15.183:6839', 'nonce': 1410734884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24512, 24524, 24496]} 2024-04-26T01:45:33.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T01:45:33.937 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-26T01:45:34.421 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:34.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:34 smithi052 ceph-mon[27851]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:34.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:34 smithi191 ceph-mon[32745]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:34.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:34 smithi183 ceph-mon[32752]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 -- 172.21.15.52:0/1064615048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 msgr2=0x7fc678108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 --2- 172.21.15.52:0/1064615048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc678108e10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc66c009990 tx=0x7fc66c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 -- 172.21.15.52:0/1064615048 shutdown_connections 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 --2- 172.21.15.52:0/1064615048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 0x7fc67810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 --2- 172.21.15.52:0/1064615048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc678108e10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 --2- 172.21.15.52:0/1064615048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 -- 172.21.15.52:0/1064615048 >> 172.21.15.52:0/1064615048 conn(0x7fc6780788f0 msgr2=0x7fc678078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 -- 172.21.15.52:0/1064615048 shutdown_connections 2024-04-26T01:45:36.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.807+0000 7fc67f600700 1 -- 172.21.15.52:0/1064615048 wait complete. 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.808+0000 7fc67f600700 1 Processor -- start 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 -- start start 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 0x7fc67819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc67819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc678112c10 con 0x7fc678108a30 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc678112a90 con 0x7fc67806b0f0 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67f600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc678112d90 con 0x7fc67806a750 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.809+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60516/0 (socket says 172.21.15.52:60516) 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 -- 172.21.15.52:0/1673418601 learned_addr learned my addr 172.21.15.52:0/1673418601 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc677e00700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 0x7fc67819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67d600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc67819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 -- 172.21.15.52:0/1673418601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 msgr2=0x7fc67819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 0x7fc67819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 -- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 msgr2=0x7fc67819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc67819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.810+0000 7fc67cc00700 1 -- 172.21.15.52:0/1673418601 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6781a0370 con 0x7fc67806a750 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.811+0000 7fc67cc00700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc66c000c00 tx=0x7fc66c0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.811+0000 7fc675600700 1 -- 172.21.15.52:0/1673418601 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c027070 con 0x7fc67806a750 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.811+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6781a0600 con 0x7fc67806a750 2024-04-26T01:45:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.811+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6781a7ee0 con 0x7fc67806a750 2024-04-26T01:45:36.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.812+0000 7fc675600700 1 -- 172.21.15.52:0/1673418601 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c003680 con 0x7fc67806a750 2024-04-26T01:45:36.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.812+0000 7fc675600700 1 -- 172.21.15.52:0/1673418601 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c022aa0 con 0x7fc67806a750 2024-04-26T01:45:36.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.815+0000 7fc675600700 1 -- 172.21.15.52:0/1673418601 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fc66c02a070 con 0x7fc67806a750 2024-04-26T01:45:36.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.817+0000 7fc68a01cb80 1 Processor -- start 2024-04-26T01:45:36.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.817+0000 7fc68a01cb80 1 -- start start 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.817+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c131536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.817+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.817+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c130a6e40 con 0x563c131532f0 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c1304d6e0 con 0x563c1315eab0 2024-04-26T01:45:36.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c130a2e00 con 0x563c13153c10 2024-04-26T01:45:36.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc675600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 0x7fc6600a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc675600700 1 -- 172.21.15.52:0/1673418601 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fc66c0d1b20 con 0x7fc67806a750 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67d600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c131536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13160ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49924/0 (socket says 172.21.15.52:49924) 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60522/0 (socket says 172.21.15.52:60522) 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc677e00700 1 -- 172.21.15.52:0/2201614144 learned_addr learned my addr 172.21.15.52:0/2201614144 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67d600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c131536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48880/0 (socket says 172.21.15.52:48880) 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67d600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 0x7fc6600a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 -- 172.21.15.52:0/2201614144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 msgr2=0x563c13160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13160ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 -- 172.21.15.52:0/2201614144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 msgr2=0x563c131536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c131536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.818+0000 7fc67cc00700 1 -- 172.21.15.52:0/2201614144 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c13043630 con 0x563c13153c10 2024-04-26T01:45:36.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.819+0000 7fc67cc00700 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0cf180 tx=0x7fc66c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdb8802f40f02c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.819+0000 7fc66a000700 1 -- 172.21.15.52:0/2201614144 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c09d4f0 con 0x563c13153c10 2024-04-26T01:45:36.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.819+0000 7fc66a000700 1 -- 172.21.15.52:0/2201614144 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c09e0b0 con 0x563c13153c10 2024-04-26T01:45:36.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.819+0000 7fc66a000700 1 -- 172.21.15.52:0/2201614144 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c09d310 con 0x563c13153c10 2024-04-26T01:45:36.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.819+0000 7fc67d600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 0x7fc6600a6380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc6700098c0 tx=0x7fc670009100 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2201614144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 msgr2=0x563c1315e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0cf180 tx=0x7fc66c01d000 comp rx=0 tx=0).stop 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2201614144 shutdown_connections 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c131536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2201614144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1315e570 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2201614144 >> 172.21.15.52:0/2201614144 conn(0x563c13082550 msgr2=0x563c1314f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2201614144 shutdown_connections 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.820+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2201614144 wait complete. 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 -- start start 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 Processor -- start 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c1315b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c1315eab0 0x563c1315b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c131613e0 con 0x563c13153c10 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c13060810 con 0x563c131532f0 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.821+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c13161560 con 0x563c1315eab0 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc68a01cb80 10 client.0 _ll_register_callbacks cb 0x563c1303f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49926/0 (socket says 172.21.15.52:49926) 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 -- 172.21.15.52:0/150772813 learned_addr learned my addr 172.21.15.52:0/150772813 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67d600700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c1315eab0 0x563c1315b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc677e00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c1315b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 -- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c1315eab0 msgr2=0x563c1315b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c1315eab0 0x563c1315b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 -- 172.21.15.52:0/150772813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 msgr2=0x563c1315b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c1315b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc67cc00700 1 -- 172.21.15.52:0/150772813 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c12ff4880 con 0x563c131532f0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.822+0000 7fc677e00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c1315b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc67cc00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc66c006010 tx=0x7fc66c09de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=620f88314b301dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c09d310 con 0x563c131532f0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fc66c0a6310 con 0x563c131532f0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66c09e0b0 con 0x563c131532f0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc66c09d4f0 con 0x563c131532f0 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc68a01cb80 10 client.25199 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:36.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563c1303daa0 con 0x563c131532f0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c09e0b0 con 0x563c131532f0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.823+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fc66c09ece0 con 0x563c131532f0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 10 client.25199 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 10 client.25199 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 20 client.25199 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c1316dc20 0x563c1316f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 4 client.25199 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c130b4630 con 0x563c1316dc20 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.824+0000 7fc677e00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c1316dc20 0x563c1316f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.825+0000 7fc677e00700 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c1316dc20 0x563c1316f090 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c87a7dc521c78813 server_cookie=53453eee013956cf in_seq=0 out_seq=0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.825+0000 7fc65ea00700 10 client.25199 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.825+0000 7fc65ea00700 1 -- 172.21.15.52:0/150772813 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c130b4630 con 0x563c1316dc20 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.825+0000 7fc65ea00700 10 client.25199 handle_command_reply: tid=0 2024-04-26T01:45:36.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 client.25199 shutdown 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 msgr2=0x563c1315e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc66c006010 tx=0x7fc66c09de40 comp rx=0 tx=0).stop 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 shutdown_connections 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c1316dc20 0x563c1316f090 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c1315eab0 0x563c1315b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c131532f0 0x563c1315e100 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc65ea00700 0 client.25199 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/150772813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c1315b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 >> 172.21.15.52:0/150772813 conn(0x563c13082550 msgr2=0x563c1314e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 shutdown_connections 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.826+0000 7fc68a01cb80 1 -- 172.21.15.52:0/150772813 wait complete. 2024-04-26T01:45:36.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.827+0000 7fc68a01cb80 20 client.25199 trim_cache size 0 max 16384 2024-04-26T01:45:36.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.850+0000 7fc68a01cb80 1 Processor -- start 2024-04-26T01:45:36.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.850+0000 7fc68a01cb80 1 -- start start 2024-04-26T01:45:36.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.850+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c1314f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.850+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c131784f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c1316d3d0 con 0x563c131532f0 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c1316daa0 con 0x563c1315eab0 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c1316e170 con 0x563c13153c10 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc677e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc67cc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c1314f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc677e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60534/0 (socket says 172.21.15.52:60534) 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc677e00700 1 -- 172.21.15.52:0/2982990664 learned_addr learned my addr 172.21.15.52:0/2982990664 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:36.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.851+0000 7fc67d600700 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c131784f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 -- 172.21.15.52:0/2982990664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 msgr2=0x563c1314f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 -- 172.21.15.52:0/2982990664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 msgr2=0x563c1314f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c1314f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 -- 172.21.15.52:0/2982990664 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c1315dd40 con 0x563c1315eab0 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc677e00700 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc67d600700 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c131784f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc670009650 tx=0x7fc67000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b4f5cf8c19fb69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc669600700 1 -- 172.21.15.52:0/2982990664 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670004bb0 con 0x563c1315eab0 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc669600700 1 -- 172.21.15.52:0/2982990664 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc670004d50 con 0x563c1315eab0 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.852+0000 7fc669600700 1 -- 172.21.15.52:0/2982990664 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6700053a0 con 0x563c1315eab0 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2982990664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 msgr2=0x563c131784f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c131784f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc670009650 tx=0x7fc67000b040 comp rx=0 tx=0).stop 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2982990664 shutdown_connections 2024-04-26T01:45:36.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c131784f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c131532f0 0x563c1314f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/2982990664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c13153c10 0x563c1314f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2982990664 >> 172.21.15.52:0/2982990664 conn(0x563c13082550 msgr2=0x563c13160ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.853+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2982990664 shutdown_connections 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.854+0000 7fc68a01cb80 1 -- 172.21.15.52:0/2982990664 wait complete. 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.854+0000 7fc68a01cb80 1 -- start start 2024-04-26T01:45:36.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.854+0000 7fc68a01cb80 1 Processor -- start 2024-04-26T01:45:36.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 0x563c13158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c13158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c1316ef10 con 0x563c13153c10 2024-04-26T01:45:36.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c1316e840 con 0x563c1315eab0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c1316f5e0 con 0x563c131532f0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc68a01cb80 10 client.0 _ll_register_callbacks cb 0x563c1303f6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc67d600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc677e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c13158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc67d600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49952/0 (socket says 172.21.15.52:49952) 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc67d600700 1 -- 172.21.15.52:0/569696196 learned_addr learned my addr 172.21.15.52:0/569696196 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc67cc00700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 0x563c13158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.855+0000 7fc67d600700 1 -- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 msgr2=0x563c13158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67d600700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 0x563c13158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67d600700 1 -- 172.21.15.52:0/569696196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 msgr2=0x563c13158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:36.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67d600700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c13158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67d600700 1 -- 172.21.15.52:0/569696196 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c13175750 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67cc00700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 0x563c13158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc677e00700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c13158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc67d600700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc670018750 tx=0x7fc670018780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f6a81033502d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67002e620 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc670004d50 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.856+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fc67002e7c0 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc67002e9c0 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 10 client.25205 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563c131d50d0 con 0x563c1315eab0 2024-04-26T01:45:36.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc670004d50 con 0x563c1315eab0 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fc670038b50 con 0x563c1315eab0 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 10 client.25205 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 10 client.25205 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 20 client.25205 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c131d9940 0x563c131d8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 4 client.25205 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.857+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "status"}) -- 0x563c130ae6d0 con 0x563c131d9940 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.858+0000 7fc67cc00700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c131d9940 0x563c131d8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.858+0000 7fc67cc00700 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c131d9940 0x563c131d8650 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62582dc40cea33c7 server_cookie=c174a72d1d47b0ef in_seq=0 out_seq=0 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.858+0000 7fc65ea00700 10 client.25205 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc65ea00700 1 -- 172.21.15.52:0/569696196 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563c130ae6d0 con 0x563c131d9940 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc65ea00700 10 client.25205 handle_command_reply: tid=0 2024-04-26T01:45:36.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 client.25205 shutdown 2024-04-26T01:45:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 msgr2=0x563c13158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc670018750 tx=0x7fc670018780 comp rx=0 tx=0).stop 2024-04-26T01:45:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 shutdown_connections 2024-04-26T01:45:36.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x563c131d9940 0x563c131d8650 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c1315eab0 0x563c13158bd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c131532f0 0x563c13158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 --2- 172.21.15.52:0/569696196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c13153c10 0x563c13158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 >> 172.21.15.52:0/569696196 conn(0x563c13082550 msgr2=0x563c13156310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc65ea00700 0 client.25205 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 shutdown_connections 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 1 -- 172.21.15.52:0/569696196 wait complete. 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.859+0000 7fc68a01cb80 20 client.25205 trim_cache size 0 max 16384 2024-04-26T01:45:36.861 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 2, 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24496, 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 252.49248111899999, 2024-04-26T01:45:36.862 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 95, 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 304.05868739800002 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 msgr2=0x7fc6600a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 0x7fc6600a6380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc6700098c0 tx=0x7fc670009100 comp rx=0 tx=0).stop 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 msgr2=0x7fc67819b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc66c000c00 tx=0x7fc66c0057a0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 shutdown_connections 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc6600a3ec0 0x7fc6600a6380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc678108a30 0x7fc67819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc67806a750 0x7fc67819b360 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 --2- 172.21.15.52:0/1673418601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc67806b0f0 0x7fc67819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.860+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 >> 172.21.15.52:0/1673418601 conn(0x7fc6780788f0 msgr2=0x7fc6780ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.861+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 shutdown_connections 2024-04-26T01:45:36.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:36.861+0000 7fc67f600700 1 -- 172.21.15.52:0/1673418601 wait complete. 2024-04-26T01:45:37.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:36 smithi183 ceph-mon[32752]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:36 smithi052 ceph-mon[27851]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:37.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:36 smithi191 ceph-mon[32745]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:37.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.a status 2024-04-26T01:45:38.343 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:39.174 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:38 smithi183 ceph-mon[32752]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:39.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:38 smithi052 ceph-mon[27851]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:39.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:38 smithi191 ceph-mon[32745]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 -- 172.21.15.52:0/512286480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0075590 msgr2=0x7faeb010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 --2- 172.21.15.52:0/512286480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0075590 0x7faeb010ef00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faea0009990 tx=0x7faea0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 -- 172.21.15.52:0/512286480 shutdown_connections 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 --2- 172.21.15.52:0/512286480 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 0x7faeb0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 --2- 172.21.15.52:0/512286480 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0074c70 0x7faeb0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 --2- 172.21.15.52:0/512286480 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0075590 0x7faeb010ef00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 -- 172.21.15.52:0/512286480 >> 172.21.15.52:0/512286480 conn(0x7faeb0079ba0 msgr2=0x7faeb0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 -- 172.21.15.52:0/512286480 shutdown_connections 2024-04-26T01:45:40.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.734+0000 7faeb7600700 1 -- 172.21.15.52:0/512286480 wait complete. 2024-04-26T01:45:40.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.736+0000 7faeb7600700 1 Processor -- start 2024-04-26T01:45:40.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.736+0000 7faeb7600700 1 -- start start 2024-04-26T01:45:40.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 0x7faeb019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 0x7faeb019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 0x7faeb019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0114e10 con 0x7faeb0074c70 2024-04-26T01:45:40.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0114c90 con 0x7faeb010f440 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb7600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0114f90 con 0x7faeb0075590 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb4c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 0x7faeb019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb4c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 0x7faeb019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48910/0 (socket says 172.21.15.52:48910) 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faeb4c00700 1 -- 172.21.15.52:0/2096183584 learned_addr learned my addr 172.21.15.52:0/2096183584 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.737+0000 7faea7e00700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 0x7faeb019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faeb5600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 0x7faeb019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 -- 172.21.15.52:0/2096183584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 msgr2=0x7faeb019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 0x7faeb019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 -- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 msgr2=0x7faeb019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 0x7faeb019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 -- 172.21.15.52:0/2096183584 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb01a0350 con 0x7faeb0075590 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.738+0000 7faea7e00700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 0x7faeb019b880 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.739+0000 7faeae000700 1 -- 172.21.15.52:0/2096183584 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0027070 con 0x7faeb0075590 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.739+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb01a05e0 con 0x7faeb0075590 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.739+0000 7faeae000700 1 -- 172.21.15.52:0/2096183584 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea0022470 con 0x7faeb0075590 2024-04-26T01:45:40.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.739+0000 7faeae000700 1 -- 172.21.15.52:0/2096183584 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0017650 con 0x7faeb0075590 2024-04-26T01:45:40.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.739+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb01a7ec0 con 0x7faeb0075590 2024-04-26T01:45:40.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.743+0000 7faeae000700 1 -- 172.21.15.52:0/2096183584 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7faea002a070 con 0x7faeb0075590 2024-04-26T01:45:40.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 Processor -- start 2024-04-26T01:45:40.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 -- start start 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faeae000700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 0x7fae900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 0x556724e969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724e97330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556724de8e40 con 0x556724e9c780 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556724d8f6e0 con 0x556724e9d0a0 2024-04-26T01:45:40.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.744+0000 7faec2034b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556724de4e00 con 0x556724e96ef0 2024-04-26T01:45:40.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeae000700 1 -- 172.21.15.52:0/2096183584 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7faea00d16c0 con 0x7faeb0075590 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb4c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb5600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 0x7fae900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 0x556724e969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb4c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48922/0 (socket says 172.21.15.52:48922) 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb4c00700 1 -- 172.21.15.52:0/2990827666 learned_addr learned my addr 172.21.15.52:0/2990827666 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb5600700 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724e97330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 -- 172.21.15.52:0/2990827666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 msgr2=0x556724e97330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724e97330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 -- 172.21.15.52:0/2990827666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 msgr2=0x556724e9cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faea7e00700 1 -- 172.21.15.52:0/2990827666 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556724d85630 con 0x556724e9d0a0 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.745+0000 7faeb5600700 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724e97330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:40.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.746+0000 7faea7e00700 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 0x556724e969b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faea00ced20 tx=0x7faea009d1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebffe8622bfce2b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.746+0000 7faeb5600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 0x7fae900a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faea8009ec0 tx=0x7faea8009920 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.746+0000 7faea6000700 1 -- 172.21.15.52:0/2990827666 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea009d670 con 0x556724e9d0a0 2024-04-26T01:45:40.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.746+0000 7faea6000700 1 -- 172.21.15.52:0/2990827666 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea00a1a20 con 0x556724e9d0a0 2024-04-26T01:45:40.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.746+0000 7faea6000700 1 -- 172.21.15.52:0/2990827666 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0061aa0 con 0x556724e9d0a0 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 -- 172.21.15.52:0/2990827666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 msgr2=0x556724e969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 0x556724e969b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faea00ced20 tx=0x7faea009d1b0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 -- 172.21.15.52:0/2990827666 shutdown_connections 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724e97330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 --2- 172.21.15.52:0/2990827666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e9d0a0 0x556724e969b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 -- 172.21.15.52:0/2990827666 >> 172.21.15.52:0/2990827666 conn(0x556724dc4f00 msgr2=0x556724e918a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 -- 172.21.15.52:0/2990827666 shutdown_connections 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.747+0000 7faec2034b80 1 -- 172.21.15.52:0/2990827666 wait complete. 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.748+0000 7faec2034b80 1 -- start start 2024-04-26T01:45:40.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.748+0000 7faec2034b80 1 Processor -- start 2024-04-26T01:45:40.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724ea2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 0x556724ea0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556724ea36d0 con 0x556724e9c780 2024-04-26T01:45:40.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556724da2810 con 0x556724ea04e0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556724ea3850 con 0x556724e96ef0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faec2034b80 10 client.0 _ll_register_callbacks cb 0x556724d816d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48932/0 (socket says 172.21.15.52:48932) 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faea7e00700 1 -- 172.21.15.52:0/4098498254 learned_addr learned my addr 172.21.15.52:0/4098498254 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faeb4c00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724ea2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.749+0000 7faeb5600700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 0x556724ea0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 -- 172.21.15.52:0/4098498254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 msgr2=0x556724ea0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 0x556724ea0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 -- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 msgr2=0x556724e9ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 -- 172.21.15.52:0/4098498254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556724d36880 con 0x556724e96ef0 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb5600700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 0x556724ea0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faea7e00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faeb4c00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724ea2c50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae98012810 tx=0x7fae98012b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccc13d68124b5a94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7fae9ea00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9800f480 con 0x556724e96ef0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fae9800f620 con 0x556724e96ef0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.750+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faea00a1a20 con 0x556724e96ef0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae9800f820 con 0x556724e96ef0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7faec2034b80 10 client.25228 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556724d7faa0 con 0x556724e96ef0 2024-04-26T01:45:40.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea00a1a20 con 0x556724e96ef0 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fae980109e0 con 0x556724e96ef0 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7faec2034b80 10 client.25228 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7faec2034b80 10 client.25228 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.751+0000 7faec2034b80 20 client.25228 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724eb07a0 0x556724eb1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7faec2034b80 4 client.25228 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556724df6630 con 0x556724eb07a0 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7faea7e00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724eb07a0 0x556724eb1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7faea7e00700 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724eb07a0 0x556724eb1c10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3eb12a4347d7859e server_cookie=6db5242fe7e2bea7 in_seq=0 out_seq=0 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.752+0000 7fae9ea00700 10 client.25228 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:45:40.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.753+0000 7fae9ea00700 1 -- 172.21.15.52:0/4098498254 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556724df6630 con 0x556724eb07a0 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.753+0000 7fae9ea00700 10 client.25228 handle_command_reply: tid=0 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.753+0000 7faec2034b80 1 client.25228 shutdown 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 msgr2=0x556724ea2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724ea2c50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae98012810 tx=0x7fae98012b20 comp rx=0 tx=0).stop 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 shutdown_connections 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724eb07a0 0x556724eb1c10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724ea04e0 0x556724ea0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724e96ef0 0x556724ea2c50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 --2- 172.21.15.52:0/4098498254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e9ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 >> 172.21.15.52:0/4098498254 conn(0x556724dc4f00 msgr2=0x556724e90e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7fae9ea00700 0 client.25228 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 shutdown_connections 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 1 -- 172.21.15.52:0/4098498254 wait complete. 2024-04-26T01:45:40.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.754+0000 7faec2034b80 20 client.25228 trim_cache size 0 max 16384 2024-04-26T01:45:40.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.776+0000 7faec2034b80 1 Processor -- start 2024-04-26T01:45:40.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 -- start start 2024-04-26T01:45:40.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724e9dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556724eaff50 con 0x556724e9c780 2024-04-26T01:45:40.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556724eb0620 con 0x556724e96ef0 2024-04-26T01:45:40.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faec2034b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556724eb0cf0 con 0x556724ea04e0 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48940/0 (socket says 172.21.15.52:48940) 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faea7e00700 1 -- 172.21.15.52:0/1994952575 learned_addr learned my addr 172.21.15.52:0/1994952575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faeb4c00700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.778+0000 7faeb5600700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724e9dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 -- 172.21.15.52:0/1994952575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 msgr2=0x556724e9dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724e9dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 -- 172.21.15.52:0/1994952575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 msgr2=0x556724e92ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 -- 172.21.15.52:0/1994952575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556724e97a30 con 0x556724e96ef0 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faea7e00700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faeb4c00700 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e969b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae98010ed0 tx=0x7fae9800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4b335c8705d1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faea5600700 1 -- 172.21.15.52:0/1994952575 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98029070 con 0x556724e96ef0 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faea5600700 1 -- 172.21.15.52:0/1994952575 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9800fe70 con 0x556724e96ef0 2024-04-26T01:45:40.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.779+0000 7faea5600700 1 -- 172.21.15.52:0/1994952575 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98010690 con 0x556724e96ef0 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 -- 172.21.15.52:0/1994952575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 msgr2=0x556724e969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e969b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae98010ed0 tx=0x7fae9800f730 comp rx=0 tx=0).stop 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 -- 172.21.15.52:0/1994952575 shutdown_connections 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724e9dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e969b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 --2- 172.21.15.52:0/1994952575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e92ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 -- 172.21.15.52:0/1994952575 >> 172.21.15.52:0/1994952575 conn(0x556724dc4f00 msgr2=0x556724e9d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 -- 172.21.15.52:0/1994952575 shutdown_connections 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.780+0000 7faec2034b80 1 -- 172.21.15.52:0/1994952575 wait complete. 2024-04-26T01:45:40.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 -- start start 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 Processor -- start 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e90cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724eb94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556724eb1a90 con 0x556724e9c780 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556724eb13c0 con 0x556724e96ef0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.781+0000 7faec2034b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556724eb2160 con 0x556724ea04e0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faec2034b80 10 client.0 _ll_register_callbacks cb 0x7fae98009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb5600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724eb94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e90cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb5600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724eb94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60592/0 (socket says 172.21.15.52:60592) 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb5600700 1 -- 172.21.15.52:0/3523232474 learned_addr learned my addr 172.21.15.52:0/3523232474 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faea7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48944/0 (socket says 172.21.15.52:48944) 2024-04-26T01:45:40.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 -- 172.21.15.52:0/3523232474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 msgr2=0x556724eb94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724eb94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 -- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 msgr2=0x556724e91230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faeb4c00700 1 -- 172.21.15.52:0/3523232474 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556724e91b10 con 0x556724e96ef0 2024-04-26T01:45:40.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.782+0000 7faea7e00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7faeb4c00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e90cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae98000c00 tx=0x7fae98005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b188e0465c57ab2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7fae9ea00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98029040 con 0x556724e96ef0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae9800fe70 con 0x556724e96ef0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fae98005040 con 0x556724e96ef0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fae98005240 con 0x556724e96ef0 2024-04-26T01:45:40.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7faec2034b80 10 client.25223 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.783+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556724f17f10 con 0x556724e96ef0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9800fe70 con 0x556724e96ef0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fae980048d0 con 0x556724e96ef0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 10 client.25223 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 10 client.25223 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 20 client.25223 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724f1c680 0x556724f1b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 4 client.25223 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "status"}) -- 0x556724df06d0 con 0x556724f1c680 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.784+0000 7faea7e00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724f1c680 0x556724f1b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7faea7e00700 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724f1c680 0x556724f1b390 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fbbc47481cea88d0 server_cookie=15dc6513996e8a36 in_seq=0 out_seq=0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7fae9ea00700 10 client.25223 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7fae9ea00700 1 -- 172.21.15.52:0/3523232474 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556724df06d0 con 0x556724f1c680 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7fae9ea00700 10 client.25223 handle_command_reply: tid=0 2024-04-26T01:45:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7faec2034b80 1 client.25223 shutdown 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.785+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 msgr2=0x556724e90cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e90cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae98000c00 tx=0x7fae98005d50 comp rx=0 tx=0).stop 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 shutdown_connections 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x556724f1c680 0x556724f1b390 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556724ea04e0 0x556724eb94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556724e96ef0 0x556724e90cf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 --2- 172.21.15.52:0/3523232474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556724e9c780 0x556724e91230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7fae9ea00700 0 client.25223 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 >> 172.21.15.52:0/3523232474 conn(0x556724dc4f00 msgr2=0x556724e9e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 shutdown_connections 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 1 -- 172.21.15.52:0/3523232474 wait complete. 2024-04-26T01:45:40.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.786+0000 7faec2034b80 20 client.25223 trim_cache size 0 max 16384 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 1, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24512, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 257.43068453699999, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 95, 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 297.77309968100002 2024-04-26T01:45:40.786 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 msgr2=0x7fae900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 0x7fae900a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faea8009ec0 tx=0x7faea8009920 comp rx=0 tx=0).stop 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 msgr2=0x7faeb019b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 0x7faeb019b880 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea000b670 comp rx=0 tx=0).stop 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 shutdown_connections 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fae900a3e40 0x7fae900a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faeb010f440 0x7faeb019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faeb0074c70 0x7faeb019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 --2- 172.21.15.52:0/2096183584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb0075590 0x7faeb019b880 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:40.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 >> 172.21.15.52:0/2096183584 conn(0x7faeb0079ba0 msgr2=0x7faeb00ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:40.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 shutdown_connections 2024-04-26T01:45:40.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:40.787+0000 7faeb7600700 1 -- 172.21.15.52:0/2096183584 wait complete. 2024-04-26T01:45:41.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:40 smithi052 ceph-mon[27851]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:41.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:40 smithi191 ceph-mon[32745]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:40 smithi183 ceph-mon[32752]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T01:45:41.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 -- 172.21.15.52:0/4052924134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3010102b50 msgr2=0x7f3010106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 --2- 172.21.15.52:0/4052924134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3010102b50 0x7f3010106b90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009860 tx=0x7f2ffc02a340 comp rx=0 tx=0).stop 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 -- 172.21.15.52:0/4052924134 shutdown_connections 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 --2- 172.21.15.52:0/4052924134 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f301010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 --2- 172.21.15.52:0/4052924134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3010102b50 0x7f3010106b90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 --2- 172.21.15.52:0/4052924134 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30101021a0 0x7f3010102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 -- 172.21.15.52:0/4052924134 >> 172.21.15.52:0/4052924134 conn(0x7f30100793e0 msgr2=0x7f30100797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 -- 172.21.15.52:0/4052924134 shutdown_connections 2024-04-26T01:45:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.223+0000 7f3018000640 1 -- 172.21.15.52:0/4052924134 wait complete. 2024-04-26T01:45:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.224+0000 7f3018000640 1 Processor -- start 2024-04-26T01:45:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 -- start start 2024-04-26T01:45:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30101121b0 con 0x7f3010074cd0 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3010112030 con 0x7f3010107270 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3018000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3010112330 con 0x7f30101021a0 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3015600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3014c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3015600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60608/0 (socket says 172.21.15.52:60608) 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3014c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:50032/0 (socket says 172.21.15.52:50032) 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3014c00640 1 -- 172.21.15.52:0/3943381393 learned_addr learned my addr 172.21.15.52:0/3943381393 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.225+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48958/0 (socket says 172.21.15.52:48958) 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 -- 172.21.15.52:0/3943381393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 msgr2=0x7f3010074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 -- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 msgr2=0x7f30101a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 -- 172.21.15.52:0/3943381393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30101a6f80 con 0x7f3010107270 2024-04-26T01:45:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3015600640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3016000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.226+0000 7f3014c00640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc000c00 tx=0x7f2ffc032d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.227+0000 7f300e000640 1 -- 172.21.15.52:0/3943381393 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc040020 con 0x7f3010107270 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.227+0000 7f300e000640 1 -- 172.21.15.52:0/3943381393 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ffc030e10 con 0x7f3010107270 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.227+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30101a7210 con 0x7f3010107270 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.227+0000 7f300e000640 1 -- 172.21.15.52:0/3943381393 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc037040 con 0x7f3010107270 2024-04-26T01:45:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.227+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30101a7700 con 0x7f3010107270 2024-04-26T01:45:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.230+0000 7f300e000640 1 -- 172.21.15.52:0/3943381393 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f2ffc043020 con 0x7f3010107270 2024-04-26T01:45:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.231+0000 7f301ae44740 1 Processor -- start 2024-04-26T01:45:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 -- start start 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f300e000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 0x7f2ff80a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88fbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ebf8854900 con 0x55ebf88ee820 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ebf87faef0 con 0x55ebf88f9a00 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f301ae44740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ebf88508c0 con 0x55ebf88ef140 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.232+0000 7f300e000640 1 -- 172.21.15.52:0/3943381393 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f2ffc0f04d0 con 0x7f3010107270 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3014c00640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 0x7f2ff80a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48972/0 (socket says 172.21.15.52:48972) 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 -- 172.21.15.52:0/1424784645 learned_addr learned my addr 172.21.15.52:0/1424784645 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3015600640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88fbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 -- 172.21.15.52:0/1424784645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 msgr2=0x55ebf88f94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88f94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 -- 172.21.15.52:0/1424784645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 msgr2=0x55ebf88fbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88fbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 -- 172.21.15.52:0/1424784645 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf8567210 con 0x55ebf88ee820 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3015600640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3014c00640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 0x7f2ff80a6350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0edca0 tx=0x7f2ffc005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.233+0000 7f3016000640 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3008009fd0 tx=0x7f3008009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbce567ddfea4841 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.234+0000 7f3002600640 1 -- 172.21.15.52:0/1424784645 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30080118a0 con 0x55ebf88ee820 2024-04-26T01:45:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.234+0000 7f3002600640 1 -- 172.21.15.52:0/1424784645 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300801f3c0 con 0x55ebf88ee820 2024-04-26T01:45:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.234+0000 7f3002600640 1 -- 172.21.15.52:0/1424784645 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300800e610 con 0x55ebf88ee820 2024-04-26T01:45:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 -- 172.21.15.52:0/1424784645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 msgr2=0x55ebf88eec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3008009fd0 tx=0x7f3008009b40 comp rx=0 tx=0).stop 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 -- 172.21.15.52:0/1424784645 shutdown_connections 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 --2- 172.21.15.52:0/1424784645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88eec00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 -- 172.21.15.52:0/1424784645 >> 172.21.15.52:0/1424784645 conn(0x55ebf88ead30 msgr2=0x55ebf88eb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 -- 172.21.15.52:0/1424784645 shutdown_connections 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.235+0000 7f301ae44740 1 -- 172.21.15.52:0/1424784645 wait complete. 2024-04-26T01:45:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.236+0000 7f301ae44740 1 -- start start 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.236+0000 7f301ae44740 1 Processor -- start 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ebf88fc630 con 0x55ebf88ef140 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ebf880e310 con 0x55ebf88f9a00 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ebf88fc7b0 con 0x55ebf88ee820 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f301ae44740 10 client.0 _ll_register_callbacks cb 0x55ebf87dccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f3016000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f3016000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60640/0 (socket says 172.21.15.52:60640) 2024-04-26T01:45:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f3016000640 1 -- 172.21.15.52:0/4221382550 learned_addr learned my addr 172.21.15.52:0/4221382550 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.237+0000 7f3016000640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3015600640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 -- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 msgr2=0x55ebf88f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 -- 172.21.15.52:0/4221382550 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 msgr2=0x55ebf88f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 -- 172.21.15.52:0/4221382550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf8633620 con 0x55ebf88ee820 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f3016000640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3008009d20 tx=0x7f3008018040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57097b2702c3ee26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f2ff6a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008031060 con 0x55ebf88ee820 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300801f3c0 con 0x55ebf88ee820 2024-04-26T01:45:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.238+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f3008011aa0 con 0x55ebf88ee820 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3008025040 con 0x55ebf88ee820 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 10 client.25246 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ebf8908920 con 0x55ebf88ee820 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f300801f3c0 con 0x55ebf88ee820 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f300802b030 con 0x55ebf88ee820 2024-04-26T01:45:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 10 client.25246 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 10 client.25246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 20 client.25246 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8909d30 0x55ebf890c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 4 client.25246 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.239+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebf88688b0 con 0x55ebf8909d30 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.240+0000 7f3015600640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8909d30 0x55ebf890c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.240+0000 7f3015600640 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8909d30 0x55ebf890c120 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acbeeb8ea1bcc113 server_cookie=681ea29f46141201 in_seq=0 out_seq=0 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.240+0000 7f2ff6a00640 10 client.25246 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.241+0000 7f2ff6a00640 1 -- 172.21.15.52:0/4221382550 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebf88688b0 con 0x55ebf8909d30 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.241+0000 7f2ff6a00640 10 client.25246 handle_command_reply: tid=0 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.241+0000 7f301ae44740 1 client.25246 shutdown 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.241+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 msgr2=0x55ebf88f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.241+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3008009d20 tx=0x7f3008018040 comp rx=0 tx=0).stop 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 shutdown_connections 2024-04-26T01:45:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8909d30 0x55ebf890c120 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf88f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 --2- 172.21.15.52:0/4221382550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ee820 0x55ebf88f6a20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 >> 172.21.15.52:0/4221382550 conn(0x55ebf88ead30 msgr2=0x55ebf88f0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f2ff6a00640 0 client.25246 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 shutdown_connections 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 1 -- 172.21.15.52:0/4221382550 wait complete. 2024-04-26T01:45:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.242+0000 7f301ae44740 20 client.25246 trim_cache size 0 max 16384 2024-04-26T01:45:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 Processor -- start 2024-04-26T01:45:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 -- start start 2024-04-26T01:45:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88ee820 0x55ebf89054f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 0x55ebf8911840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30101b2580 con 0x55ebf88ef140 2024-04-26T01:45:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f30101afe50 con 0x55ebf88ee820 2024-04-26T01:45:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f301ae44740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30101adca0 con 0x55ebf88f9a00 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f3015600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f3015600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48980/0 (socket says 172.21.15.52:48980) 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.259+0000 7f3015600640 1 -- 172.21.15.52:0/3150704539 learned_addr learned my addr 172.21.15.52:0/3150704539 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 0x55ebf8911840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88ee820 0x55ebf89054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 -- 172.21.15.52:0/3150704539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88ee820 msgr2=0x55ebf89054f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88ee820 0x55ebf89054f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 -- 172.21.15.52:0/3150704539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 msgr2=0x55ebf88f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 -- 172.21.15.52:0/3150704539 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf88eef70 con 0x55ebf88f9a00 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3016000640 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 0x55ebf8911840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3008001eb0 tx=0x7f3008015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae227d987c401764 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3001c00640 1 -- 172.21.15.52:0/3150704539 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008031040 con 0x55ebf88f9a00 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3001c00640 1 -- 172.21.15.52:0/3150704539 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30080054f0 con 0x55ebf88f9a00 2024-04-26T01:45:42.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.260+0000 7f3001c00640 1 -- 172.21.15.52:0/3150704539 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008035420 con 0x55ebf88f9a00 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 -- 172.21.15.52:0/3150704539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 msgr2=0x55ebf8911840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 0x55ebf8911840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3008001eb0 tx=0x7f3008015040 comp rx=0 tx=0).stop 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 -- 172.21.15.52:0/3150704539 shutdown_connections 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88f9a00 0x55ebf8911840 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ef140 0x55ebf88f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 --2- 172.21.15.52:0/3150704539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88ee820 0x55ebf89054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.261+0000 7f301ae44740 1 -- 172.21.15.52:0/3150704539 >> 172.21.15.52:0/3150704539 conn(0x55ebf88ead30 msgr2=0x55ebf88eb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.262+0000 7f301ae44740 1 -- 172.21.15.52:0/3150704539 shutdown_connections 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.262+0000 7f301ae44740 1 -- 172.21.15.52:0/3150704539 wait complete. 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.262+0000 7f301ae44740 1 -- start start 2024-04-26T01:45:42.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.262+0000 7f301ae44740 1 Processor -- start 2024-04-26T01:45:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88fed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88ff280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf89680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30101ae310 con 0x55ebf88ee820 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f30101af780 con 0x55ebf88f9a00 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30101ae9e0 con 0x55ebf88ef140 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f301ae44740 10 client.0 _ll_register_callbacks cb 0x7f3008013a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f3015600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88ff280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f3016000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88fed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48992/0 (socket says 172.21.15.52:48992) 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.263+0000 7f3016000640 1 -- 172.21.15.52:0/946450234 learned_addr learned my addr 172.21.15.52:0/946450234 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3016000640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf89680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 -- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 msgr2=0x55ebf89680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf89680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 -- 172.21.15.52:0/946450234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 msgr2=0x55ebf88fed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88fed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 -- 172.21.15.52:0/946450234 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf8968730 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3016000640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf89680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f3015600640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88ff280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3004000f80 tx=0x7f3004003d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9f8559fd59e797 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f2ff6a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004004d20 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f3004004490 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.264+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30080054f0 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.265+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f300402e050 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.265+0000 7f301ae44740 10 client.25252 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.265+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ebf896fb50 con 0x55ebf88ef140 2024-04-26T01:45:42.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.265+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30080054f0 con 0x55ebf88ef140 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.265+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f300400b040 con 0x55ebf88ef140 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 10 client.25252 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 10 client.25252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 20 client.25252 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8970280 0x55ebf89726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 4 client.25252 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebf8862320 con 0x55ebf8970280 2024-04-26T01:45:42.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.266+0000 7f3016000640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8970280 0x55ebf89726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.267+0000 7f3016000640 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8970280 0x55ebf89726a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb545fc6d37d942 server_cookie=775b3f41b1c0d1c1 in_seq=0 out_seq=0 2024-04-26T01:45:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.267+0000 7f2ff6a00640 10 client.25252 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f2ff6a00640 1 -- 172.21.15.52:0/946450234 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ebf8862320 con 0x55ebf8970280 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f2ff6a00640 10 client.25252 handle_command_reply: tid=0 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 client.25252 shutdown 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 msgr2=0x55ebf88ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88ff280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3004000f80 tx=0x7f3004003d40 comp rx=0 tx=0).stop 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 shutdown_connections 2024-04-26T01:45:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ebf8970280 0x55ebf89726a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ebf88f9a00 0x55ebf89680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f2ff6a00640 0 client.25252 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebf88ef140 0x55ebf88ff280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 --2- 172.21.15.52:0/946450234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ebf88ee820 0x55ebf88fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 >> 172.21.15.52:0/946450234 conn(0x55ebf88ead30 msgr2=0x55ebf88e9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.276+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 shutdown_connections 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f301ae44740 1 -- 172.21.15.52:0/946450234 wait complete. 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f301ae44740 20 client.25252 trim_cache size 0 max 16384 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:45:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 msgr2=0x7f2ff80a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 0x7f2ff80a6350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0edca0 tx=0x7f2ffc005f90 comp rx=0 tx=0).stop 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 msgr2=0x7f3010074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.277+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc000c00 tx=0x7f2ffc032d40 comp rx=0 tx=0).stop 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 shutdown_connections 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ff80a3e90 0x7f2ff80a6350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3010074cd0 0x7f30101a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3010107270 0x7f3010074790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 --2- 172.21.15.52:0/3943381393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30101021a0 0x7f3010074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 >> 172.21.15.52:0/3943381393 conn(0x7f30100793e0 msgr2=0x7f30100ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 shutdown_connections 2024-04-26T01:45:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:42.278+0000 7f3018000640 1 -- 172.21.15.52:0/3943381393 wait complete. 2024-04-26T01:45:42.335 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:45:43.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:42 smithi052 ceph-mon[27851]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:43.226 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:42 smithi183 ceph-mon[32752]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:43.394 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:42 smithi191 ceph-mon[32745]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T01:45:44.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 -- 172.21.15.52:0/1001334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174113c10 msgr2=0x7f4174116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 --2- 172.21.15.52:0/1001334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174113c10 0x7f4174116000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f416c0098f0 tx=0x7f416c009c00 comp rx=0 tx=0).stop 2024-04-26T01:45:44.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 -- 172.21.15.52:0/1001334555 shutdown_connections 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 --2- 172.21.15.52:0/1001334555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174113c10 0x7f4174116000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 --2- 172.21.15.52:0/1001334555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 0x7f4174075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 --2- 172.21.15.52:0/1001334555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174075590 0x7f41741136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 -- 172.21.15.52:0/1001334555 >> 172.21.15.52:0/1001334555 conn(0x7f4174079ba0 msgr2=0x7f4174079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 -- 172.21.15.52:0/1001334555 shutdown_connections 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.740+0000 7f417ba00700 1 -- 172.21.15.52:0/1001334555 wait complete. 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 Processor -- start 2024-04-26T01:45:44.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 -- start start 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 0x7f41741a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174113c10 0x7f417419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4174119220 con 0x7f4174074c70 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f41741190a0 con 0x7f4174113c10 2024-04-26T01:45:44.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.742+0000 7f417ba00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41741193a0 con 0x7f4174075590 2024-04-26T01:45:44.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4179a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174113c10 0x7f417419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4179000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 0x7f41741a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56692/0 (socket says 172.21.15.52:56692) 2024-04-26T01:45:44.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 -- 172.21.15.52:0/1073187783 learned_addr learned my addr 172.21.15.52:0/1073187783 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 -- 172.21.15.52:0/1073187783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174113c10 msgr2=0x7f417419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174113c10 0x7f417419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 -- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 msgr2=0x7f41741a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 0x7f41741a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.743+0000 7f4173e00700 1 -- 172.21.15.52:0/1073187783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417419fd40 con 0x7f4174075590 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f4173e00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9a0 tx=0x7f416000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f4171600700 1 -- 172.21.15.52:0/1073187783 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416000ca10 con 0x7f4174075590 2024-04-26T01:45:44.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f4171600700 1 -- 172.21.15.52:0/1073187783 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416000cbb0 con 0x7f4174075590 2024-04-26T01:45:44.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f4171600700 1 -- 172.21.15.52:0/1073187783 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4160018680 con 0x7f4174075590 2024-04-26T01:45:44.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41741a0030 con 0x7f4174075590 2024-04-26T01:45:44.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.744+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41741ac2f0 con 0x7f4174075590 2024-04-26T01:45:44.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.748+0000 7f4171600700 1 -- 172.21.15.52:0/1073187783 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f4160010430 con 0x7f4174075590 2024-04-26T01:45:44.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.749+0000 7f4186299b80 1 Processor -- start 2024-04-26T01:45:44.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.749+0000 7f4171600700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 0x7f415c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 -- start start 2024-04-26T01:45:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4171600700 1 -- 172.21.15.52:0/1073187783 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f41600d0a70 con 0x7f4174075590 2024-04-26T01:45:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4179a00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 0x7f415c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796704570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796706ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55779664ce40 con 0x5577966f9c10 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5577965f36e0 con 0x557796704ab0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.750+0000 7f4186299b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557796648e00 con 0x5577966f92f0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56698/0 (socket says 172.21.15.52:56698) 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 -- 172.21.15.52:0/4056233346 learned_addr learned my addr 172.21.15.52:0/4056233346 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179a00700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796706ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179a00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 0x7f415c0a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f416c0098c0 tx=0x7f416c019040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4173e00700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796704570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 -- 172.21.15.52:0/4056233346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 msgr2=0x557796706ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796706ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 -- 172.21.15.52:0/4056233346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 msgr2=0x557796704570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796704570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179000700 1 -- 172.21.15.52:0/4056233346 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577965e9630 con 0x5577966f92f0 2024-04-26T01:45:44.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.751+0000 7f4179a00700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796706ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:45:44.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.752+0000 7f4173e00700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796704570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:44.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.752+0000 7f4179000700 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f41680093a0 tx=0x7f41680099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d09f8eb9f432812 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.752+0000 7f4166000700 1 -- 172.21.15.52:0/4056233346 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416801b030 con 0x5577966f92f0 2024-04-26T01:45:44.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.752+0000 7f4166000700 1 -- 172.21.15.52:0/4056233346 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4168004bb0 con 0x5577966f92f0 2024-04-26T01:45:44.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.752+0000 7f4166000700 1 -- 172.21.15.52:0/4056233346 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168017e60 con 0x5577966f92f0 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 -- 172.21.15.52:0/4056233346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 msgr2=0x5577966f96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f41680093a0 tx=0x7f41680099e0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 -- 172.21.15.52:0/4056233346 shutdown_connections 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796706ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x5577966f96d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 --2- 172.21.15.52:0/4056233346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796704570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.753+0000 7f4186299b80 1 -- 172.21.15.52:0/4056233346 >> 172.21.15.52:0/4056233346 conn(0x557796628550 msgr2=0x5577966f5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.754+0000 7f4186299b80 1 -- 172.21.15.52:0/4056233346 shutdown_connections 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.754+0000 7f4186299b80 1 -- 172.21.15.52:0/4056233346 wait complete. 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.754+0000 7f4186299b80 1 -- start start 2024-04-26T01:45:44.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 Processor -- start 2024-04-26T01:45:44.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796704190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796701490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577967019d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5577967075a0 con 0x5577966f9c10 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557796606810 con 0x5577966f92f0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557796707720 con 0x557796704ab0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4186299b80 10 client.0 _ll_register_callbacks cb 0x5577965e56d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.755+0000 7f4173e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796701490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796704190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4173e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796701490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42982/0 (socket says 172.21.15.52:42982) 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4173e00700 1 -- 172.21.15.52:0/3702461923 learned_addr learned my addr 172.21.15.52:0/3702461923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179a00700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577967019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 -- 172.21.15.52:0/3702461923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 msgr2=0x5577967019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577967019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 -- 172.21.15.52:0/3702461923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 msgr2=0x557796701490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796701490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 -- 172.21.15.52:0/3702461923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55779659a880 con 0x5577966f92f0 2024-04-26T01:45:44.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f4179000700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796704190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f416800e1c0 tx=0x7f4168017710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b2b75a7b7c66f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.756+0000 7f415aa00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:45:44.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168020880 con 0x5577966f92f0 2024-04-26T01:45:44.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f4168015410 con 0x5577966f92f0 2024-04-26T01:45:44.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4168004bb0 con 0x5577966f92f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4168017ab0 con 0x5577966f92f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f4186299b80 10 client.25250 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5577965e3aa0 con 0x5577966f92f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.757+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4168004bb0 con 0x5577966f92f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f416802c780 con 0x5577966f92f0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 10 client.25250 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 10 client.25250 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 20 client.25250 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557796713d70 0x5577967151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 4 client.25250 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55779665a630 con 0x557796713d70 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.758+0000 7f4173e00700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557796713d70 0x5577967151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.759+0000 7f4173e00700 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557796713d70 0x5577967151e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6946d8ebaf7ceace server_cookie=3f01dc242a3113fb in_seq=0 out_seq=0 2024-04-26T01:45:44.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.759+0000 7f415aa00700 10 client.25250 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f415aa00700 1 -- 172.21.15.52:0/3702461923 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55779665a630 con 0x557796713d70 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f415aa00700 10 client.25250 handle_command_reply: tid=0 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 client.25250 shutdown 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 msgr2=0x557796704190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796704190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f416800e1c0 tx=0x7f4168017710 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 shutdown_connections 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557796713d70 0x5577967151e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577967019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796704190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 --2- 172.21.15.52:0/3702461923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796701490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 >> 172.21.15.52:0/3702461923 conn(0x557796628550 msgr2=0x5577966fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.760+0000 7f415aa00700 0 client.25250 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.761+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 shutdown_connections 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.761+0000 7f4186299b80 1 -- 172.21.15.52:0/3702461923 wait complete. 2024-04-26T01:45:44.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.761+0000 7f4186299b80 20 client.25250 trim_cache size 0 max 16384 2024-04-26T01:45:44.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 Processor -- start 2024-04-26T01:45:44.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 -- start start 2024-04-26T01:45:44.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796706e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577966f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557796713520 con 0x5577966f9c10 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557796713bf0 con 0x5577966f92f0 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.784+0000 7f4186299b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5577967142c0 con 0x557796704ab0 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48782/0 (socket says 172.21.15.52:48782) 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577966f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4173e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796706e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 -- 172.21.15.52:0/2000593116 learned_addr learned my addr 172.21.15.52:0/2000593116 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 -- 172.21.15.52:0/2000593116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 msgr2=0x5577966f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577966f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 -- 172.21.15.52:0/2000593116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 msgr2=0x557796706e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796706e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.785+0000 7f4179000700 1 -- 172.21.15.52:0/2000593116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557796713a10 con 0x5577966f92f0 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.786+0000 7f4179000700 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4168005bd0 tx=0x7f4168008f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad99efbef430c979 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.786+0000 7f4165600700 1 -- 172.21.15.52:0/2000593116 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416801b040 con 0x5577966f92f0 2024-04-26T01:45:44.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.786+0000 7f4165600700 1 -- 172.21.15.52:0/2000593116 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416802c930 con 0x5577966f92f0 2024-04-26T01:45:44.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.786+0000 7f4165600700 1 -- 172.21.15.52:0/2000593116 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416801f570 con 0x5577966f92f0 2024-04-26T01:45:44.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 -- 172.21.15.52:0/2000593116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 msgr2=0x557796713d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4168005bd0 tx=0x7f4168008f70 comp rx=0 tx=0).stop 2024-04-26T01:45:44.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 -- 172.21.15.52:0/2000593116 shutdown_connections 2024-04-26T01:45:44.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557796704ab0 0x5577966f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5577966f92f0 0x557796713d70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 --2- 172.21.15.52:0/2000593116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796706e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 -- 172.21.15.52:0/2000593116 >> 172.21.15.52:0/2000593116 conn(0x557796628550 msgr2=0x55779670f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 -- 172.21.15.52:0/2000593116 shutdown_connections 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.787+0000 7f4186299b80 1 -- 172.21.15.52:0/2000593116 wait complete. 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 -- start start 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 Processor -- start 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796718650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557796715060 con 0x5577966f9c10 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557796714990 con 0x557796704ab0 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557796715730 con 0x5577966f92f0 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.788+0000 7f4186299b80 10 client.0 _ll_register_callbacks cb 0x5577965e56d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4179000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4179a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796718650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43010/0 (socket says 172.21.15.52:43010) 2024-04-26T01:45:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4179000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56718/0 (socket says 172.21.15.52:56718) 2024-04-26T01:45:44.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 -- 172.21.15.52:0/1300547967 learned_addr learned my addr 172.21.15.52:0/1300547967 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 -- 172.21.15.52:0/1300547967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 msgr2=0x55779671ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 -- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 msgr2=0x557796718650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796718650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4173e00700 1 -- 172.21.15.52:0/1300547967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55779671c6d0 con 0x5577966f9c10 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.789+0000 7f4179000700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:44.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f4173e00700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4160099240 tx=0x7f4160087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db1d208b3e1c220 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f415aa00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416009c870 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f41600ce380 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f416802c930 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f41600a0c20 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f4186299b80 10 client.15156 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55779677b080 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.790+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416802c930 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f41600a5050 con 0x5577966f9c10 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 10 client.15156 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 10 client.15156 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 20 client.15156 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55779677fbd0 0x55779677e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 4 client.15156 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T01:45:44.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "status"}) -- 0x5577966546d0 con 0x55779677fbd0 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.791+0000 7f4179000700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55779677fbd0 0x55779677e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4179000700 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55779677fbd0 0x55779677e8e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9972ce811ade5ee1 server_cookie=86e71c9fc2763a86 in_seq=0 out_seq=0 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f415aa00700 10 client.15156 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f415aa00700 1 -- 172.21.15.52:0/1300547967 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5577966546d0 con 0x55779677fbd0 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f415aa00700 10 client.15156 handle_command_reply: tid=0 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 client.15156 shutdown 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 msgr2=0x557796717f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4160099240 tx=0x7f4160087040 comp rx=0 tx=0).stop 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 shutdown_connections 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55779677fbd0 0x55779677e8e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557796704ab0 0x557796718650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5577966f92f0 0x55779671ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 --2- 172.21.15.52:0/1300547967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5577966f9c10 0x557796717f10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 >> 172.21.15.52:0/1300547967 conn(0x557796628550 msgr2=0x5577967031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.792+0000 7f415aa00700 0 client.15156 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:44.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 shutdown_connections 2024-04-26T01:45:44.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f4186299b80 1 -- 172.21.15.52:0/1300547967 wait complete. 2024-04-26T01:45:44.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f4186299b80 20 client.15156 trim_cache size 0 max 16384 2024-04-26T01:45:44.793 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "8b150f36-036c-11ef-bc93-c7b262605968", 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24524, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 277.49350303599999, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 25, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 95, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 95, 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 298.86403345600002 2024-04-26T01:45:44.794 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 msgr2=0x7f415c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 0x7f415c0a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f416c0098c0 tx=0x7f416c019040 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 msgr2=0x7f41741a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.793+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9a0 tx=0x7f416000ecb0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 shutdown_connections 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f415c0a3ec0 0x7f415c0a6380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4174113c10 0x7f417419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4174074c70 0x7f41741a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 --2- 172.21.15.52:0/1073187783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174075590 0x7f41741a5960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 >> 172.21.15.52:0/1073187783 conn(0x7f4174079ba0 msgr2=0x7f4174110ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 shutdown_connections 2024-04-26T01:45:44.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:45:44.794+0000 7f417ba00700 1 -- 172.21.15.52:0/1073187783 wait complete. 2024-04-26T01:45:45.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:44 smithi052 ceph-mon[27851]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:45.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:44 smithi191 ceph-mon[32745]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:44 smithi183 ceph-mon[32752]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:45.759 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-26T01:45:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:46 smithi052 ceph-mon[27851]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:47.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:46 smithi191 ceph-mon[32745]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:47.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:46 smithi183 ceph-mon[32752]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T01:45:49.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:48 smithi052 ceph-mon[27851]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:49.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:48 smithi191 ceph-mon[32745]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:48 smithi183 ceph-mon[32752]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:51.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:50 smithi052 ceph-mon[27851]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:51.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:50 smithi191 ceph-mon[32745]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:50 smithi183 ceph-mon[32752]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:53.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:52 smithi052 ceph-mon[27851]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:53.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:52 smithi191 ceph-mon[32745]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:52 smithi183 ceph-mon[32752]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-26T01:45:55.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:54 smithi052 ceph-mon[27851]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:55.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:54 smithi191 ceph-mon[32745]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:55.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:54 smithi183 ceph-mon[32752]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 -- 172.21.15.52:0/2028338132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 msgr2=0x7f95fc1033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 --2- 172.21.15.52:0/2028338132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1033b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009860 tx=0x7f95ec02a3d0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 -- 172.21.15.52:0/2028338132 shutdown_connections 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 --2- 172.21.15.52:0/2028338132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc10c3a0 0x7f95fc10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 --2- 172.21.15.52:0/2028338132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc1038f0 0x7f95fc10bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 --2- 172.21.15.52:0/2028338132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1033b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 -- 172.21.15.52:0/2028338132 >> 172.21.15.52:0/2028338132 conn(0x7f95fc0fc770 msgr2=0x7f95fc0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 -- 172.21.15.52:0/2028338132 shutdown_connections 2024-04-26T01:45:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.438+0000 7f9600e00640 1 -- 172.21.15.52:0/2028338132 wait complete. 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 Processor -- start 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 -- start start 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 0x7f95fc1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc10c3a0 0x7f95fc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc112040 con 0x7f95fc10c3a0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc111ec0 con 0x7f95fc1038f0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.440+0000 7f9600e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1121c0 con 0x7f95fc102fd0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45964/0 (socket says 172.21.15.52:45964) 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 -- 172.21.15.52:0/1038342431 learned_addr learned my addr 172.21.15.52:0/1038342431 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95faa00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 0x7f95fc1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc10c3a0 0x7f95fc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 -- 172.21.15.52:0/1038342431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 msgr2=0x7f95fc1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 0x7f95fc1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 -- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc10c3a0 msgr2=0x7f95fc19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc10c3a0 0x7f95fc19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95fb400640 1 -- 172.21.15.52:0/1038342431 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc19ada0 con 0x7f95fc102fd0 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.441+0000 7f95faa00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 0x7f95fc1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f95fb400640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009980 tx=0x7f95ec030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f95f3e00640 1 -- 172.21.15.52:0/1038342431 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec040070 con 0x7f95fc102fd0 2024-04-26T01:45:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95fc19b030 con 0x7f95fc102fd0 2024-04-26T01:45:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f95f3e00640 1 -- 172.21.15.52:0/1038342431 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ec005570 con 0x7f95fc102fd0 2024-04-26T01:45:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f95f3e00640 1 -- 172.21.15.52:0/1038342431 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0046c0 con 0x7f95fc102fd0 2024-04-26T01:45:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.442+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95fc1a72e0 con 0x7f95fc102fd0 2024-04-26T01:45:55.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.446+0000 7f9605224740 1 Processor -- start 2024-04-26T01:45:55.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.446+0000 7f9605224740 1 -- start start 2024-04-26T01:45:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 0x55dacf9afd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9ba650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dacf915900 con 0x55dacf9af9b0 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dacf8bbef0 con 0x55dacf9bab90 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f9605224740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dacf9118c0 con 0x55dacf9b02d0 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95f3e00640 1 -- 172.21.15.52:0/1038342431 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f95ec043070 con 0x7f95fc102fd0 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 0x55dacf9afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95faa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95faa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40904/0 (socket says 172.21.15.52:40904) 2024-04-26T01:45:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95faa00640 1 -- 172.21.15.52:0/611833888 learned_addr learned my addr 172.21.15.52:0/611833888 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95faa00640 1 -- 172.21.15.52:0/611833888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 msgr2=0x55dacf9ba650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.447+0000 7f95fb400640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9ba650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95faa00640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9ba650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95faa00640 1 -- 172.21.15.52:0/611833888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 msgr2=0x55dacf9afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95faa00640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 0x55dacf9afd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95faa00640 1 -- 172.21.15.52:0/611833888 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dacf628210 con 0x55dacf9bab90 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95fbe00640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 0x55dacf9afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95fb400640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9ba650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95faa00640 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95e80114f0 tx=0x7f95e80119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f4c3a3f2dd8cd9e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.448+0000 7f95f0c00640 1 -- 172.21.15.52:0/611833888 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801e070 con 0x55dacf9bab90 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f95f0c00640 1 -- 172.21.15.52:0/611833888 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e800fad0 con 0x55dacf9bab90 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f95f0c00640 1 -- 172.21.15.52:0/611833888 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8019a60 con 0x55dacf9bab90 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f95f3e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 0x7f95e00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f95f3e00640 1 -- 172.21.15.52:0/1038342431 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f95ec0f1490 con 0x7f95fc102fd0 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 0x7f95e00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f9605224740 1 -- 172.21.15.52:0/611833888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 msgr2=0x55dacf9bcf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f9605224740 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95e80114f0 tx=0x7f95e80119b0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f9605224740 1 -- 172.21.15.52:0/611833888 shutdown_connections 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.449+0000 7f9605224740 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9bcf80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9ba650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 --2- 172.21.15.52:0/611833888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9af9b0 0x55dacf9afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 -- 172.21.15.52:0/611833888 >> 172.21.15.52:0/611833888 conn(0x55dacf9abec0 msgr2=0x55dacf9ac2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 -- 172.21.15.52:0/611833888 shutdown_connections 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 -- 172.21.15.52:0/611833888 wait complete. 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 0x7f95e00a6260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009fd0 tx=0x7f95f4009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 -- start start 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.450+0000 7f9605224740 1 Processor -- start 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9b02d0 0x55dacf9b8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9b8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dacf9bd7c0 con 0x55dacf9bab90 2024-04-26T01:45:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dacf8cf310 con 0x55dacf9b02d0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dacf9bd940 con 0x55dacf9af9b0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f9605224740 10 client.0 _ll_register_callbacks cb 0x55dacf89dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9b8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f95fb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f95fb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9b7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45978/0 (socket says 172.21.15.52:45978) 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.451+0000 7f95fb400640 1 -- 172.21.15.52:0/1060009450 learned_addr learned my addr 172.21.15.52:0/1060009450 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95faa00640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9b02d0 0x55dacf9b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 -- 172.21.15.52:0/1060009450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 msgr2=0x55dacf9b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 -- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9b02d0 msgr2=0x55dacf9b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9b02d0 0x55dacf9b8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 -- 172.21.15.52:0/1060009450 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dacf6f4620 con 0x55dacf9bab90 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95fbe00640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9b8560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f95f4005fd0 tx=0x7f95f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df0819e99dbf074 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.452+0000 7f95e5600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4003c40 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e800fad0 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f95f4017030 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95f4003de0 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f9605224740 10 client.15168 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dacf9e3210 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.453+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e800fad0 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f95f401f8d0 con 0x55dacf9bab90 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 10 client.15168 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 10 client.15168 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 20 client.15168 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacf9e4720 0x55dacf9e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 4 client.15168 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:45:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dacf9298b0 con 0x55dacf9e4720 2024-04-26T01:45:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.454+0000 7f95fb400640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacf9e4720 0x55dacf9e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.455+0000 7f95fb400640 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacf9e4720 0x55dacf9e6b10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1918d1fef4a28c1b server_cookie=5f98b439465eedc7 in_seq=0 out_seq=0 2024-04-26T01:45:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.455+0000 7f95e5600640 10 client.15168 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f95e5600640 1 -- 172.21.15.52:0/1060009450 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dacf9298b0 con 0x55dacf9e4720 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f95e5600640 10 client.15168 handle_command_reply: tid=0 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 client.15168 shutdown 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 msgr2=0x55dacf9b8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9b8560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f95f4005fd0 tx=0x7f95f4004ab0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 shutdown_connections 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacf9e4720 0x55dacf9e6b10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9b8560 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9b02d0 0x55dacf9b8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 --2- 172.21.15.52:0/1060009450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 >> 172.21.15.52:0/1060009450 conn(0x55dacf9abec0 msgr2=0x55dacf9b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f95e5600640 0 client.15168 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 shutdown_connections 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.456+0000 7f9605224740 1 -- 172.21.15.52:0/1060009450 wait complete. 2024-04-26T01:45:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.457+0000 7f9605224740 20 client.15168 trim_cache size 0 max 16384 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.474+0000 7f9605224740 1 Processor -- start 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.474+0000 7f9605224740 1 -- start start 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9d76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 0x55dacf9b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9ec420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc072410 con 0x55dacf9b02d0 2024-04-26T01:45:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc06fce0 con 0x55dacf9bab90 2024-04-26T01:45:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f9605224740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc06db90 con 0x55dacf9af9b0 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f95faa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 0x55dacf9b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f95fb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9d76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f95faa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 0x55dacf9b2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40500/0 (socket says 172.21.15.52:40500) 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f95faa00640 1 -- 172.21.15.52:0/470306120 learned_addr learned my addr 172.21.15.52:0/470306120 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.475+0000 7f95fbe00640 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 -- 172.21.15.52:0/470306120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 msgr2=0x55dacf9ec420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9ec420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 -- 172.21.15.52:0/470306120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 msgr2=0x55dacf9b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 0x55dacf9b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 -- 172.21.15.52:0/470306120 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dacf9a6490 con 0x55dacf9af9b0 2024-04-26T01:45:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95fb400640 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9d76d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0eeb80 tx=0x7f95ec0f38e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1b368960c63b06a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95e7e00640 1 -- 172.21.15.52:0/470306120 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec04a070 con 0x55dacf9af9b0 2024-04-26T01:45:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95e7e00640 1 -- 172.21.15.52:0/470306120 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ec0c16d0 con 0x55dacf9af9b0 2024-04-26T01:45:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.476+0000 7f95e7e00640 1 -- 172.21.15.52:0/470306120 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec031b80 con 0x55dacf9af9b0 2024-04-26T01:45:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 -- 172.21.15.52:0/470306120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 msgr2=0x55dacf9d76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9d76d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0eeb80 tx=0x7f95ec0f38e0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 -- 172.21.15.52:0/470306120 shutdown_connections 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9bab90 0x55dacf9ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9b02d0 0x55dacf9b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 --2- 172.21.15.52:0/470306120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9af9b0 0x55dacf9d76d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 -- 172.21.15.52:0/470306120 >> 172.21.15.52:0/470306120 conn(0x55dacf9abec0 msgr2=0x55dacf9b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 -- 172.21.15.52:0/470306120 shutdown_connections 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.477+0000 7f9605224740 1 -- 172.21.15.52:0/470306120 wait complete. 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.478+0000 7f9605224740 1 -- start start 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.478+0000 7f9605224740 1 Processor -- start 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9d3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc06e1e0 con 0x55dacf9bab90 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc06f610 con 0x55dacf9af9b0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc06e870 con 0x55dacf9b02d0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f9605224740 10 client.0 _ll_register_callbacks cb 0x7f95f400f540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95fb400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95faa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9d3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40516/0 (socket says 172.21.15.52:40516) 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95fb400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40916/0 (socket says 172.21.15.52:40916) 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95fbe00640 1 -- 172.21.15.52:0/3774565256 learned_addr learned my addr 172.21.15.52:0/3774565256 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:45:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.479+0000 7f95faa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9d3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45990/0 (socket says 172.21.15.52:45990) 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 -- 172.21.15.52:0/3774565256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 msgr2=0x55dacf9d3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9d3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 -- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 msgr2=0x55dacf9d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 -- 172.21.15.52:0/3774565256 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dacf9d1650 con 0x55dacf9bab90 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fb400640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95fbe00640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95f401ff30 tx=0x7f95f401f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26efed7d07e2485f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95e5600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:45:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4035070 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f95f401f980 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.480+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ec0c16d0 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.481+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95f4041050 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.481+0000 7f9605224740 10 client.15174 fetch_fsmap learned FSMap version 25 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.481+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dacfa4a750 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.481+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ec0c16d0 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.481+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f95f4031450 con 0x55dacf9bab90 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 10 client.15174 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 10 client.15174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 20 client.15174 populate_metadata read hostname 'smithi052' 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacfa4ae50 0x55dacfa4d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 4 client.15174 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dacf923320 con 0x55dacfa4ae50 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.482+0000 7f95fb400640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacfa4ae50 0x55dacfa4d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.483+0000 7f95fb400640 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacfa4ae50 0x55dacfa4d270 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a40c3f89b0328b0 server_cookie=104cf50517a0b4d1 in_seq=0 out_seq=0 2024-04-26T01:45:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.483+0000 7f95e5600640 10 client.15174 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f95e5600640 1 -- 172.21.15.52:0/3774565256 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dacf923320 con 0x55dacfa4ae50 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f95e5600640 10 client.15174 handle_command_reply: tid=0 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 client.15174 shutdown 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 msgr2=0x55dacf9d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95f401ff30 tx=0x7f95f401f2d0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 shutdown_connections 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dacfa4ae50 0x55dacfa4d270 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dacf9bab90 0x55dacf9d0f60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dacf9b02d0 0x55dacf9d3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 --2- 172.21.15.52:0/3774565256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dacf9af9b0 0x55dacf9d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 >> 172.21.15.52:0/3774565256 conn(0x55dacf9abec0 msgr2=0x55dacf9b7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f95e5600640 0 client.15174 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 shutdown_connections 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 1 -- 172.21.15.52:0/3774565256 wait complete. 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.487+0000 7f9605224740 20 client.15174 trim_cache size 0 max 16384 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:45:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 msgr2=0x7f95e00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 0x7f95e00a6260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009fd0 tx=0x7f95f4009500 comp rx=0 tx=0).stop 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 msgr2=0x7f95fc1a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009980 tx=0x7f95ec030fa0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 shutdown_connections 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f95e00a3da0 0x7f95e00a6260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95fc10c3a0 0x7f95fc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95fc1038f0 0x7f95fc1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 --2- 172.21.15.52:0/1038342431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95fc102fd0 0x7f95fc1a0420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:45:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 >> 172.21.15.52:0/1038342431 conn(0x7f95fc0fc770 msgr2=0x7f95fc101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:45:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 shutdown_connections 2024-04-26T01:45:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:45:55.488+0000 7f9600e00640 1 -- 172.21.15.52:0/1038342431 wait complete. 2024-04-26T01:45:57.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:56 smithi191 ceph-mon[32745]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:57.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:56 smithi183 ceph-mon[32752]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:56 smithi052 ceph-mon[27851]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-26T01:45:59.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:45:58 smithi191 ceph-mon[32745]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:45:59.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:45:58 smithi183 ceph-mon[32752]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:45:59.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:45:58 smithi052 ceph-mon[27851]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-26T01:46:00.761 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.6 2024-04-26T01:46:00.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:00.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:46:00.555+0000 7f16bc000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:00.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:46:00.555+0000 7f16bc000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:00.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:46:00.555+0000 7f16bc000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:00.850 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:46:00.555+0000 7f16bc000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:00.850 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T01:46:00.555+0000 7f16bc000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T01:46:01.208 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:01.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:00 smithi052 ceph-mon[27851]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:01.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:00 smithi191 ceph-mon[32745]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:01.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:00 smithi183 ceph-mon[32752]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T01:46:03.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:02 smithi052 ceph-mon[27851]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-26T01:46:03.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:02 smithi052 ceph-mon[27851]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-04-26T01:46:03.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:02 smithi191 ceph-mon[32745]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-26T01:46:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:02 smithi191 ceph-mon[32745]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-04-26T01:46:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:02 smithi183 ceph-mon[32752]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-26T01:46:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:02 smithi183 ceph-mon[32752]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.481+0000 7f622be00700 1 -- 172.21.15.52:0/1653755738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 msgr2=0x7f622c10f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.481+0000 7f622be00700 1 --2- 172.21.15.52:0/1653755738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c10f840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6224009f90 tx=0x7f622400b330 comp rx=0 tx=0).stop 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 -- 172.21.15.52:0/1653755738 shutdown_connections 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 --2- 172.21.15.52:0/1653755738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c10f840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 --2- 172.21.15.52:0/1653755738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 0x7f622c108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 --2- 172.21.15.52:0/1653755738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 -- 172.21.15.52:0/1653755738 >> 172.21.15.52:0/1653755738 conn(0x7f622c0fe4c0 msgr2=0x7f622c1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 -- 172.21.15.52:0/1653755738 shutdown_connections 2024-04-26T01:46:03.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.482+0000 7f622be00700 1 -- 172.21.15.52:0/1653755738 wait complete. 2024-04-26T01:46:03.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.484+0000 7f622be00700 1 Processor -- start 2024-04-26T01:46:03.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.484+0000 7f622be00700 1 -- start start 2024-04-26T01:46:03.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.484+0000 7f622be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:03.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.484+0000 7f622be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c1a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:03.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 0x7f622c19b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:03.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622be00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f622c075390 con 0x7f622c102970 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622be00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f622c075210 con 0x7f622c103330 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622be00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f622c075510 con 0x7f622c108970 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40546/0 (socket says 172.21.15.52:40546) 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622aa00700 1 -- 172.21.15.52:0/3623495835 learned_addr learned my addr 172.21.15.52:0/3623495835 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622a000700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c1a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:03.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.485+0000 7f622b400700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 0x7f622c19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:03.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 -- 172.21.15.52:0/3623495835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 msgr2=0x7f622c19b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:03.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 0x7f622c19b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 -- 172.21.15.52:0/3623495835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 msgr2=0x7f622c1a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:03.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c1a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 -- 172.21.15.52:0/3623495835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622c19be40 con 0x7f622c102970 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.486+0000 7f622aa00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e9a0 tx=0x7f621c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.487+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c00cbd0 con 0x7f622c102970 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.487+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622c19c130 con 0x7f622c102970 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.487+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f621c00cd70 con 0x7f622c102970 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.487+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c018680 con 0x7f622c102970 2024-04-26T01:46:03.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.488+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622c1a7ec0 con 0x7f622c102970 2024-04-26T01:46:03.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.490+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f622c1040b0 con 0x7f622c102970 2024-04-26T01:46:03.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.492+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f621c010430 con 0x7f622c102970 2024-04-26T01:46:03.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.493+0000 7f6223400700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 0x7f62100a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:03.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.494+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f621c0d0de0 con 0x7f622c102970 2024-04-26T01:46:03.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.495+0000 7f622a000700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 0x7f62100a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:03.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.496+0000 7f622a000700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 0x7f62100a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f622c10f9b0 tx=0x7f6218009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:03.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.500+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f621c09d260 con 0x7f622c102970 2024-04-26T01:46:03.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.965+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f622c1a8090 con 0x7f622c102970 2024-04-26T01:46:03.966 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:46:03.966 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:44:59.710897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:46:03.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.968+0000 7f6223400700 1 -- 172.21.15.52:0/3623495835 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f621c09cb60 con 0x7f622c102970 2024-04-26T01:46:03.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.971+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 msgr2=0x7f62100a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:03.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.971+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 0x7f62100a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f622c10f9b0 tx=0x7f6218009310 comp rx=0 tx=0).stop 2024-04-26T01:46:03.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.971+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 msgr2=0x7f622c19fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:03.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.971+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e9a0 tx=0x7f621c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 shutdown_connections 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f62100a3f50 0x7f62100a6410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f622c108970 0x7f622c19b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f622c102970 0x7f622c19fdc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 --2- 172.21.15.52:0/3623495835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f622c103330 0x7f622c1a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 >> 172.21.15.52:0/3623495835 conn(0x7f622c0fe4c0 msgr2=0x7f622c075bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 shutdown_connections 2024-04-26T01:46:03.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:03.972+0000 7f622be00700 1 -- 172.21.15.52:0/3623495835 wait complete. 2024-04-26T01:46:03.971 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 25 2024-04-26T01:46:04.330 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:03 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3623495835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:03 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3623495835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:03 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3623495835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:04.816 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0' 2024-04-26T01:46:04.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 85.6 --await-for 42.0 2024-04-26T01:46:05.171 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:04 smithi052 ceph-mon[27851]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:05.384 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:05.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:04 smithi191 ceph-mon[32745]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:04 smithi183 ceph-mon[32752]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:07.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:07 smithi052 ceph-mon[27851]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:07 smithi191 ceph-mon[32745]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:07 smithi183 ceph-mon[32752]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-26T01:46:07.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 -- 172.21.15.52:0/1019146286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a8113730 msgr2=0x7f09a8115b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 --2- 172.21.15.52:0/1019146286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a8113730 0x7f09a8115b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009f90 tx=0x7f09a000b330 comp rx=0 tx=0).stop 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 -- 172.21.15.52:0/1019146286 shutdown_connections 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 --2- 172.21.15.52:0/1019146286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a8113730 0x7f09a8115b20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 --2- 172.21.15.52:0/1019146286 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09a8076a70 0x7f09a8074c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 --2- 172.21.15.52:0/1019146286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a80751b0 0x7f09a8075630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 -- 172.21.15.52:0/1019146286 >> 172.21.15.52:0/1019146286 conn(0x7f09a8079b20 msgr2=0x7f09a8079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 -- 172.21.15.52:0/1019146286 shutdown_connections 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.648+0000 7f09ad800700 1 -- 172.21.15.52:0/1019146286 wait complete. 2024-04-26T01:46:07.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.649+0000 7f09ad800700 1 Processor -- start 2024-04-26T01:46:07.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 -- start start 2024-04-26T01:46:07.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09a8076a70 0x7f09a819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a8113730 0x7f09a81a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09a8119230 con 0x7f09a80751b0 2024-04-26T01:46:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81190b0 con 0x7f09a8113730 2024-04-26T01:46:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.650+0000 7f09ad800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81193b0 con 0x7f09a8076a70 2024-04-26T01:46:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59716/0 (socket says 172.21.15.52:59716) 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 -- 172.21.15.52:0/565367059 learned_addr learned my addr 172.21.15.52:0/565367059 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 -- 172.21.15.52:0/565367059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09a8076a70 msgr2=0x7f09a819fb80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09a8076a70 0x7f09a819fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 -- 172.21.15.52:0/565367059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a8113730 msgr2=0x7f09a81a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a8113730 0x7f09a81a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.651+0000 7f09a6a00700 1 -- 172.21.15.52:0/565367059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a81a4600 con 0x7f09a80751b0 2024-04-26T01:46:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.652+0000 7f09a6a00700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9f0 tx=0x7f099800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.653+0000 7f099f400700 1 -- 172.21.15.52:0/565367059 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099800cbd0 con 0x7f09a80751b0 2024-04-26T01:46:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.653+0000 7f099f400700 1 -- 172.21.15.52:0/565367059 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f099800cd70 con 0x7f09a80751b0 2024-04-26T01:46:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.653+0000 7f099f400700 1 -- 172.21.15.52:0/565367059 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998010430 con 0x7f09a80751b0 2024-04-26T01:46:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.653+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81a48f0 con 0x7f09a80751b0 2024-04-26T01:46:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.653+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81ac170 con 0x7f09a80751b0 2024-04-26T01:46:07.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.656+0000 7f099f400700 1 -- 172.21.15.52:0/565367059 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f0998004750 con 0x7f09a80751b0 2024-04-26T01:46:07.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.658+0000 7f099f400700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 0x7f098c0a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.658+0000 7f099f400700 1 -- 172.21.15.52:0/565367059 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f09980d0ce0 con 0x7f09a80751b0 2024-04-26T01:46:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.659+0000 7f09b813db80 1 Processor -- start 2024-04-26T01:46:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.659+0000 7f09a6000700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 0x7f098c0a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.659+0000 7f09b813db80 1 -- start start 2024-04-26T01:46:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2fb180 0x55f3bd2fd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd243020 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd1e98c0 con 0x55f3bd2f02d0 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09b813db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd23efe0 con 0x55f3bd2fb180 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59718/0 (socket says 172.21.15.52:59718) 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.660+0000 7f09a6a00700 1 -- 172.21.15.52:0/3667163900 learned_addr learned my addr 172.21.15.52:0/3667163900 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.661+0000 7f09a7400700 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2fb180 0x55f3bd2fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.661+0000 7f09a6a00700 1 -- 172.21.15.52:0/3667163900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2fb180 msgr2=0x55f3bd2fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.661+0000 7f09a6a00700 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2fb180 0x55f3bd2fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.661+0000 7f09a6a00700 1 -- 172.21.15.52:0/3667163900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2f02d0 msgr2=0x55f3bd2fac40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:46:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.662+0000 7f09a6000700 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.662+0000 7f09a6a00700 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.662+0000 7f09a6a00700 1 -- 172.21.15.52:0/3667163900 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3bd1df810 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.662+0000 7f09a6a00700 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0998098f30 tx=0x7f099809c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c674b4b9963a862d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.663+0000 7f0993e00700 1 -- 172.21.15.52:0/3667163900 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a0e80 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.663+0000 7f0993e00700 1 -- 172.21.15.52:0/3667163900 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09980a5740 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.663+0000 7f0993e00700 1 -- 172.21.15.52:0/3667163900 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998099140 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.664+0000 7f09a6000700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 0x7f098c0a6530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09a807a240 tx=0x7f09940123f0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.664+0000 7f09b813db80 1 -- 172.21.15.52:0/3667163900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 msgr2=0x55f3bd2efd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.664+0000 7f09b813db80 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0998098f30 tx=0x7f099809c500 comp rx=0 tx=0).stop 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.664+0000 7f09b813db80 1 -- 172.21.15.52:0/3667163900 shutdown_connections 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.665+0000 7f09b813db80 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2fb180 0x55f3bd2fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.665+0000 7f09b813db80 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2efd90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.665+0000 7f09b813db80 1 --2- 172.21.15.52:0/3667163900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.665+0000 7f09b813db80 1 -- 172.21.15.52:0/3667163900 >> 172.21.15.52:0/3667163900 conn(0x55f3bd21e730 msgr2=0x55f3bd2ebe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.666+0000 7f09b813db80 1 -- 172.21.15.52:0/3667163900 shutdown_connections 2024-04-26T01:46:07.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.666+0000 7f09b813db80 1 -- 172.21.15.52:0/3667163900 wait complete. 2024-04-26T01:46:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.667+0000 7f09b813db80 1 -- start start 2024-04-26T01:46:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.667+0000 7f09b813db80 1 Processor -- start 2024-04-26T01:46:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.668+0000 7f09b813db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.668+0000 7f09b813db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.668+0000 7f09b813db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd2f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09b813db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd2fdca0 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09b813db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd1fc9f0 con 0x55f3bd2fb180 2024-04-26T01:46:07.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09b813db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f3bd2fde20 con 0x55f3bd2f02d0 2024-04-26T01:46:07.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59726/0 (socket says 172.21.15.52:59726) 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 -- 172.21.15.52:0/240108989 learned_addr learned my addr 172.21.15.52:0/240108989 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 -- 172.21.15.52:0/240108989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 msgr2=0x55f3bd2f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 -- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 msgr2=0x55f3bd2f8090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd2f8090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.669+0000 7f09a6a00700 1 -- 172.21.15.52:0/240108989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3bd190a60 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.670+0000 7f09a6a00700 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9c0 tx=0x7f099809cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b02960841872c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.670+0000 7f0990c00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:46:07.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.670+0000 7f0990c00700 1 -- 172.21.15.52:0/240108989 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a0e80 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.670+0000 7f0990c00700 1 -- 172.21.15.52:0/240108989 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0998016dd0 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.670+0000 7f0990c00700 10 client.0 inactive, discarding osd_map(95..95 src has 1..95) 2024-04-26T01:46:07.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.671+0000 7f09b813db80 10 client.0 _ll_register_callbacks cb 0x55f3bd1db8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:07.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.671+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09980a5740 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f0990c00700 1 -- 172.21.15.52:0/240108989 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0998080050 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 10 client.15198 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:07.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55f3bd1d9c80 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f0990c00700 1 -- 172.21.15.52:0/240108989 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f099809cd30 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 10 client.15198 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 10 client.15198 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 20 client.15198 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd30a390 0x55f3bd30c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 4 client.15198 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:07.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.672+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3bd250810 con 0x55f3bd30a390 2024-04-26T01:46:07.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.673+0000 7f09a6000700 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd30a390 0x55f3bd30c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.674+0000 7f09a6000700 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd30a390 0x55f3bd30c780 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42756d6f4d28941 server_cookie=77f85c10c4e91338 in_seq=0 out_seq=0 2024-04-26T01:46:07.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.674+0000 7f0990c00700 10 client.15198 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f0990c00700 1 -- 172.21.15.52:0/240108989 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3bd250810 con 0x55f3bd30a390 2024-04-26T01:46:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f0990c00700 10 client.15198 handle_command_reply: tid=0 2024-04-26T01:46:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 client.15198 shutdown 2024-04-26T01:46:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 msgr2=0x55f3bd2fa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9c0 tx=0x7f099809cc50 comp rx=0 tx=0).stop 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 shutdown_connections 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd30a390 0x55f3bd30c780 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd2f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2fa850 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 --2- 172.21.15.52:0/240108989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f0990c00700 0 client.15198 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.675+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 >> 172.21.15.52:0/240108989 conn(0x55f3bd21e730 msgr2=0x55f3bd2eba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.676+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 shutdown_connections 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.676+0000 7f09b813db80 1 -- 172.21.15.52:0/240108989 wait complete. 2024-04-26T01:46:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.676+0000 7f09b813db80 20 client.15198 trim_cache size 0 max 16384 2024-04-26T01:46:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.717+0000 7f09b813db80 1 Processor -- start 2024-04-26T01:46:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.717+0000 7f09b813db80 1 -- start start 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd306ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b6070 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b3900 con 0x55f3bd2fb180 2024-04-26T01:46:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.718+0000 7f09b813db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b1750 con 0x55f3bd2f02d0 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59742/0 (socket says 172.21.15.52:59742) 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a6a00700 1 -- 172.21.15.52:0/370225121 learned_addr learned my addr 172.21.15.52:0/370225121 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd306ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 -- 172.21.15.52:0/370225121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 msgr2=0x55f3bd2fd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 -- 172.21.15.52:0/370225121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 msgr2=0x55f3bd2f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a7400700 1 -- 172.21.15.52:0/370225121 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3bd2f06b0 con 0x55f3bd2fb180 2024-04-26T01:46:07.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.719+0000 7f09a6a00700 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:07.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.720+0000 7f09a7400700 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd306ba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009f90 tx=0x7f09a0009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72e46604166d48c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.720+0000 7f0993400700 1 -- 172.21.15.52:0/370225121 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0020070 con 0x55f3bd2fb180 2024-04-26T01:46:07.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.720+0000 7f0993400700 1 -- 172.21.15.52:0/370225121 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a00093c0 con 0x55f3bd2fb180 2024-04-26T01:46:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.720+0000 7f0993400700 1 -- 172.21.15.52:0/370225121 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0007a80 con 0x55f3bd2fb180 2024-04-26T01:46:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 -- 172.21.15.52:0/370225121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 msgr2=0x55f3bd306ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd306ba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009f90 tx=0x7f09a0009c50 comp rx=0 tx=0).stop 2024-04-26T01:46:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 -- 172.21.15.52:0/370225121 shutdown_connections 2024-04-26T01:46:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd306ba0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 --2- 172.21.15.52:0/370225121 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd2fd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.721+0000 7f09b813db80 1 -- 172.21.15.52:0/370225121 >> 172.21.15.52:0/370225121 conn(0x55f3bd21e730 msgr2=0x55f3bd2eaf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.722+0000 7f09b813db80 1 -- 172.21.15.52:0/370225121 shutdown_connections 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.722+0000 7f09b813db80 1 -- 172.21.15.52:0/370225121 wait complete. 2024-04-26T01:46:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 -- start start 2024-04-26T01:46:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 Processor -- start 2024-04-26T01:46:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd3003c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd3707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b1dc0 con 0x55f3bd2ef9b0 2024-04-26T01:46:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b3230 con 0x55f3bd2fb180 2024-04-26T01:46:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81b2490 con 0x55f3bd2f02d0 2024-04-26T01:46:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.723+0000 7f09b813db80 10 client.0 _ll_register_callbacks cb 0x55f3bd1db8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.724+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.724+0000 7f09a6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59750/0 (socket says 172.21.15.52:59750) 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.724+0000 7f09a6a00700 1 -- 172.21.15.52:0/2179762420 learned_addr learned my addr 172.21.15.52:0/2179762420 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.724+0000 7f09a7400700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd3707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.724+0000 7f09a6000700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd3003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a7400700 1 -- 172.21.15.52:0/2179762420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 msgr2=0x55f3bd3003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a7400700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd3003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a7400700 1 -- 172.21.15.52:0/2179762420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 msgr2=0x55f3bd2ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a7400700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a7400700 1 -- 172.21.15.52:0/2179762420 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3bd300cb0 con 0x55f3bd2fb180 2024-04-26T01:46:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.725+0000 7f09a6a00700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.726+0000 7f09a7400700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd3707e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009f90 tx=0x7f09a000bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e213cae47a8f4d96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.726+0000 7f0990c00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.726+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0020030 con 0x55f3bd2fb180 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.726+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3bd24a8b0 con 0x55f3bd2fb180 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.726+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f09a0007900 con 0x55f3bd2fb180 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.727+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09a000be10 con 0x55f3bd2fb180 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.727+0000 7f09b813db80 10 client.25295 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.727+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f3bd371e30 con 0x55f3bd2fb180 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.727+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f3bd24a8b0 con 0x55f3bd2fb180 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.727+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f09a0011070 con 0x55f3bd2fb180 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 10 client.25295 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 10 client.25295 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 20 client.25295 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd3724b0 0x55f3bd3748a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 4 client.25295 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}] 2024-04-26T01:46:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.728+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.6, "await_for": 42.0}) -- 0x55f3bd0c33e0 con 0x55f3bd3724b0 2024-04-26T01:46:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.729+0000 7f09a6a00700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd3724b0 0x55f3bd3748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.729+0000 7f09a6a00700 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd3724b0 0x55f3bd3748a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31033ab6d2224e1 server_cookie=4396bc61abf9a49c in_seq=0 out_seq=0 2024-04-26T01:46:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.729+0000 7f0990c00700 10 client.25295 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.742+0000 7f0990c00700 1 -- 172.21.15.52:0/2179762420 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55f3bd0c33e0 con 0x55f3bd3724b0 2024-04-26T01:46:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.742+0000 7f0990c00700 10 client.25295 handle_command_reply: tid=0 2024-04-26T01:46:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.742+0000 7f09b813db80 1 client.25295 shutdown 2024-04-26T01:46:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 msgr2=0x55f3bd3707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd3707e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009f90 tx=0x7f09a000bc20 comp rx=0 tx=0).stop 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 shutdown_connections 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f3bd3724b0 0x55f3bd3748a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f3bd2fb180 0x55f3bd3707e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f3bd2ef9b0 0x55f3bd2ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 --2- 172.21.15.52:0/2179762420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f3bd2f02d0 0x55f3bd3003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 >> 172.21.15.52:0/2179762420 conn(0x55f3bd21e730 msgr2=0x55f3bd2eb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.743+0000 7f0990c00700 0 client.25295 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.744+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 shutdown_connections 2024-04-26T01:46:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.744+0000 7f09b813db80 1 -- 172.21.15.52:0/2179762420 wait complete. 2024-04-26T01:46:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.744+0000 7f09b813db80 20 client.25295 trim_cache size 0 max 16384 2024-04-26T01:46:07.743 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:46:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 25, 2024-04-26T01:46:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:46:07.743 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 2, 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "263809e6": { 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 2, 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:46:07.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 85.6, 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:46:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:07.746 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 msgr2=0x7f098c0a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 0x7f098c0a6530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09a807a240 tx=0x7f09940123f0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 msgr2=0x7f09a819f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9f0 tx=0x7f099800edb0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 shutdown_connections 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f098c0a4070 0x7f098c0a6530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a8113730 0x7f09a81a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09a80751b0 0x7f09a819f640 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 --2- 172.21.15.52:0/565367059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09a8076a70 0x7f09a819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 >> 172.21.15.52:0/565367059 conn(0x7f09a8079b20 msgr2=0x7f09a8112390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 shutdown_connections 2024-04-26T01:46:07.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:07.746+0000 7f09ad800700 1 -- 172.21.15.52:0/565367059 wait complete. 2024-04-26T01:46:08.663 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '263809e6', quiesce took 7.9 seconds. Will release after: 25.6 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 -- 172.21.15.52:0/2463255433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 msgr2=0x7f9b4c104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 --2- 172.21.15.52:0/2463255433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c104780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 -- 172.21.15.52:0/2463255433 shutdown_connections 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 --2- 172.21.15.52:0/2463255433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 --2- 172.21.15.52:0/2463255433 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c104d50 0x7f9b4c108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 --2- 172.21.15.52:0/2463255433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c104780 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 -- 172.21.15.52:0/2463255433 >> 172.21.15.52:0/2463255433 conn(0x7f9b4c0ffc40 msgr2=0x7f9b4c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 -- 172.21.15.52:0/2463255433 shutdown_connections 2024-04-26T01:46:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.681+0000 7f9b51400640 1 -- 172.21.15.52:0/2463255433 wait complete. 2024-04-26T01:46:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.682+0000 7f9b51400640 1 Processor -- start 2024-04-26T01:46:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 -- start start 2024-04-26T01:46:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c19a7b0 0x7f9b4c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c114430 con 0x7f9b4c1043a0 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1142b0 con 0x7f9b4c19a7b0 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.683+0000 7f9b51400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1145b0 con 0x7f9b4c109470 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59760/0 (socket says 172.21.15.52:59760) 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b4a800640 1 -- 172.21.15.52:0/3841789273 learned_addr learned my addr 172.21.15.52:0/3841789273 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b4b200640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c19a7b0 0x7f9b4c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 -- 172.21.15.52:0/3841789273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c19a7b0 msgr2=0x7f9b4c19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c19a7b0 0x7f9b4c19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 -- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 msgr2=0x7f9b4c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b49e00640 1 -- 172.21.15.52:0/3841789273 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c19b3b0 con 0x7f9b4c109470 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.684+0000 7f9b4a800640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c1a0be0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e9c0 tx=0x7f9b3800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b43400640 1 -- 172.21.15.52:0/3841789273 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800ccc0 con 0x7f9b4c109470 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b43400640 1 -- 172.21.15.52:0/3841789273 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b380108e0 con 0x7f9b4c109470 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b43400640 1 -- 172.21.15.52:0/3841789273 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380187c0 con 0x7f9b4c109470 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c1a7440 con 0x7f9b4c109470 2024-04-26T01:46:08.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.685+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c1a78a0 con 0x7f9b4c109470 2024-04-26T01:46:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.690+0000 7f9b54228740 1 Processor -- start 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 -- start start 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6badf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dcde613900 con 0x55dcde6ae140 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dcde5b9ef0 con 0x55dcde6b8a00 2024-04-26T01:46:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.691+0000 7f9b54228740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dcde60f8c0 con 0x55dcde6ad820 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59766/0 (socket says 172.21.15.52:59766) 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b43400640 1 -- 172.21.15.52:0/3841789273 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f9b38010430 con 0x7f9b4c109470 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 -- 172.21.15.52:0/4261950925 learned_addr learned my addr 172.21.15.52:0/4261950925 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b49e00640 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6badf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 -- 172.21.15.52:0/4261950925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 msgr2=0x55dcde6adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 -- 172.21.15.52:0/4261950925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 msgr2=0x55dcde6badf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6badf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.692+0000 7f9b4a800640 1 -- 172.21.15.52:0/4261950925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcde326210 con 0x55dcde6ae140 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.693+0000 7f9b4a800640 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c006010 tx=0x7f9b3c0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7dfd6eb51b6df7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.693+0000 7f9b37400640 1 -- 172.21.15.52:0/4261950925 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c040070 con 0x55dcde6ae140 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.693+0000 7f9b37400640 1 -- 172.21.15.52:0/4261950925 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c0047a0 con 0x55dcde6ae140 2024-04-26T01:46:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.693+0000 7f9b37400640 1 -- 172.21.15.52:0/4261950925 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c040070 con 0x55dcde6ae140 2024-04-26T01:46:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.694+0000 7f9b43400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 0x7f9b300a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.694+0000 7f9b43400640 1 -- 172.21.15.52:0/3841789273 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f9b380d0e20 con 0x7f9b4c109470 2024-04-26T01:46:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 -- 172.21.15.52:0/4261950925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 msgr2=0x55dcde6b84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c006010 tx=0x7f9b3c0044b0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b4b200640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 0x7f9b300a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 -- 172.21.15.52:0/4261950925 shutdown_connections 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6badf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6b84c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 --2- 172.21.15.52:0/4261950925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.695+0000 7f9b54228740 1 -- 172.21.15.52:0/4261950925 >> 172.21.15.52:0/4261950925 conn(0x55dcde6a9d30 msgr2=0x55dcde6aa110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.696+0000 7f9b54228740 1 -- 172.21.15.52:0/4261950925 shutdown_connections 2024-04-26T01:46:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.696+0000 7f9b54228740 1 -- 172.21.15.52:0/4261950925 wait complete. 2024-04-26T01:46:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.696+0000 7f9b54228740 1 -- start start 2024-04-26T01:46:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.697+0000 7f9b4b200640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 0x7f9b300a64a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005fd0 tx=0x7f9b44004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.697+0000 7f9b54228740 1 Processor -- start 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 0x55dcde6b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6b8a00 0x55dcde6b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dcde6bb5d0 con 0x55dcde6b8a00 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55dcde5cd310 con 0x55dcde6ae140 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b54228740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dcde6bb750 con 0x55dcde6ad820 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6b5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41870/0 (socket says 172.21.15.52:41870) 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b4a800640 1 -- 172.21.15.52:0/1138091306 learned_addr learned my addr 172.21.15.52:0/1138091306 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b49e00640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 0x55dcde6b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.698+0000 7f9b4b200640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6b8a00 0x55dcde6b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b54228740 10 client.0 _ll_register_callbacks cb 0x55dcde59bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b49e00640 1 -- 172.21.15.52:0/1138091306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 msgr2=0x55dcde6b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b49e00640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b49e00640 1 -- 172.21.15.52:0/1138091306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6b8a00 msgr2=0x55dcde6b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b49e00640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6b8a00 0x55dcde6b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.699+0000 7f9b49e00640 1 -- 172.21.15.52:0/1138091306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcde3f2620 con 0x55dcde6ae140 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b49e00640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 0x55dcde6b5e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b380ce3c0 tx=0x7f9b3809c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2e2f1767756119 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b2be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3809cb40 con 0x55dcde6ae140 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f9b38010a90 con 0x55dcde6ae140 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c0047a0 con 0x55dcde6ae140 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b380a0ef0 con 0x55dcde6ae140 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b54228740 10 client.25307 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.700+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcde6be790 con 0x55dcde6ae140 2024-04-26T01:46:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c0047a0 con 0x55dcde6ae140 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9b38098890 con 0x55dcde6ae140 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 10 client.25307 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 10 client.25307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 20 client.25307 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde6c2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 4 client.25307 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.701+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcde6278b0 con 0x55dcde6bfc40 2024-04-26T01:46:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.702+0000 7f9b4a800640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde6c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.702+0000 7f9b4a800640 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde6c2050 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d46a9c8274eb6f99 server_cookie=3f8b146709a5a0e6 in_seq=0 out_seq=0 2024-04-26T01:46:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.702+0000 7f9b2be00640 10 client.25307 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.703+0000 7f9b2be00640 1 -- 172.21.15.52:0/1138091306 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcde6278b0 con 0x55dcde6bfc40 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.703+0000 7f9b2be00640 10 client.25307 handle_command_reply: tid=0 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.703+0000 7f9b54228740 1 client.25307 shutdown 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 msgr2=0x55dcde6b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 0x55dcde6b5e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b380ce3c0 tx=0x7f9b3809c4a0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 shutdown_connections 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde6c2050 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b2be00640 0 client.25307 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6b8a00 0x55dcde6b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6ae140 0x55dcde6b5e60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 --2- 172.21.15.52:0/1138091306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 >> 172.21.15.52:0/1138091306 conn(0x55dcde6a9d30 msgr2=0x55dcde6afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 shutdown_connections 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 1 -- 172.21.15.52:0/1138091306 wait complete. 2024-04-26T01:46:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.704+0000 7f9b54228740 20 client.25307 trim_cache size 0 max 16384 2024-04-26T01:46:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 Processor -- start 2024-04-26T01:46:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 -- start start 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6e3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6e3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1ae140 con 0x55dcde6ae140 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1aba10 con 0x55dcde6b8a00 2024-04-26T01:46:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.735+0000 7f9b54228740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1a98a0 con 0x55dcde6ad820 2024-04-26T01:46:08.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41874/0 (socket says 172.21.15.52:41874) 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 -- 172.21.15.52:0/3916632875 learned_addr learned my addr 172.21.15.52:0/3916632875 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4b200640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 -- 172.21.15.52:0/3916632875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 msgr2=0x55dcde6e3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6e3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 -- 172.21.15.52:0/3916632875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 msgr2=0x55dcde6e3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6e3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 -- 172.21.15.52:0/3916632875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcde6b3ed0 con 0x55dcde6ad820 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b49e00640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.736+0000 7f9b4a800640 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005c40 tx=0x7f9b3c02a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb14a693a55ea3e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.737+0000 7f9b36a00640 1 -- 172.21.15.52:0/3916632875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c040070 con 0x55dcde6ad820 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.737+0000 7f9b36a00640 1 -- 172.21.15.52:0/3916632875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c004b60 con 0x55dcde6ad820 2024-04-26T01:46:08.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.737+0000 7f9b36a00640 1 -- 172.21.15.52:0/3916632875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035040 con 0x55dcde6ad820 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- 172.21.15.52:0/3916632875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 msgr2=0x55dcde6a9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005c40 tx=0x7f9b3c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- 172.21.15.52:0/3916632875 shutdown_connections 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde6e3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ae140 0x55dcde6e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 --2- 172.21.15.52:0/3916632875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ad820 0x55dcde6a9610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- 172.21.15.52:0/3916632875 >> 172.21.15.52:0/3916632875 conn(0x55dcde6a9d30 msgr2=0x55dcde6aa110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- 172.21.15.52:0/3916632875 shutdown_connections 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- 172.21.15.52:0/3916632875 wait complete. 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.738+0000 7f9b54228740 1 -- start start 2024-04-26T01:46:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 Processor -- start 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 0x55dcde6c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde738b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1a9ef0 con 0x55dcde6ad820 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1ab340 con 0x55dcde6b8a00 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.739+0000 7f9b54228740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1aa5a0 con 0x55dcde6ae140 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b54228740 10 client.0 _ll_register_callbacks cb 0x7f9b3809c910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59790/0 (socket says 172.21.15.52:59790) 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 -- 172.21.15.52:0/2458432946 learned_addr learned my addr 172.21.15.52:0/2458432946 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4b200640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde738b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b49e00640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 0x55dcde6c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 -- 172.21.15.52:0/2458432946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 msgr2=0x55dcde6c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 0x55dcde6c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 -- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 msgr2=0x55dcde738b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde738b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4a800640 1 -- 172.21.15.52:0/2458432946 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcde739140 con 0x55dcde6ad820 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b4b200640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde738b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.740+0000 7f9b49e00640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 0x55dcde6c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.741+0000 7f9b4a800640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c003ee0 tx=0x7f9b3c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688815d9da8650d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.741+0000 7f9b2be00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.741+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035070 con 0x55dcde6ad820 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.741+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f9b3c052290 con 0x55dcde6ad820 2024-04-26T01:46:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.741+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c004b60 con 0x55dcde6ad820 2024-04-26T01:46:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c062050 con 0x55dcde6ad820 2024-04-26T01:46:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b54228740 10 client.15222 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcde7514d0 con 0x55dcde6ad820 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c004b60 con 0x55dcde6ad820 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9b3c005350 con 0x55dcde6ad820 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b54228740 10 client.15222 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.742+0000 7f9b54228740 10 client.15222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.743+0000 7f9b54228740 20 client.15222 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.743+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde753710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.743+0000 7f9b54228740 4 client.15222 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.743+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dcde621320 con 0x55dcde6bfc40 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.743+0000 7f9b49e00640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde753710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.744+0000 7f9b49e00640 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde753710 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=211f7e0430971e6e server_cookie=fc1221b82482fd88 in_seq=0 out_seq=0 2024-04-26T01:46:08.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.744+0000 7f9b2be00640 10 client.15222 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.753+0000 7f9b2be00640 1 -- 172.21.15.52:0/2458432946 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dcde621320 con 0x55dcde6bfc40 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.753+0000 7f9b2be00640 10 client.15222 handle_command_reply: tid=0 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.753+0000 7f9b54228740 1 client.15222 shutdown 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 msgr2=0x55dcde6c5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c003ee0 tx=0x7f9b3c009260 comp rx=0 tx=0).stop 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 shutdown_connections 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55dcde6bfc40 0x55dcde753710 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55dcde6b8a00 0x55dcde738b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dcde6ae140 0x55dcde6c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 --2- 172.21.15.52:0/2458432946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dcde6ad820 0x55dcde6c5bc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 >> 172.21.15.52:0/2458432946 conn(0x55dcde6a9d30 msgr2=0x55dcde6cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b2be00640 0 client.15222 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 shutdown_connections 2024-04-26T01:46:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 1 -- 172.21.15.52:0/2458432946 wait complete. 2024-04-26T01:46:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.754+0000 7f9b54228740 20 client.15222 trim_cache size 0 max 16384 2024-04-26T01:46:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:46:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:46:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:46:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:46:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 msgr2=0x7f9b300a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 0x7f9b300a64a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005fd0 tx=0x7f9b44004a40 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 msgr2=0x7f9b4c1a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c1a0be0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e9c0 tx=0x7f9b3800ee80 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 shutdown_connections 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9b300a3fe0 0x7f9b300a64a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b4c19a7b0 0x7f9b4c19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9b4c109470 0x7f9b4c1a0be0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 --2- 172.21.15.52:0/3841789273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b4c1043a0 0x7f9b4c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.755+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 >> 172.21.15.52:0/3841789273 conn(0x7f9b4c0ffc40 msgr2=0x7f9b4c101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.756+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 shutdown_connections 2024-04-26T01:46:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:08.756+0000 7f9b51400640 1 -- 172.21.15.52:0/3841789273 wait complete. 2024-04-26T01:46:09.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:09 smithi191 ceph-mon[32745]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T01:46:09.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:09 smithi183 ceph-mon[32752]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T01:46:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:09 smithi052 ceph-mon[27851]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T01:46:10.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:10 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:10.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:10 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:10.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:10 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:46:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:11 smithi191 ceph-mon[32745]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-26T01:46:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:11 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:11 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:11 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:46:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:11 smithi183 ceph-mon[32752]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-26T01:46:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:11 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:11 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:11 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:46:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:11 smithi052 ceph-mon[27851]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-26T01:46:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:11 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:46:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:11 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:46:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:11 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:46:13.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:13 smithi191 ceph-mon[32745]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-26T01:46:13.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:13 smithi183 ceph-mon[32752]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-26T01:46:13.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:13 smithi052 ceph-mon[27851]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-26T01:46:15.297 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:15 smithi052 ceph-mon[27851]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:46:15.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:15 smithi191 ceph-mon[32745]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:46:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:15 smithi183 ceph-mon[32752]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-26T01:46:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:17 smithi191 ceph-mon[32745]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:46:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:17 smithi183 ceph-mon[32752]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:46:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:17 smithi052 ceph-mon[27851]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-26T01:46:19.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:19 smithi052 ceph-mon[27851]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T01:46:19.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:19 smithi191 ceph-mon[32745]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T01:46:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:19 smithi183 ceph-mon[32752]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-26T01:46:21.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:21 smithi191 ceph-mon[32745]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T01:46:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:21 smithi183 ceph-mon[32752]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T01:46:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:21 smithi052 ceph-mon[27851]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T01:46:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.895+0000 7fa49dc00640 1 -- 172.21.15.52:0/1559134693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 msgr2=0x7fa498107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.895+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1559134693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa498107450 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa488009860 tx=0x7fa48802a340 comp rx=0 tx=0).stop 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 -- 172.21.15.52:0/1559134693 shutdown_connections 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1559134693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4981009e0 0x7fa49810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1559134693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1559134693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa498107450 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 -- 172.21.15.52:0/1559134693 >> 172.21.15.52:0/1559134693 conn(0x7fa498076f50 msgr2=0x7fa4980fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 -- 172.21.15.52:0/1559134693 shutdown_connections 2024-04-26T01:46:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.896+0000 7fa49dc00640 1 -- 172.21.15.52:0/1559134693 wait complete. 2024-04-26T01:46:21.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.897+0000 7fa49dc00640 1 Processor -- start 2024-04-26T01:46:21.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.897+0000 7fa49dc00640 1 -- start start 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4981009e0 0x7fa4981a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa49819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112190 con 0x7fa498100020 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112010 con 0x7fa498107070 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa49dc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112310 con 0x7fa4981009e0 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa496a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa496a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37484/0 (socket says 172.21.15.52:37484) 2024-04-26T01:46:21.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.898+0000 7fa496a00640 1 -- 172.21.15.52:0/1536823885 learned_addr learned my addr 172.21.15.52:0/1536823885 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:21.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa497400640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa49819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 -- 172.21.15.52:0/1536823885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4981009e0 msgr2=0x7fa4981a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4981009e0 0x7fa4981a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 -- 172.21.15.52:0/1536823885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 msgr2=0x7fa49819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa49819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 -- 172.21.15.52:0/1536823885 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49819af30 con 0x7fa498100020 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.899+0000 7fa496a00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa488035040 tx=0x7fa488032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.900+0000 7fa48f400640 1 -- 172.21.15.52:0/1536823885 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488040020 con 0x7fa498100020 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.900+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49819b1c0 con 0x7fa498100020 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.900+0000 7fa48f400640 1 -- 172.21.15.52:0/1536823885 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa488030dc0 con 0x7fa498100020 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.900+0000 7fa48f400640 1 -- 172.21.15.52:0/1536823885 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488037040 con 0x7fa498100020 2024-04-26T01:46:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.900+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4981a7530 con 0x7fa498100020 2024-04-26T01:46:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.904+0000 7fa4a0baa740 1 Processor -- start 2024-04-26T01:46:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 -- start start 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 0x562d8a279b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 0x562d8a28ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562d8a1df900 con 0x562d8a279760 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562d8a185ef0 con 0x562d8a27a080 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.905+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562d8a1db8c0 con 0x562d8a28d210 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa48f400640 1 -- 172.21.15.52:0/1536823885 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fa488043020 con 0x7fa498100020 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa496000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa496000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54352/0 (socket says 172.21.15.52:54352) 2024-04-26T01:46:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa496000640 1 -- 172.21.15.52:0/1439670341 learned_addr learned my addr 172.21.15.52:0/1439670341 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa497400640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 0x562d8a279b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa496a00640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 0x562d8a28ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.906+0000 7fa496000640 1 -- 172.21.15.52:0/1439670341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 msgr2=0x562d8a28ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496000640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 0x562d8a28ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496000640 1 -- 172.21.15.52:0/1439670341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 msgr2=0x562d8a279b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496000640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 0x562d8a279b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496000640 1 -- 172.21.15.52:0/1439670341 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d89ef2210 con 0x562d8a28d210 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496a00640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 0x562d8a28ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa497400640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 0x562d8a279b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.907+0000 7fa496000640 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa484009b30 tx=0x7fa48400e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=206b31403934383 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.908+0000 7fa483e00640 1 -- 172.21.15.52:0/1439670341 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48400ed80 con 0x562d8a28d210 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.908+0000 7fa48f400640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 0x7fa47c0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.908+0000 7fa483e00640 1 -- 172.21.15.52:0/1439670341 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4840108e0 con 0x562d8a28d210 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.908+0000 7fa483e00640 1 -- 172.21.15.52:0/1439670341 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4840189a0 con 0x562d8a28d210 2024-04-26T01:46:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.908+0000 7fa48f400640 1 -- 172.21.15.52:0/1536823885 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa4880f0610 con 0x7fa498100020 2024-04-26T01:46:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa497400640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 0x7fa47c0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 -- 172.21.15.52:0/1439670341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 msgr2=0x562d8a28f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa484009b30 tx=0x7fa48400e350 comp rx=0 tx=0).stop 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 -- 172.21.15.52:0/1439670341 shutdown_connections 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28f600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a27a080 0x562d8a28ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/1439670341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a279760 0x562d8a279b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.909+0000 7fa4a0baa740 1 -- 172.21.15.52:0/1439670341 >> 172.21.15.52:0/1439670341 conn(0x562d8a275d30 msgr2=0x562d8a276110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.910+0000 7fa4a0baa740 1 -- 172.21.15.52:0/1439670341 shutdown_connections 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.910+0000 7fa497400640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 0x7fa47c0a6420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa490009fd0 tx=0x7fa490009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.910+0000 7fa4a0baa740 1 -- 172.21.15.52:0/1439670341 wait complete. 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.910+0000 7fa4a0baa740 1 -- start start 2024-04-26T01:46:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.910+0000 7fa4a0baa740 1 Processor -- start 2024-04-26T01:46:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a286d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562d8a27c700 con 0x562d8a27a080 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562d8a199310 con 0x562d8a279760 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562d8a27c880 con 0x562d8a28d210 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa4a0baa740 10 client.0 _ll_register_callbacks cb 0x562d8a167cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa496000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa496000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37514/0 (socket says 172.21.15.52:37514) 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.911+0000 7fa496000640 1 -- 172.21.15.52:0/761771958 learned_addr learned my addr 172.21.15.52:0/761771958 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496000640 1 -- 172.21.15.52:0/761771958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 msgr2=0x562d8a28ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496000640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496000640 1 -- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 msgr2=0x562d8a286d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496a00640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a286d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496000640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a286d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.912+0000 7fa496000640 1 -- 172.21.15.52:0/761771958 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d89fbe620 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.913+0000 7fa496000640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa48400edb0 tx=0x7fa4840184c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa7eff5e456fe93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.913+0000 7fa480c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.913+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484005d10 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.913+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa48401e040 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.913+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4840108e0 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa484005d10 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 10 client.15234 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d8a2b5840 con 0x562d8a27a080 2024-04-26T01:46:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4840108e0 con 0x562d8a27a080 2024-04-26T01:46:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa4840047a0 con 0x562d8a27a080 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 10 client.15234 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 10 client.15234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 20 client.15234 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a2b6c50 0x562d8a2b9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 4 client.15234 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.914+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d8a1f38b0 con 0x562d8a2b6c50 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.915+0000 7fa496a00640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a2b6c50 0x562d8a2b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.915+0000 7fa496a00640 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a2b6c50 0x562d8a2b9040 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b18c942045ca77 server_cookie=b75391a841b41d72 in_seq=0 out_seq=0 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.915+0000 7fa480c00640 10 client.15234 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.916+0000 7fa480c00640 1 -- 172.21.15.52:0/761771958 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d8a1f38b0 con 0x562d8a2b6c50 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.916+0000 7fa480c00640 10 client.15234 handle_command_reply: tid=0 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.916+0000 7fa4a0baa740 1 client.15234 shutdown 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.916+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 msgr2=0x562d8a287260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.916+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa48400edb0 tx=0x7fa4840184c0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 shutdown_connections 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a2b6c50 0x562d8a2b9040 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a28ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a287260 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/761771958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a286d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 >> 172.21.15.52:0/761771958 conn(0x562d8a275d30 msgr2=0x562d8a27bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa480c00640 0 client.15234 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 shutdown_connections 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 1 -- 172.21.15.52:0/761771958 wait complete. 2024-04-26T01:46:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.917+0000 7fa4a0baa740 20 client.15234 trim_cache size 0 max 16384 2024-04-26T01:46:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 Processor -- start 2024-04-26T01:46:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 -- start start 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a28c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2be810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2bed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa498072390 con 0x562d8a27a080 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa49806fc60 con 0x562d8a279760 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.933+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa49806dad0 con 0x562d8a28d210 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa496a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a28c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa496a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a28c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41658/0 (socket says 172.21.15.52:41658) 2024-04-26T01:46:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa496a00640 1 -- 172.21.15.52:0/2164934100 learned_addr learned my addr 172.21.15.52:0/2164934100 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa496000640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2be810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2bed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 -- 172.21.15.52:0/2164934100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 msgr2=0x562d8a28c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a28c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 -- 172.21.15.52:0/2164934100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 msgr2=0x562d8a2be810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2be810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa497400640 1 -- 172.21.15.52:0/2164934100 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d8a279eb0 con 0x562d8a28d210 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.934+0000 7fa496000640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2be810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.935+0000 7fa497400640 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2bed50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x562d8a270dd0 tx=0x7fa490004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118471c9d6839603 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.935+0000 7fa483400640 1 -- 172.21.15.52:0/2164934100 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49000fde0 con 0x562d8a28d210 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.935+0000 7fa483400640 1 -- 172.21.15.52:0/2164934100 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4900108e0 con 0x562d8a28d210 2024-04-26T01:46:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.935+0000 7fa483400640 1 -- 172.21.15.52:0/2164934100 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49001ea60 con 0x562d8a28d210 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- 172.21.15.52:0/2164934100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 msgr2=0x562d8a2bed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2bed50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x562d8a270dd0 tx=0x7fa490004a40 comp rx=0 tx=0).stop 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- 172.21.15.52:0/2164934100 shutdown_connections 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2bed50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2be810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/2164934100 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a28c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- 172.21.15.52:0/2164934100 >> 172.21.15.52:0/2164934100 conn(0x562d8a275d30 msgr2=0x562d8a270e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- 172.21.15.52:0/2164934100 shutdown_connections 2024-04-26T01:46:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- 172.21.15.52:0/2164934100 wait complete. 2024-04-26T01:46:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.936+0000 7fa4a0baa740 1 -- start start 2024-04-26T01:46:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 Processor -- start 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a2c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa49806e120 con 0x562d8a27a080 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa49806f590 con 0x562d8a279760 2024-04-26T01:46:21.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa49806e7f0 con 0x562d8a28d210 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa4a0baa740 10 client.0 _ll_register_callbacks cb 0x7fa484010760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.937+0000 7fa497400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54368/0 (socket says 172.21.15.52:54368) 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa496000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 -- 172.21.15.52:0/4020684312 learned_addr learned my addr 172.21.15.52:0/4020684312 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa496a00640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a2c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 -- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 msgr2=0x562d8a2c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a2c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 -- 172.21.15.52:0/4020684312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 msgr2=0x562d8a2c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 -- 172.21.15.52:0/4020684312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d8a2c7e50 con 0x562d8a28d210 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa496a00640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a2c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa496000640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa497400640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa4900115e0 tx=0x7fa490011610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccf440383b7a9ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.938+0000 7fa480c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:46:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49000fde0 con 0x562d8a28d210 2024-04-26T01:46:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4900108e0 con 0x562d8a28d210 2024-04-26T01:46:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa490004cc0 con 0x562d8a28d210 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa490010a90 con 0x562d8a28d210 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa4a0baa740 10 client.25333 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562d8a31c8b0 con 0x562d8a28d210 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.939+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4900108e0 con 0x562d8a28d210 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa490026070 con 0x562d8a28d210 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 10 client.25333 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 10 client.25333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 20 client.25333 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a31cf00 0x562d8a31f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 4 client.25333 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d8a1ed320 con 0x562d8a31cf00 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.940+0000 7fa496a00640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a31cf00 0x562d8a31f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.941+0000 7fa496a00640 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a31cf00 0x562d8a31f320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dad069d5e66a8acc server_cookie=e88b9f7d2afcd1cd in_seq=0 out_seq=0 2024-04-26T01:46:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.941+0000 7fa480c00640 10 client.25333 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.946+0000 7fa480c00640 1 -- 172.21.15.52:0/4020684312 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d8a1ed320 con 0x562d8a31cf00 2024-04-26T01:46:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.946+0000 7fa480c00640 10 client.25333 handle_command_reply: tid=0 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.946+0000 7fa4a0baa740 1 client.25333 shutdown 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 msgr2=0x562d8a2c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa4900115e0 tx=0x7fa490011610 comp rx=0 tx=0).stop 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 shutdown_connections 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562d8a31cf00 0x562d8a31f320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562d8a28d210 0x562d8a2c7910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562d8a27a080 0x562d8a2c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 --2- 172.21.15.52:0/4020684312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562d8a279760 0x562d8a2c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 >> 172.21.15.52:0/4020684312 conn(0x562d8a275d30 msgr2=0x562d8a280550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa480c00640 0 client.25333 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 shutdown_connections 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 1 -- 172.21.15.52:0/4020684312 wait complete. 2024-04-26T01:46:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.947+0000 7fa4a0baa740 20 client.25333 trim_cache size 0 max 16384 2024-04-26T01:46:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:46:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:46:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:46:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 msgr2=0x7fa47c0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 0x7fa47c0a6420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa490009fd0 tx=0x7fa490009500 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 msgr2=0x7fa4981a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa488035040 tx=0x7fa488032d40 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 shutdown_connections 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa47c0a3f60 0x7fa47c0a6420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa498107070 0x7fa49819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4981009e0 0x7fa4981a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 --2- 172.21.15.52:0/1536823885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa498100020 0x7fa4981a05b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.948+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 >> 172.21.15.52:0/1536823885 conn(0x7fa498076f50 msgr2=0x7fa49810e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.949+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 shutdown_connections 2024-04-26T01:46:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:21.949+0000 7fa49dc00640 1 -- 172.21.15.52:0/1536823885 wait complete. 2024-04-26T01:46:23.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:23 smithi191 ceph-mon[32745]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-26T01:46:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:23 smithi183 ceph-mon[32752]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-26T01:46:23.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:23 smithi052 ceph-mon[27851]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-26T01:46:25.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:25 smithi191 ceph-mon[32745]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:46:25.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:25 smithi183 ceph-mon[32752]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:46:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:25 smithi052 ceph-mon[27851]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T01:46:27.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:27 smithi191 ceph-mon[32745]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:46:27.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:27 smithi183 ceph-mon[32752]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:46:27.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:27 smithi052 ceph-mon[27851]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T01:46:29.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:29 smithi191 ceph-mon[32745]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:46:29.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:29 smithi183 ceph-mon[32752]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:46:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:29 smithi052 ceph-mon[27851]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T01:46:31.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:31 smithi191 ceph-mon[32745]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:46:31.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:31 smithi183 ceph-mon[32752]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:46:31.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:31 smithi052 ceph-mon[27851]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T01:46:32.421 INFO:tasks.workunit.client.0.smithi052.stderr:Updating files: 63% (7911/12492) Updating files: 64% (7995/12492) Updating files: 65% (8120/12492) Updating files: 66% (8245/12492) Updating files: 67% (8370/12492) Updating files: 68% (8495/12492) Updating files: 69% (8620/12492) Updating files: 70% (8745/12492) Updating files: 71% (8870/12492) Updating files: 72% (8995/12492) Updating files: 73% (9120/12492) Updating files: 74% (9245/12492) Updating files: 75% (9369/12492) Updating files: 76% (9494/12492) Updating files: 77% (9619/12492) Updating files: 78% (9744/12492) Updating files: 79% (9869/12492) Updating files: 80% (9994/12492) Updating files: 81% (10119/12492) Updating files: 82% (10244/12492) Updating files: 83% (10369/12492) Updating files: 84% (10494/12492) Updating files: 85% (10619/12492) Updating files: 86% (10744/12492) Updating files: 87% (10869/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:Note: switching to 'afa1933b0fdbb6c99c947d1eda34d661d23cd327'. 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.835 INFO:tasks.workunit.client.0.smithi052.stderr: git switch -c 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr:Or undo this operation with: 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr: git switch - 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-04-26T01:46:32.836 INFO:tasks.workunit.client.0.smithi052.stderr:HEAD is now at afa1933b0fd qa/suites/fs: add quiescer to the fs suite 2024-04-26T01:46:32.839 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T01:46:32.945 INFO:tasks.workunit.client.0.smithi052.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T01:46:32.953 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T01:46:32.953 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T01:46:33.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:33 smithi191 ceph-mon[32745]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:33.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:33 smithi191 ceph-mon[32745]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:46:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:33 smithi183 ceph-mon[32752]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:33 smithi183 ceph-mon[32752]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:46:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:33 smithi052 ceph-mon[27851]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T01:46:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:33 smithi052 ceph-mon[27851]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-26T01:46:33.761 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T01:46:33.893 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T01:46:33.987 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T01:46:33.993 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T01:46:33.993 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T01:46:34.082 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T01:46:34.091 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T01:46:34.091 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T01:46:34.142 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-26T01:46:34.143 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-26T01:46:34.143 DEBUG:teuthology.orchestra.run.smithi052:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-26T01:46:34.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:46:34.746 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 -- 172.21.15.52:0/904973447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf140ffe60 msgr2=0x7faf141002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 --2- 172.21.15.52:0/904973447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf140ffe60 0x7faf141002e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf00009930 tx=0x7faf0002a420 comp rx=0 tx=0).stop 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 -- 172.21.15.52:0/904973447 shutdown_connections 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 --2- 172.21.15.52:0/904973447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14100820 0x7faf1410eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 --2- 172.21.15.52:0/904973447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf140ffe60 0x7faf141002e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 --2- 172.21.15.52:0/904973447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 0x7faf14107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.098+0000 7faf1aa00640 1 -- 172.21.15.52:0/904973447 >> 172.21.15.52:0/904973447 conn(0x7faf14076db0 msgr2=0x7faf140fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.099+0000 7faf1aa00640 1 -- 172.21.15.52:0/904973447 shutdown_connections 2024-04-26T01:46:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.099+0000 7faf1aa00640 1 -- 172.21.15.52:0/904973447 wait complete. 2024-04-26T01:46:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.099+0000 7faf1aa00640 1 Processor -- start 2024-04-26T01:46:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 -- start start 2024-04-26T01:46:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 0x7faf141018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 0x7faf14102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf14112040 con 0x7faf14100820 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf14111ec0 con 0x7faf14106eb0 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf1aa00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf141121c0 con 0x7faf140ffe60 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf18c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf19600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 0x7faf141018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf18c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40922/0 (socket says 172.21.15.52:40922) 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.100+0000 7faf18c00640 1 -- 172.21.15.52:0/3193682246 learned_addr learned my addr 172.21.15.52:0/3193682246 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf1a000640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 0x7faf14102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 -- 172.21.15.52:0/3193682246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 msgr2=0x7faf14102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 0x7faf14102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 -- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 msgr2=0x7faf14101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 -- 172.21.15.52:0/3193682246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf141a6a80 con 0x7faf140ffe60 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf18c00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.101+0000 7faf19600640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 0x7faf141018d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf0800e9c0 tx=0x7faf0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.102+0000 7faf12000640 1 -- 172.21.15.52:0/3193682246 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0800ccc0 con 0x7faf140ffe60 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.102+0000 7faf12000640 1 -- 172.21.15.52:0/3193682246 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf080035f0 con 0x7faf140ffe60 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.102+0000 7faf12000640 1 -- 172.21.15.52:0/3193682246 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0800ce60 con 0x7faf140ffe60 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.102+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf141a6d70 con 0x7faf140ffe60 2024-04-26T01:46:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.102+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf141a71e0 con 0x7faf140ffe60 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.105+0000 7faf1ee2e740 1 Processor -- start 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.105+0000 7faf12000640 1 -- 172.21.15.52:0/3193682246 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7faf08004000 con 0x7faf140ffe60 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.105+0000 7faf1ee2e740 1 -- start start 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560c3b323900 con 0x560c3b3c8b90 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560c3b2c9ef0 con 0x560c3b3be2d0 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560c3b31f8c0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf18c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1a000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf19600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf18c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40928/0 (socket says 172.21.15.52:40928) 2024-04-26T01:46:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf18c00640 1 -- 172.21.15.52:0/2389747882 learned_addr learned my addr 172.21.15.52:0/2389747882 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.106+0000 7faf1a000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3bdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53720/0 (socket says 172.21.15.52:53720) 2024-04-26T01:46:35.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 -- 172.21.15.52:0/2389747882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 msgr2=0x560c3b3bdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 -- 172.21.15.52:0/2389747882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 msgr2=0x560c3b3c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 -- 172.21.15.52:0/2389747882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c3b036210 con 0x560c3b3c8b90 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf18c00640 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf00009900 tx=0x7faf000043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b883882ca4cd3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf12000640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 0x7faefc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf12000640 1 -- 172.21.15.52:0/3193682246 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7faf080d0b20 con 0x7faf140ffe60 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf06600640 1 -- 172.21.15.52:0/2389747882 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00040070 con 0x560c3b3c8b90 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf1a000640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 0x7faefc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf06600640 1 -- 172.21.15.52:0/2389747882 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf000047a0 con 0x560c3b3c8b90 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.107+0000 7faf06600640 1 -- 172.21.15.52:0/2389747882 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00035040 con 0x560c3b3c8b90 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2389747882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 msgr2=0x560c3b3caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1a000640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 0x7faefc0a6250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faf0c005fd0 tx=0x7faf0c009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf00009900 tx=0x7faf000043f0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2389747882 shutdown_connections 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3caf80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2389747882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2389747882 >> 172.21.15.52:0/2389747882 conn(0x560c3b3b9ec0 msgr2=0x560c3b3ba2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2389747882 shutdown_connections 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.108+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2389747882 wait complete. 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.109+0000 7faf1ee2e740 1 -- start start 2024-04-26T01:46:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.109+0000 7faf1ee2e740 1 Processor -- start 2024-04-26T01:46:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.109+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.109+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560c3b3cb760 con 0x560c3b3c8b90 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560c3b2dd310 con 0x560c3b3be2d0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560c3b3cb8e0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1ee2e740 10 client.0 _ll_register_callbacks cb 0x560c3b2abcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40942/0 (socket says 172.21.15.52:40942) 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf1a000640 1 -- 172.21.15.52:0/3749466151 learned_addr learned my addr 172.21.15.52:0/3749466151 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf18c00640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 -- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 msgr2=0x560c3b3c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 -- 172.21.15.52:0/3749466151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 msgr2=0x560c3b3c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.110+0000 7faf19600640 1 -- 172.21.15.52:0/3749466151 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c3b102620 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faf19600640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3c5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf080ce210 tx=0x7faf08098df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e65320171d88ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf080d6070 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7faf080037a0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf000047a0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf0809d190 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faf1ee2e740 10 client.25345 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560c3b3d7a10 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf000047a0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.111+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf080ce4c0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 10 client.25345 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 10 client.25345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 20 client.25345 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b3d8ed0 0x560c3b3db2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 4 client.25345 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c3b3378b0 con 0x560c3b3d8ed0 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf18c00640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b3d8ed0 0x560c3b3db2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faf18c00640 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b3d8ed0 0x560c3b3db2c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11db109842d4c0b server_cookie=d385ccdf1c86beeb in_seq=0 out_seq=0 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.112+0000 7faefaa00640 10 client.25345 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.113+0000 7faefaa00640 1 -- 172.21.15.52:0/3749466151 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c3b3378b0 con 0x560c3b3d8ed0 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.113+0000 7faefaa00640 10 client.25345 handle_command_reply: tid=0 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.113+0000 7faf1ee2e740 1 client.25345 shutdown 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 msgr2=0x560c3b3c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3c5b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf080ce210 tx=0x7faf08098df0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 shutdown_connections 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b3d8ed0 0x560c3b3db2c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3749466151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3c5b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 >> 172.21.15.52:0/3749466151 conn(0x560c3b3b9ec0 msgr2=0x560c3b3bfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faefaa00640 0 client.25345 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 shutdown_connections 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3749466151 wait complete. 2024-04-26T01:46:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.114+0000 7faf1ee2e740 20 client.25345 trim_cache size 0 max 16384 2024-04-26T01:46:35.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 Processor -- start 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 -- start start 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3be2d0 0x560c3b3e0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf141b2460 con 0x560c3b3be2d0 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf141afd30 con 0x560c3b3c8b90 2024-04-26T01:46:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf141adb80 con 0x560c3b3bd9b0 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf18c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3be2d0 0x560c3b3e0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf19600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.131+0000 7faf1a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53742/0 (socket says 172.21.15.52:53742) 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 -- 172.21.15.52:0/3857930434 learned_addr learned my addr 172.21.15.52:0/3857930434 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf1a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58074/0 (socket says 172.21.15.52:58074) 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 -- 172.21.15.52:0/3857930434 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 msgr2=0x560c3b3e10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 -- 172.21.15.52:0/3857930434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3be2d0 msgr2=0x560c3b3e0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3be2d0 0x560c3b3e0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 -- 172.21.15.52:0/3857930434 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c3b3db8f0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf1a000640 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.132+0000 7faf19600640 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf08099460 tx=0x7faf0809afa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ae56367882e05d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf05c00640 1 -- 172.21.15.52:0/3857930434 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf080d6070 con 0x560c3b3bd9b0 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf05c00640 1 -- 172.21.15.52:0/3857930434 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf0809ca90 con 0x560c3b3bd9b0 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf05c00640 1 -- 172.21.15.52:0/3857930434 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf080a0e40 con 0x560c3b3bd9b0 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3857930434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 msgr2=0x560c3b3d4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf08099460 tx=0x7faf0809afa0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3857930434 shutdown_connections 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3c8b90 0x560c3b3e10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3be2d0 0x560c3b3e0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/3857930434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3d4620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3857930434 >> 172.21.15.52:0/3857930434 conn(0x560c3b3b9ec0 msgr2=0x560c3b3b9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.133+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3857930434 shutdown_connections 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.134+0000 7faf1ee2e740 1 -- 172.21.15.52:0/3857930434 wait complete. 2024-04-26T01:46:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.134+0000 7faf1ee2e740 1 -- start start 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.134+0000 7faf1ee2e740 1 Processor -- start 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.134+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.134+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1ee2e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf141ae1f0 con 0x560c3b3c8b90 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf141af660 con 0x560c3b3be2d0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1ee2e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faf141ae8c0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1ee2e740 10 client.0 _ll_register_callbacks cb 0x7faf08098610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40966/0 (socket says 172.21.15.52:40966) 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1a000640 1 -- 172.21.15.52:0/2265514590 learned_addr learned my addr 172.21.15.52:0/2265514590 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf18c00640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 -- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 msgr2=0x560c3b3ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 -- 172.21.15.52:0/2265514590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 msgr2=0x560c3b3cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf19600640 1 -- 172.21.15.52:0/2265514590 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c3b43e6b0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf1a000640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.135+0000 7faf18c00640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faf19600640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3ce840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faf0800c6a0 tx=0x7faf0809c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b14efb2fab60ad4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0807f070 con 0x560c3b3bd9b0 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7faf080ce4c0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf0809ca90 con 0x560c3b3bd9b0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf080d6070 con 0x560c3b3bd9b0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faf1ee2e740 10 client.25351 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560c3b43f9c0 con 0x560c3b3bd9b0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf0809ca90 con 0x560c3b3bd9b0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.136+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7faf0809b950 con 0x560c3b3bd9b0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 10 client.25351 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 20 client.25351 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b43fe40 0x560c3b442230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 4 client.25351 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c3b331320 con 0x560c3b43fe40 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf18c00640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b43fe40 0x560c3b442230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faf18c00640 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b43fe40 0x560c3b442230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7863d531e31ec05 server_cookie=48e3bddb1bfd43ab in_seq=0 out_seq=0 2024-04-26T01:46:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.137+0000 7faefaa00640 10 client.25351 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.143+0000 7faefaa00640 1 -- 172.21.15.52:0/2265514590 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c3b331320 con 0x560c3b43fe40 2024-04-26T01:46:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.143+0000 7faefaa00640 10 client.25351 handle_command_reply: tid=0 2024-04-26T01:46:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.143+0000 7faf1ee2e740 1 client.25351 shutdown 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 msgr2=0x560c3b3ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3ce840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faf0800c6a0 tx=0x7faf0809c990 comp rx=0 tx=0).stop 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 shutdown_connections 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560c3b43fe40 0x560c3b442230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560c3b3c8b90 0x560c3b3cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560c3b3be2d0 0x560c3b3ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 --2- 172.21.15.52:0/2265514590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560c3b3bd9b0 0x560c3b3ce840 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faefaa00640 0 client.25351 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 >> 172.21.15.52:0/2265514590 conn(0x560c3b3b9ec0 msgr2=0x560c3b3c5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 shutdown_connections 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 1 -- 172.21.15.52:0/2265514590 wait complete. 2024-04-26T01:46:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.144+0000 7faf1ee2e740 20 client.25351 trim_cache size 0 max 16384 2024-04-26T01:46:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:46:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:46:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:46:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:46:35.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 msgr2=0x7faefc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 0x7faefc0a6250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faf0c005fd0 tx=0x7faf0c009500 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 msgr2=0x7faf141018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 0x7faf141018d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf0800e9c0 tx=0x7faf0800ee80 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 shutdown_connections 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faefc0a3d90 0x7faefc0a6250 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf14106eb0 0x7faf14102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf14100820 0x7faf14101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 --2- 172.21.15.52:0/3193682246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf140ffe60 0x7faf141018d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 >> 172.21.15.52:0/3193682246 conn(0x7faf14076db0 msgr2=0x7faf140fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 shutdown_connections 2024-04-26T01:46:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:35.145+0000 7faf1aa00640 1 -- 172.21.15.52:0/3193682246 wait complete. 2024-04-26T01:46:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:35 smithi052 ceph-mon[27851]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T01:46:35.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:35 smithi191 ceph-mon[32745]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T01:46:35.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:35 smithi183 ceph-mon[32752]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T01:46:37.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 -- 172.21.15.52:0/403021891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0075590 msgr2=0x7fdcb010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 --2- 172.21.15.52:0/403021891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0075590 0x7fdcb010ef00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009990 tx=0x7fdc98009ca0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 -- 172.21.15.52:0/403021891 shutdown_connections 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 --2- 172.21.15.52:0/403021891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb010f440 0x7fdcb0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 --2- 172.21.15.52:0/403021891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb0074c70 0x7fdcb0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 --2- 172.21.15.52:0/403021891 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0075590 0x7fdcb010ef00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 -- 172.21.15.52:0/403021891 >> 172.21.15.52:0/403021891 conn(0x7fdcb0079ba0 msgr2=0x7fdcb0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 -- 172.21.15.52:0/403021891 shutdown_connections 2024-04-26T01:46:37.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.117+0000 7fdcb7000700 1 -- 172.21.15.52:0/403021891 wait complete. 2024-04-26T01:46:37.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.118+0000 7fdcb7000700 1 Processor -- start 2024-04-26T01:46:37.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 -- start start 2024-04-26T01:46:37.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb0075590 0x7fdcb019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 0x7fdcb019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb0114e00 con 0x7fdcb0075590 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb0114c80 con 0x7fdcb0074c70 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcb7000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb0114f80 con 0x7fdcb010f440 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcafe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdca7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb0075590 0x7fdcb019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcafe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58102/0 (socket says 172.21.15.52:58102) 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.119+0000 7fdcafe00700 1 -- 172.21.15.52:0/96854188 learned_addr learned my addr 172.21.15.52:0/96854188 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcb5000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 0x7fdcb019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 -- 172.21.15.52:0/96854188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 msgr2=0x7fdcb019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 0x7fdcb019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 -- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb0075590 msgr2=0x7fdcb019b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb0075590 0x7fdcb019b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 -- 172.21.15.52:0/96854188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb01a0380 con 0x7fdcb0074c70 2024-04-26T01:46:37.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcb5000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 0x7fdcb019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcafe00700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c370 tx=0x7fdca000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca000cea0 con 0x7fdcb0074c70 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.120+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdca00090d0 con 0x7fdcb0074c70 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.121+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb01a0610 con 0x7fdcb0074c70 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.121+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0004770 con 0x7fdcb0074c70 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.121+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb01a7ef0 con 0x7fdcb0074c70 2024-04-26T01:46:37.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.122+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb0069730 con 0x7fdcb0074c70 2024-04-26T01:46:37.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.124+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdca001a020 con 0x7fdcb0074c70 2024-04-26T01:46:37.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.125+0000 7fdcad600700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 0x7fdc940a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:37.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.125+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fdca00d0420 con 0x7fdcb0074c70 2024-04-26T01:46:37.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.126+0000 7fdca7400700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 0x7fdc940a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:37.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.126+0000 7fdca7400700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 0x7fdc940a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006010 tx=0x7fdc98005960 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:37.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.128+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdca0010040 con 0x7fdcb0074c70 2024-04-26T01:46:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:37 smithi052 ceph-mon[27851]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:37.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:37 smithi191 ceph-mon[32745]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:37.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.410+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcb002e140 con 0x7fdcb0074c70 2024-04-26T01:46:37.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.412+0000 7fdcad600700 1 -- 172.21.15.52:0/96854188 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fdca009ca90 con 0x7fdcb0074c70 2024-04-26T01:46:37.410 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:46:37.411 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:44:59.710897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:46:37.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 msgr2=0x7fdc940a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 0x7fdc940a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006010 tx=0x7fdc98005960 comp rx=0 tx=0).stop 2024-04-26T01:46:37.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 msgr2=0x7fdcb019b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:37.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c370 tx=0x7fdca000c680 comp rx=0 tx=0).stop 2024-04-26T01:46:37.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 shutdown_connections 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdc940a3e30 0x7fdc940a62f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcb010f440 0x7fdcb019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdcb0074c70 0x7fdcb019b370 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 --2- 172.21.15.52:0/96854188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdcb0075590 0x7fdcb019b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.415+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 >> 172.21.15.52:0/96854188 conn(0x7fdcb0079ba0 msgr2=0x7fdcb0101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.416+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 shutdown_connections 2024-04-26T01:46:37.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:37.416+0000 7fdcb7000700 1 -- 172.21.15.52:0/96854188 wait complete. 2024-04-26T01:46:37.415 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 25 2024-04-26T01:46:37.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:37 smithi183 ceph-mon[32752]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-04-26T01:46:38.202 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 263809e6 --release --await' 2024-04-26T01:46:38.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 263809e6 --release --await 2024-04-26T01:46:38.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:38 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/96854188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:38.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:38 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/96854188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:38.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:38 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/96854188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:46:38.617 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:46:39.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:39 smithi191 ceph-mon[32745]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:46:39.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:39 smithi183 ceph-mon[32752]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:46:39.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:39 smithi052 ceph-mon[27851]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T01:46:40.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 -- 172.21.15.52:0/497749351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4113c10 msgr2=0x7f86a4116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 --2- 172.21.15.52:0/497749351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4113c10 0x7f86a4116000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f869c00b0a0 tx=0x7f869c00b3b0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 -- 172.21.15.52:0/497749351 shutdown_connections 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 --2- 172.21.15.52:0/497749351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4113c10 0x7f86a4116000 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 --2- 172.21.15.52:0/497749351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 --2- 172.21.15.52:0/497749351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4075590 0x7f86a41136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.863+0000 7f86aba00700 1 -- 172.21.15.52:0/497749351 >> 172.21.15.52:0/497749351 conn(0x7f86a4079ba0 msgr2=0x7f86a4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.864+0000 7f86aba00700 1 -- 172.21.15.52:0/497749351 shutdown_connections 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.864+0000 7f86aba00700 1 -- 172.21.15.52:0/497749351 wait complete. 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 Processor -- start 2024-04-26T01:46:40.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 -- start start 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4075590 0x7f86a41a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 0x7f86a419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f86a4119250 con 0x7f86a4074c70 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41190d0 con 0x7f86a4113c10 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.865+0000 7f86aba00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41193d0 con 0x7f86a4075590 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41012/0 (socket says 172.21.15.52:41012) 2024-04-26T01:46:40.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9000700 1 -- 172.21.15.52:0/4105676272 learned_addr learned my addr 172.21.15.52:0/4105676272 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 0x7f86a419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a3e00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4075590 0x7f86a41a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 -- 172.21.15.52:0/4105676272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4075590 msgr2=0x7f86a41a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4075590 0x7f86a41a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 -- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 msgr2=0x7f86a41a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9a00700 1 -- 172.21.15.52:0/4105676272 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a419fd50 con 0x7f86a4113c10 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.866+0000 7f86a9000700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86a9a00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 0x7f86a419f5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d010 tx=0x7f869c004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86a1600700 1 -- 172.21.15.52:0/4105676272 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c021070 con 0x7f86a4113c10 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86a1600700 1 -- 172.21.15.52:0/4105676272 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f869c007e30 con 0x7f86a4113c10 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a419ffe0 con 0x7f86a4113c10 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86a1600700 1 -- 172.21.15.52:0/4105676272 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c0045a0 con 0x7f86a4113c10 2024-04-26T01:46:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.867+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a41ac300 con 0x7f86a4113c10 2024-04-26T01:46:40.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.871+0000 7f86a1600700 1 -- 172.21.15.52:0/4105676272 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f869c004740 con 0x7f86a4113c10 2024-04-26T01:46:40.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.872+0000 7f86a1600700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 0x7f868c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.872+0000 7f86b6363b80 1 Processor -- start 2024-04-26T01:46:40.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.872+0000 7f86a1600700 1 -- 172.21.15.52:0/4105676272 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f869c00f020 con 0x7f86a4113c10 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.872+0000 7f86b6363b80 1 -- start start 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.872+0000 7f86a9000700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 0x7f868c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc763660 0x5596dc76df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc76e4d0 0x5596dc7708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5596dc6b5f90 con 0x5596dc762d40 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5596dc65c830 con 0x5596dc763660 2024-04-26T01:46:40.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86b6363b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5596dc6b1f50 con 0x5596dc76e4d0 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86a3e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc763660 0x5596dc76df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41028/0 (socket says 172.21.15.52:41028) 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86a9000700 1 -- 172.21.15.52:0/3742420337 learned_addr learned my addr 172.21.15.52:0/3742420337 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.873+0000 7f86a9a00700 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc76e4d0 0x5596dc7708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 0x7f868c0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f86980115c0 tx=0x7f8698011980 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 -- 172.21.15.52:0/3742420337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc76e4d0 msgr2=0x5596dc7708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc76e4d0 0x5596dc7708c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 -- 172.21.15.52:0/3742420337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc763660 msgr2=0x5596dc76df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc763660 0x5596dc76df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 -- 172.21.15.52:0/3742420337 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596dc652780 con 0x5596dc762d40 2024-04-26T01:46:40.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.874+0000 7f86a9000700 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f869801d7e0 tx=0x7f869801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dbc9bf93c6af8c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.875+0000 7f8696000700 1 -- 172.21.15.52:0/3742420337 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698017040 con 0x5596dc762d40 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.875+0000 7f8696000700 1 -- 172.21.15.52:0/3742420337 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8698019050 con 0x5596dc762d40 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.875+0000 7f8696000700 1 -- 172.21.15.52:0/3742420337 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869800f6e0 con 0x5596dc762d40 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- 172.21.15.52:0/3742420337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 msgr2=0x5596dc763120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f869801d7e0 tx=0x7f869801c000 comp rx=0 tx=0).stop 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- 172.21.15.52:0/3742420337 shutdown_connections 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc76e4d0 0x5596dc7708c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc763120 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3742420337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc763660 0x5596dc76df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- 172.21.15.52:0/3742420337 >> 172.21.15.52:0/3742420337 conn(0x5596dc692050 msgr2=0x5596dc75f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- 172.21.15.52:0/3742420337 shutdown_connections 2024-04-26T01:46:40.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- 172.21.15.52:0/3742420337 wait complete. 2024-04-26T01:46:40.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.876+0000 7f86b6363b80 1 -- start start 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 Processor -- start 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc76dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc76aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc76b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5596dc770f60 con 0x5596dc762d40 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5596dc66f960 con 0x5596dc76e4d0 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.877+0000 7f86b6363b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5596dc7710e0 con 0x5596dc763660 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86b6363b80 10 client.0 _ll_register_callbacks cb 0x5596dc64e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc76dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc76aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc76dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41036/0 (socket says 172.21.15.52:41036) 2024-04-26T01:46:40.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a9000700 1 -- 172.21.15.52:0/464868552 learned_addr learned my addr 172.21.15.52:0/464868552 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:40.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a9a00700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc76b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 -- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 msgr2=0x5596dc76b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc76b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 -- 172.21.15.52:0/464868552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 msgr2=0x5596dc76dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc76dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.878+0000 7f86a3e00700 1 -- 172.21.15.52:0/464868552 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596dc6039d0 con 0x5596dc763660 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f86a3e00700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc76aeb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86900116c0 tx=0x7f86900119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31a2b14099e6446 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f868aa00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690004300 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f86900044a0 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8698019050 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8690004830 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f86b6363b80 10 client.25369 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.879+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596dc64cbf0 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8698019050 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8690019030 con 0x5596dc763660 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 10 client.25369 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 10 client.25369 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 20 client.25369 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc77dcf0 0x5596dc780100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 4 client.25369 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.880+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596dc6c3780 con 0x5596dc77dcf0 2024-04-26T01:46:40.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.881+0000 7f86a9000700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc77dcf0 0x5596dc780100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.881+0000 7f86a9000700 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc77dcf0 0x5596dc780100 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd12210a6e29c242 server_cookie=1caad2c1333a4f7e in_seq=0 out_seq=0 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.881+0000 7f868aa00700 10 client.25369 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.882+0000 7f868aa00700 1 -- 172.21.15.52:0/464868552 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596dc6c3780 con 0x5596dc77dcf0 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.882+0000 7f868aa00700 10 client.25369 handle_command_reply: tid=0 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.882+0000 7f86b6363b80 1 client.25369 shutdown 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.882+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 msgr2=0x5596dc76aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.882+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc76aeb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86900116c0 tx=0x7f86900119d0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 shutdown_connections 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc77dcf0 0x5596dc780100 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc76b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc76dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 --2- 172.21.15.52:0/464868552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc76aeb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 >> 172.21.15.52:0/464868552 conn(0x5596dc692050 msgr2=0x5596dc75ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f868aa00700 0 client.25369 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 shutdown_connections 2024-04-26T01:46:40.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 1 -- 172.21.15.52:0/464868552 wait complete. 2024-04-26T01:46:40.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.883+0000 7f86b6363b80 20 client.25369 trim_cache size 0 max 16384 2024-04-26T01:46:40.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 Processor -- start 2024-04-26T01:46:40.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 -- start start 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc776360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7768a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b6010 con 0x5596dc762d40 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b38a0 con 0x5596dc76e4d0 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86b6363b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b16f0 con 0x5596dc763660 2024-04-26T01:46:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.905+0000 7f86a9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41044/0 (socket says 172.21.15.52:41044) 2024-04-26T01:46:40.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9000700 1 -- 172.21.15.52:0/3175412411 learned_addr learned my addr 172.21.15.52:0/3175412411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:40.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7768a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a3e00700 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc776360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 -- 172.21.15.52:0/3175412411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 msgr2=0x5596dc776360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc776360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 -- 172.21.15.52:0/3175412411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 msgr2=0x5596dc779e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 -- 172.21.15.52:0/3175412411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596dc780780 con 0x5596dc76e4d0 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9000700 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.906+0000 7f86a9a00700 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7768a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f869c0cf2a0 tx=0x7f869c0be040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e839930a970c94e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f8695600700 1 -- 172.21.15.52:0/3175412411 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c09db80 con 0x5596dc76e4d0 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f8695600700 1 -- 172.21.15.52:0/3175412411 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f869c0a1f30 con 0x5596dc76e4d0 2024-04-26T01:46:40.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f8695600700 1 -- 172.21.15.52:0/3175412411 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c02d350 con 0x5596dc76e4d0 2024-04-26T01:46:40.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 -- 172.21.15.52:0/3175412411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 msgr2=0x5596dc7768a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7768a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f869c0cf2a0 tx=0x7f869c0be040 comp rx=0 tx=0).stop 2024-04-26T01:46:40.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 -- 172.21.15.52:0/3175412411 shutdown_connections 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7768a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc762d40 0x5596dc779e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.907+0000 7f86b6363b80 1 --2- 172.21.15.52:0/3175412411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc763660 0x5596dc776360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.908+0000 7f86b6363b80 1 -- 172.21.15.52:0/3175412411 >> 172.21.15.52:0/3175412411 conn(0x5596dc692050 msgr2=0x5596dc75e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.908+0000 7f86b6363b80 1 -- 172.21.15.52:0/3175412411 shutdown_connections 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.908+0000 7f86b6363b80 1 -- 172.21.15.52:0/3175412411 wait complete. 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.908+0000 7f86b6363b80 1 -- start start 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 Processor -- start 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc762d40 0x5596dc771510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7e4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b1d60 con 0x5596dc763660 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b31d0 con 0x5596dc76e4d0 2024-04-26T01:46:40.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f86a41b2430 con 0x5596dc762d40 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86b6363b80 10 client.0 _ll_register_callbacks cb 0x5596dc64e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86a3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86a3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41054/0 (socket says 172.21.15.52:41054) 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.909+0000 7f86a3e00700 1 -- 172.21.15.52:0/1249153527 learned_addr learned my addr 172.21.15.52:0/1249153527 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a9a00700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a9000700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc762d40 0x5596dc771510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 -- 172.21.15.52:0/1249153527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc762d40 msgr2=0x5596dc771510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc762d40 0x5596dc771510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 -- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 msgr2=0x5596dc7e4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7e4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 -- 172.21.15.52:0/1249153527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596dc77b6c0 con 0x5596dc763660 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f86a3e00700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5596dc76c480 tx=0x7f869001a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbca5acad50e01e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.910+0000 7f868aa00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690030020 con 0x5596dc763660 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f869c0a1f30 con 0x5596dc763660 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f8690004590 con 0x5596dc763660 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f869001aa40 con 0x5596dc763660 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f86b6363b80 10 client.15282 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:40.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596dc7e5900 con 0x5596dc763660 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f869c0a1f30 con 0x5596dc763660 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.911+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8690019040 con 0x5596dc763660 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 10 client.15282 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 10 client.15282 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 20 client.15282 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc7e6030 0x5596dc7e8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 4 client.15282 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "263809e6", "release": true, "await": true}] 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "263809e6", "release": true, "await": true}) -- 0x5596dc6bd820 con 0x5596dc7e6030 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.912+0000 7f86a9000700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc7e6030 0x5596dc7e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.913+0000 7f86a9000700 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc7e6030 0x5596dc7e8450 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde9933e6be05b8f server_cookie=10be3f72b176416c in_seq=0 out_seq=0 2024-04-26T01:46:40.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.913+0000 7f868aa00700 10 client.15282 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f868aa00700 1 -- 172.21.15.52:0/1249153527 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5596dc6bd820 con 0x5596dc7e6030 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f868aa00700 10 client.15282 handle_command_reply: tid=0 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 client.15282 shutdown 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 msgr2=0x5596dc771a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5596dc76c480 tx=0x7f869001a620 comp rx=0 tx=0).stop 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 shutdown_connections 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5596dc7e6030 0x5596dc7e8450 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5596dc76e4d0 0x5596dc7e4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5596dc762d40 0x5596dc771510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 --2- 172.21.15.52:0/1249153527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5596dc763660 0x5596dc771a50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f868aa00700 0 client.15282 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.916+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 >> 172.21.15.52:0/1249153527 conn(0x5596dc692050 msgr2=0x5596dc75e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.917+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 shutdown_connections 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.917+0000 7f86b6363b80 1 -- 172.21.15.52:0/1249153527 wait complete. 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.917+0000 7f86b6363b80 20 client.15282 trim_cache size 0 max 16384 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 25, 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:46:40.916 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 4, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "263809e6": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 4, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 85.6, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:46:40.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:46:40.918 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 msgr2=0x7f868c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 0x7f868c0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f86980115c0 tx=0x7f8698011980 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 msgr2=0x7f86a419f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 0x7f86a419f5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d010 tx=0x7f869c004c80 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 shutdown_connections 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f868c0a3db0 0x7f868c0a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a4113c10 0x7f86a419f5f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86a4074c70 0x7f86a41a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 --2- 172.21.15.52:0/4105676272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86a4075590 0x7f86a41a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:40.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 >> 172.21.15.52:0/4105676272 conn(0x7f86a4079ba0 msgr2=0x7f86a410f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:40.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 shutdown_connections 2024-04-26T01:46:40.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:46:40.918+0000 7f86aba00700 1 -- 172.21.15.52:0/4105676272 wait complete. 2024-04-26T01:46:40.929 INFO:tasks.workunit.client.0.smithi052.stderr:+ dbench 1 2024-04-26T01:46:40.936 INFO:tasks.workunit.client.0.smithi052.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T01:46:40.936 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-04-26T01:46:40.936 INFO:tasks.workunit.client.0.smithi052.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T01:46:40.955 INFO:tasks.workunit.client.0.smithi052.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-26T01:46:41.055 INFO:tasks.workunit.client.0.smithi052.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-26T01:46:41.056 INFO:tasks.workunit.client.0.smithi052.stdout:releasing clients 2024-04-26T01:46:41.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:41 smithi052 ceph-mon[27851]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T01:46:41.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:41 smithi191 ceph-mon[32745]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T01:46:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:41 smithi183 ceph-mon[32752]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T01:46:41.997 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '263809e6', seconds elapsed: 41.2 2024-04-26T01:46:42.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 1588 26.15 MB/sec warmup 1 sec latency 16.336 ms 2024-04-26T01:46:43.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 3928 20.03 MB/sec warmup 2 sec latency 9.642 ms 2024-04-26T01:46:43.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:43 smithi191 ceph-mon[32745]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:43 smithi183 ceph-mon[32752]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:43 smithi052 ceph-mon[27851]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-26T01:46:44.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 5894 15.65 MB/sec warmup 3 sec latency 13.062 ms 2024-04-26T01:46:45.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 7608 14.80 MB/sec warmup 4 sec latency 25.982 ms 2024-04-26T01:46:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:45 smithi191 ceph-mon[32745]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-26T01:46:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:45 smithi183 ceph-mon[32752]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-26T01:46:45.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:45 smithi052 ceph-mon[27851]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-26T01:46:46.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 9601 13.39 MB/sec warmup 5 sec latency 12.401 ms 2024-04-26T01:46:47.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 11216 13.05 MB/sec warmup 6 sec latency 15.694 ms 2024-04-26T01:46:47.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:47 smithi191 ceph-mon[32745]: pgmap v56: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 464 KiB/s wr, 61 op/s 2024-04-26T01:46:47.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:47 smithi183 ceph-mon[32752]: pgmap v56: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 464 KiB/s wr, 61 op/s 2024-04-26T01:46:47.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:47 smithi052 ceph-mon[27851]: pgmap v56: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 464 KiB/s wr, 61 op/s 2024-04-26T01:46:48.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 13131 12.30 MB/sec warmup 7 sec latency 13.683 ms 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.300+0000 7fbf13c00640 1 -- 172.21.15.52:0/968059135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 msgr2=0x7fbf0c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.300+0000 7fbf13c00640 1 --2- 172.21.15.52:0/968059135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c10ee40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009ca0 tx=0x7fbf0402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 -- 172.21.15.52:0/968059135 shutdown_connections 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 --2- 172.21.15.52:0/968059135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c10ee40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 --2- 172.21.15.52:0/968059135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c102b50 0x7fbf0c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 --2- 172.21.15.52:0/968059135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 -- 172.21.15.52:0/968059135 >> 172.21.15.52:0/968059135 conn(0x7fbf0c0793e0 msgr2=0x7fbf0c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 -- 172.21.15.52:0/968059135 shutdown_connections 2024-04-26T01:46:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.301+0000 7fbf13c00640 1 -- 172.21.15.52:0/968059135 wait complete. 2024-04-26T01:46:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.302+0000 7fbf13c00640 1 Processor -- start 2024-04-26T01:46:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.302+0000 7fbf13c00640 1 -- start start 2024-04-26T01:46:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 0x7fbf0c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c112210 con 0x7fbf0c19a660 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c112090 con 0x7fbf0c1021a0 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf13c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c112390 con 0x7fbf0c107270 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf11200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 0x7fbf0c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf11200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53188/0 (socket says 172.21.15.52:53188) 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.303+0000 7fbf11200640 1 -- 172.21.15.52:0/202528171 learned_addr learned my addr 172.21.15.52:0/202528171 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 -- 172.21.15.52:0/202528171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 msgr2=0x7fbf0c1a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c1a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 -- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 msgr2=0x7fbf0c19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 0x7fbf0c19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 -- 172.21.15.52:0/202528171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c19b260 con 0x7fbf0c1021a0 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 0x7fbf0c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf11200640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e9f0 tx=0x7fbf0000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf09600640 1 -- 172.21.15.52:0/202528171 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000cd80 con 0x7fbf0c1021a0 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf09600640 1 -- 172.21.15.52:0/202528171 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf000108e0 con 0x7fbf0c1021a0 2024-04-26T01:46:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c1a7340 con 0x7fbf0c1021a0 2024-04-26T01:46:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.304+0000 7fbf09600640 1 -- 172.21.15.52:0/202528171 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00018980 con 0x7fbf0c1021a0 2024-04-26T01:46:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.305+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c1a77a0 con 0x7fbf0c1021a0 2024-04-26T01:46:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 Processor -- start 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 -- start start 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf09600640 1 -- 172.21.15.52:0/202528171 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbf00018b20 con 0x7fbf0c1021a0 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e96cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260e99600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556260de9900 con 0x556260e97210 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556260d8fef0 con 0x556260e83760 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.308+0000 7fbf169b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556260de58c0 con 0x556260e84080 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53198/0 (socket says 172.21.15.52:53198) 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 -- 172.21.15.52:0/3032978984 learned_addr learned my addr 172.21.15.52:0/3032978984 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11200640 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e96cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf0be00640 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260e99600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 -- 172.21.15.52:0/3032978984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 msgr2=0x556260e96cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e96cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 -- 172.21.15.52:0/3032978984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 msgr2=0x556260e99600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260e99600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 -- 172.21.15.52:0/3032978984 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556260afc210 con 0x556260e83760 2024-04-26T01:46:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.309+0000 7fbf11c00640 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf04035000 tx=0x7fbf04009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494bb7c96ef08e87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.310+0000 7fbefe000640 1 -- 172.21.15.52:0/3032978984 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04031cf0 con 0x556260e83760 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.310+0000 7fbefe000640 1 -- 172.21.15.52:0/3032978984 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04036040 con 0x556260e83760 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.310+0000 7fbefe000640 1 -- 172.21.15.52:0/3032978984 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040042a0 con 0x556260e83760 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.310+0000 7fbf09600640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 0x7fbef40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 -- 172.21.15.52:0/3032978984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 msgr2=0x556260e83b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf04035000 tx=0x7fbf04009870 comp rx=0 tx=0).stop 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf09600640 1 -- 172.21.15.52:0/202528171 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbf000d0be0 con 0x7fbf0c1021a0 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf11200640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 0x7fbef40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 -- 172.21.15.52:0/3032978984 shutdown_connections 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260e99600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e96cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 --2- 172.21.15.52:0/3032978984 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260e83b40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 -- 172.21.15.52:0/3032978984 >> 172.21.15.52:0/3032978984 conn(0x556260e7fd30 msgr2=0x556260e80110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 -- 172.21.15.52:0/3032978984 shutdown_connections 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.311+0000 7fbf169b3740 1 -- 172.21.15.52:0/3032978984 wait complete. 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf11200640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 0x7fbef40a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf00098fb0 tx=0x7fbf00000bc0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 -- start start 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 Processor -- start 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 0x556260e941a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e97210 0x556260e94c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556260e99d80 con 0x556260e83760 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556260da3310 con 0x556260e84080 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.312+0000 7fbf169b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556260e99f00 con 0x556260e97210 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf169b3740 10 client.0 _ll_register_callbacks cb 0x556260d71cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53210/0 (socket says 172.21.15.52:53210) 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 -- 172.21.15.52:0/2992869028 learned_addr learned my addr 172.21.15.52:0/2992869028 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 0x556260e941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e97210 0x556260e94c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 -- 172.21.15.52:0/2992869028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e97210 msgr2=0x556260e94c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e97210 0x556260e94c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 -- 172.21.15.52:0/2992869028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 msgr2=0x556260e941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 0x556260e941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf0be00640 1 -- 172.21.15.52:0/2992869028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556260bc8620 con 0x556260e84080 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.313+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 0x556260e941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbef800b940 tx=0x7fbef800be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=594336e832883ade server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbef2a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef801b440 con 0x556260e84080 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf04036040 con 0x556260e84080 2024-04-26T01:46:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbef801b5e0 con 0x556260e84080 2024-04-26T01:46:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbef801b7e0 con 0x556260e84080 2024-04-26T01:46:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbf169b3740 10 client.25376 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556260ea6140 con 0x556260e84080 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.314+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04036040 con 0x556260e84080 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fbef8019020 con 0x556260e84080 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 10 client.25376 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 10 client.25376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 20 client.25376 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260ea7600 0x556260ea9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 4 client.25376 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556260dfd8b0 con 0x556260ea7600 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.315+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260ea7600 0x556260ea9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.316+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260ea7600 0x556260ea9a10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a85fc8fdafa6723 server_cookie=8bc0c2a7895655d0 in_seq=0 out_seq=0 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.316+0000 7fbef2a00640 10 client.25376 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.317+0000 7fbef2a00640 1 -- 172.21.15.52:0/2992869028 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556260dfd8b0 con 0x556260ea7600 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.317+0000 7fbef2a00640 10 client.25376 handle_command_reply: tid=0 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 client.25376 shutdown 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 msgr2=0x556260e946e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbef800b940 tx=0x7fbef800be00 comp rx=0 tx=0).stop 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 shutdown_connections 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260ea7600 0x556260ea9a10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e97210 0x556260e94c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e946e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2992869028 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e83760 0x556260e941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbef2a00640 0 client.25376 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 >> 172.21.15.52:0/2992869028 conn(0x556260e7fd30 msgr2=0x556260e85a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 shutdown_connections 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 1 -- 172.21.15.52:0/2992869028 wait complete. 2024-04-26T01:46:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.318+0000 7fbf169b3740 20 client.25376 trim_cache size 0 max 16384 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 Processor -- start 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 -- start start 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260ea2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e7ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260eaf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1ae150 con 0x556260e97210 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1aba20 con 0x556260e83760 2024-04-26T01:46:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.335+0000 7fbf169b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1a9890 con 0x556260e84080 2024-04-26T01:46:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260eaf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e7ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260eaf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41222/0 (socket says 172.21.15.52:41222) 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf11c00640 1 -- 172.21.15.52:0/1312115539 learned_addr learned my addr 172.21.15.52:0/1312115539 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf11c00640 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260ea2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 -- 172.21.15.52:0/1312115539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 msgr2=0x556260ea2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260ea2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 -- 172.21.15.52:0/1312115539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 msgr2=0x556260eaf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260eaf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.336+0000 7fbf0be00640 1 -- 172.21.15.52:0/1312115539 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556260e806b0 con 0x556260e84080 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.337+0000 7fbf0be00640 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e7ed10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbef8011cd0 tx=0x7fbef8005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc5113ea6ca38fa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.337+0000 7fbefd600640 1 -- 172.21.15.52:0/1312115539 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8028070 con 0x556260e84080 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.337+0000 7fbefd600640 1 -- 172.21.15.52:0/1312115539 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbef8011d50 con 0x556260e84080 2024-04-26T01:46:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.337+0000 7fbefd600640 1 -- 172.21.15.52:0/1312115539 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8024400 con 0x556260e84080 2024-04-26T01:46:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 -- 172.21.15.52:0/1312115539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 msgr2=0x556260e7ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e7ed10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbef8011cd0 tx=0x7fbef8005730 comp rx=0 tx=0).stop 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 -- 172.21.15.52:0/1312115539 shutdown_connections 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260eaf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e84080 0x556260e7ed10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 --2- 172.21.15.52:0/1312115539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e83760 0x556260ea2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 -- 172.21.15.52:0/1312115539 >> 172.21.15.52:0/1312115539 conn(0x556260e7fd30 msgr2=0x556260e7e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 -- 172.21.15.52:0/1312115539 shutdown_connections 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.338+0000 7fbf169b3740 1 -- 172.21.15.52:0/1312115539 wait complete. 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 -- start start 2024-04-26T01:46:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 Processor -- start 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e83760 0x556260e9d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260f0cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1a9ee0 con 0x556260e97210 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1ab350 con 0x556260e84080 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.339+0000 7fbf169b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1aa5b0 con 0x556260e83760 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf169b3740 10 client.0 _ll_register_callbacks cb 0x7fbef8004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53230/0 (socket says 172.21.15.52:53230) 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 -- 172.21.15.52:0/2345039358 learned_addr learned my addr 172.21.15.52:0/2345039358 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:46:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf11c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e83760 0x556260e9d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260f0cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 -- 172.21.15.52:0/2345039358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e83760 msgr2=0x556260e9d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e83760 0x556260e9d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 -- 172.21.15.52:0/2345039358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 msgr2=0x556260f0cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260f0cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.340+0000 7fbf0be00640 1 -- 172.21.15.52:0/2345039358 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556260f0d130 con 0x556260e84080 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbf0be00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbef80057c0 tx=0x7fbef80039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1237d58bd69378f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbef2a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef803b060 con 0x556260e84080 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbef8011d50 con 0x556260e84080 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbef8011440 con 0x556260e84080 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbef8028040 con 0x556260e84080 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbf169b3740 10 client.25382 fetch_fsmap learned FSMap version 25 2024-04-26T01:46:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.341+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556260f0e440 con 0x556260e84080 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbef8011d50 con 0x556260e84080 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fbef8019040 con 0x556260e84080 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 10 client.25382 fetch_fsmap finished waiting for FSMap version 25 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 10 client.25382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 20 client.25382 populate_metadata read hostname 'smithi052' 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260f0eb90 0x556260f10fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 4 client.25382 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556260df7320 con 0x556260f0eb90 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.342+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260f0eb90 0x556260f10fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:46:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.343+0000 7fbf11c00640 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260f0eb90 0x556260f10fb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8088bba91fb9d5f server_cookie=7c3ac763fd079694 in_seq=0 out_seq=0 2024-04-26T01:46:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.343+0000 7fbef2a00640 10 client.25382 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbef2a00640 1 -- 172.21.15.52:0/2345039358 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556260df7320 con 0x556260f0eb90 2024-04-26T01:46:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbef2a00640 10 client.25382 handle_command_reply: tid=0 2024-04-26T01:46:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 client.25382 shutdown 2024-04-26T01:46:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 msgr2=0x556260e9d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbef80057c0 tx=0x7fbef80039c0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 shutdown_connections 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556260f0eb90 0x556260f10fb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556260e97210 0x556260f0cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556260e84080 0x556260e9d8b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 --2- 172.21.15.52:0/2345039358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556260e83760 0x556260e9d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbef2a00640 0 client.25382 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 >> 172.21.15.52:0/2345039358 conn(0x556260e7fd30 msgr2=0x556260e7ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 shutdown_connections 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.421+0000 7fbf169b3740 1 -- 172.21.15.52:0/2345039358 wait complete. 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.422+0000 7fbf169b3740 20 client.25382 trim_cache size 0 max 16384 2024-04-26T01:46:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:46:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:46:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:46:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:46:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.422+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 msgr2=0x7fbef40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.422+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 0x7fbef40a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf00098fb0 tx=0x7fbf00000bc0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.422+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 msgr2=0x7fbf0c1a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.422+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e9f0 tx=0x7fbf0000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 shutdown_connections 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbef40a3e40 0x7fbef40a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf0c19a660 0x7fbf0c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf0c107270 0x7fbf0c1a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 --2- 172.21.15.52:0/202528171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf0c1021a0 0x7fbf0c1a05a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 >> 172.21.15.52:0/202528171 conn(0x7fbf0c0793e0 msgr2=0x7fbf0c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 shutdown_connections 2024-04-26T01:46:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:46:48.423+0000 7fbf13c00640 1 -- 172.21.15.52:0/202528171 wait complete. 2024-04-26T01:46:49.056 INFO:tasks.workunit.client.0.smithi052.stdout: 1 14804 12.18 MB/sec warmup 8 sec latency 26.953 ms 2024-04-26T01:46:49.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:49 smithi191 ceph-mon[32745]: pgmap v57: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 464 KiB/s wr, 58 op/s 2024-04-26T01:46:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:49 smithi183 ceph-mon[32752]: pgmap v57: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 464 KiB/s wr, 58 op/s 2024-04-26T01:46:49.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:49 smithi052 ceph-mon[27851]: pgmap v57: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 464 KiB/s wr, 58 op/s 2024-04-26T01:46:50.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 16803 11.72 MB/sec warmup 9 sec latency 11.988 ms 2024-04-26T01:46:51.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 18429 11.68 MB/sec warmup 10 sec latency 19.456 ms 2024-04-26T01:46:51.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:51 smithi191 ceph-mon[32745]: pgmap v58: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 463 KiB/s wr, 58 op/s 2024-04-26T01:46:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:51 smithi183 ceph-mon[32752]: pgmap v58: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 463 KiB/s wr, 58 op/s 2024-04-26T01:46:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:51 smithi052 ceph-mon[27851]: pgmap v58: 97 pgs: 97 active+clean; 6.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 463 KiB/s wr, 58 op/s 2024-04-26T01:46:52.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 18722 10.67 MB/sec warmup 11 sec latency 643.680 ms 2024-04-26T01:46:52.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:52 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:52.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:52 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:52.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:52 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:53.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 18723 9.78 MB/sec warmup 12 sec latency 1635.120 ms 2024-04-26T01:46:53.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:53 smithi191 ceph-mon[32745]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 925 KiB/s wr, 175 op/s 2024-04-26T01:46:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:53 smithi183 ceph-mon[32752]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 925 KiB/s wr, 175 op/s 2024-04-26T01:46:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:53 smithi052 ceph-mon[27851]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 925 KiB/s wr, 175 op/s 2024-04-26T01:46:54.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 21048 10.07 MB/sec warmup 13 sec latency 12.847 ms 2024-04-26T01:46:55.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 21570 9.67 MB/sec warmup 14 sec latency 370.471 ms 2024-04-26T01:46:55.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:55 smithi191 ceph-mon[32745]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 925 KiB/s wr, 173 op/s 2024-04-26T01:46:55.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:55 smithi183 ceph-mon[32752]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 925 KiB/s wr, 173 op/s 2024-04-26T01:46:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:55 smithi052 ceph-mon[27851]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 925 KiB/s wr, 173 op/s 2024-04-26T01:46:56.057 INFO:tasks.workunit.client.0.smithi052.stdout: 1 21570 9.03 MB/sec warmup 15 sec latency 1370.628 ms 2024-04-26T01:46:57.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 21653 8.46 MB/sec warmup 16 sec latency 1505.667 ms 2024-04-26T01:46:57.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:57 smithi183 ceph-mon[32752]: pgmap v61: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-26T01:46:57.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:57 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:57 smithi052 ceph-mon[27851]: pgmap v61: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-26T01:46:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:57 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:57 smithi191 ceph-mon[32745]: pgmap v61: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-26T01:46:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:57 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:46:58.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 21669 7.98 MB/sec warmup 17 sec latency 1583.901 ms 2024-04-26T01:46:59.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 22151 7.57 MB/sec warmup 18 sec latency 21.863 ms 2024-04-26T01:46:59.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:46:59 smithi183 ceph-mon[32752]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:46:59.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:46:59 smithi052 ceph-mon[27851]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:46:59.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:46:59 smithi191 ceph-mon[32745]: pgmap v62: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:47:00.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 23671 7.51 MB/sec warmup 19 sec latency 32.760 ms 2024-04-26T01:47:01.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 25004 7.68 MB/sec warmup 20 sec latency 13.314 ms 2024-04-26T01:47:01.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:01 smithi183 ceph-mon[32752]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:47:01.453 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:01 smithi191 ceph-mon[32745]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:47:01.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:01 smithi052 ceph-mon[27851]: pgmap v63: 97 pgs: 97 active+clean; 11 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 633 KiB/s wr, 145 op/s 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 -- 172.21.15.52:0/1166651502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f30102b50 msgr2=0x7f0f30106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1166651502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f30102b50 0x7f0f30106b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009900 tx=0x7f0f1802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 -- 172.21.15.52:0/1166651502 shutdown_connections 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1166651502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f30107270 0x7f0f3010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1166651502 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f30102b50 0x7f0f30106b90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1166651502 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f301021a0 0x7f0f30102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 -- 172.21.15.52:0/1166651502 >> 172.21.15.52:0/1166651502 conn(0x7f0f300793e0 msgr2=0x7f0f300797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 -- 172.21.15.52:0/1166651502 shutdown_connections 2024-04-26T01:47:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.575+0000 7f0f34c00640 1 -- 172.21.15.52:0/1166651502 wait complete. 2024-04-26T01:47:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.576+0000 7f0f34c00640 1 Processor -- start 2024-04-26T01:47:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 -- start start 2024-04-26T01:47:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30107270 0x7f0f30198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f3019cf60 0x7f0f3019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30112200 con 0x7f0f3019cf60 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30112080 con 0x7f0f30107270 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f34c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30112380 con 0x7f0f301021a0 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33760/0 (socket says 172.21.15.52:33760) 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f2de00640 1 -- 172.21.15.52:0/1174284452 learned_addr learned my addr 172.21.15.52:0/1174284452 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.577+0000 7f0f2e800640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f3019cf60 0x7f0f3019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f25400640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30107270 0x7f0f30198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 -- 172.21.15.52:0/1174284452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30107270 msgr2=0x7f0f30198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30107270 0x7f0f30198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 -- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f3019cf60 msgr2=0x7f0f3019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f3019cf60 0x7f0f3019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 -- 172.21.15.52:0/1174284452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3019da50 con 0x7f0f301021a0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.578+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e9f0 tx=0x7f0f2000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.579+0000 7f0f27400640 1 -- 172.21.15.52:0/1174284452 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000cd80 con 0x7f0f301021a0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.579+0000 7f0f27400640 1 -- 172.21.15.52:0/1174284452 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200054d0 con 0x7f0f301021a0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.579+0000 7f0f27400640 1 -- 172.21.15.52:0/1174284452 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20010690 con 0x7f0f301021a0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.579+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3019dd40 con 0x7f0f301021a0 2024-04-26T01:47:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.579+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f30074420 con 0x7f0f301021a0 2024-04-26T01:47:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.582+0000 7f0f37980740 1 Processor -- start 2024-04-26T01:47:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f27400640 1 -- 172.21.15.52:0/1174284452 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0f20004e90 con 0x7f0f301021a0 2024-04-26T01:47:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 -- start start 2024-04-26T01:47:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9c9820 0x56296a9c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56296a92f900 con 0x56296a9d4a00 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56296a8d5ef0 con 0x56296a9c9820 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f37980740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56296a92b8c0 con 0x56296a9ca140 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33772/0 (socket says 172.21.15.52:33772) 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f2de00640 1 -- 172.21.15.52:0/1458274310 learned_addr learned my addr 172.21.15.52:0/1458274310 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f25400640 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.583+0000 7f0f2e800640 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9c9820 0x56296a9c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 -- 172.21.15.52:0/1458274310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9c9820 msgr2=0x56296a9c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9c9820 0x56296a9c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 -- 172.21.15.52:0/1458274310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 msgr2=0x56296a9d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 -- 172.21.15.52:0/1458274310 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56296a642210 con 0x56296a9ca140 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f2de00640 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f200ce230 tx=0x7f0f200d4c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93e507bcf32c56e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f27400640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 0x7f0f140a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f1f000640 1 -- 172.21.15.52:0/1458274310 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200e0020 con 0x56296a9ca140 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.584+0000 7f0f1f000640 1 -- 172.21.15.52:0/1458274310 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2009d240 con 0x56296a9ca140 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f2e800640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 0x7f0f140a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f27400640 1 -- 172.21.15.52:0/1174284452 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0f200d0bd0 con 0x7f0f301021a0 2024-04-26T01:47:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f1f000640 1 -- 172.21.15.52:0/1458274310 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200b0520 con 0x56296a9ca140 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 -- 172.21.15.52:0/1458274310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 msgr2=0x56296a9d44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f200ce230 tx=0x7f0f200d4c70 comp rx=0 tx=0).stop 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 -- 172.21.15.52:0/1458274310 shutdown_connections 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d44c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 --2- 172.21.15.52:0/1458274310 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9c9820 0x56296a9c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.585+0000 7f0f37980740 1 -- 172.21.15.52:0/1458274310 >> 172.21.15.52:0/1458274310 conn(0x56296a9c5d30 msgr2=0x56296a9c6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.586+0000 7f0f2e800640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 0x7f0f140a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005fd0 tx=0x7f0f28009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.586+0000 7f0f37980740 1 -- 172.21.15.52:0/1458274310 shutdown_connections 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.586+0000 7f0f37980740 1 -- 172.21.15.52:0/1458274310 wait complete. 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.586+0000 7f0f37980740 1 -- start start 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.586+0000 7f0f37980740 1 Processor -- start 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9c9820 0x56296a9d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 0x56296a9d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56296a9d7630 con 0x56296a9c9820 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56296a8e9310 con 0x56296a9d4a00 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56296a9d77b0 con 0x56296a9ca140 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f37980740 10 client.0 _ll_register_callbacks cb 0x56296a8b7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9c9820 0x56296a9d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f25400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f25400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33788/0 (socket says 172.21.15.52:33788) 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f25400640 1 -- 172.21.15.52:0/71770146 learned_addr learned my addr 172.21.15.52:0/71770146 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f2e800640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 0x56296a9d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.587+0000 7f0f2e800640 1 -- 172.21.15.52:0/71770146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 msgr2=0x56296a9d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f2e800640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f2e800640 1 -- 172.21.15.52:0/71770146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9c9820 msgr2=0x56296a9d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f2e800640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9c9820 0x56296a9d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f2e800640 1 -- 172.21.15.52:0/71770146 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56296a70e620 con 0x56296a9d4a00 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f25400640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f2e800640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 0x56296a9d2320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f28003f20 tx=0x7f0f28004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4833a7b74df7443 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f13400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28004b20 con 0x56296a9d4a00 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0f28004cc0 con 0x56296a9d4a00 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.588+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2009d240 con 0x56296a9d4a00 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0f2800f750 con 0x56296a9d4a00 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 10 client.25394 fetch_fsmap learned FSMap version 27 2024-04-26T01:47:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56296a9e38a0 con 0x56296a9d4a00 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f2009d240 con 0x56296a9d4a00 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0f28026070 con 0x56296a9d4a00 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 10 client.25394 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 10 client.25394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 20 client.25394 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296a9e4e70 0x56296a9e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 4 client.25394 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.589+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56296a9438b0 con 0x56296a9e4e70 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.590+0000 7f0f2de00640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296a9e4e70 0x56296a9e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.590+0000 7f0f2de00640 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296a9e4e70 0x56296a9e72a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91c7a0034a98eff server_cookie=e3efbd339121c754 in_seq=0 out_seq=0 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.590+0000 7f0f13400640 10 client.25394 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f13400640 1 -- 172.21.15.52:0/71770146 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56296a9438b0 con 0x56296a9e4e70 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f13400640 10 client.25394 handle_command_reply: tid=0 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 client.25394 shutdown 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 msgr2=0x56296a9d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 0x56296a9d2320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f28003f20 tx=0x7f0f28004a40 comp rx=0 tx=0).stop 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 shutdown_connections 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296a9e4e70 0x56296a9e72a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9d4a00 0x56296a9d2320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9ca140 0x56296a9d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 --2- 172.21.15.52:0/71770146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9c9820 0x56296a9d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f13400640 0 client.25394 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.591+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 >> 172.21.15.52:0/71770146 conn(0x56296a9c5d30 msgr2=0x56296a9cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.592+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 shutdown_connections 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.592+0000 7f0f37980740 1 -- 172.21.15.52:0/71770146 wait complete. 2024-04-26T01:47:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.592+0000 7f0f37980740 20 client.25394 trim_cache size 0 max 16384 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 Processor -- start 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 -- start start 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9e0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301b2580 con 0x56296a9d4a00 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301afe50 con 0x56296a9ca140 2024-04-26T01:47:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.608+0000 7f0f37980740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301adca0 con 0x56296a9c9820 2024-04-26T01:47:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9e0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f25400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f25400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53178/0 (socket says 172.21.15.52:53178) 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f25400640 1 -- 172.21.15.52:0/3756489775 learned_addr learned my addr 172.21.15.52:0/3756489775 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 -- 172.21.15.52:0/3756489775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 msgr2=0x56296a9d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 -- 172.21.15.52:0/3756489775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 msgr2=0x56296a9ecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9ecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2de00640 1 -- 172.21.15.52:0/3756489775 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56296a9c1690 con 0x56296a9c9820 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.609+0000 7f0f2e800640 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.610+0000 7f0f2de00640 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9e0540 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f200d9690 tx=0x7f0f20004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e2d009ac053eb99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.610+0000 7f0f1e600640 1 -- 172.21.15.52:0/3756489775 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200b1990 con 0x56296a9c9820 2024-04-26T01:47:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.610+0000 7f0f1e600640 1 -- 172.21.15.52:0/3756489775 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200d4450 con 0x56296a9c9820 2024-04-26T01:47:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.610+0000 7f0f1e600640 1 -- 172.21.15.52:0/3756489775 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200b1d10 con 0x56296a9c9820 2024-04-26T01:47:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- 172.21.15.52:0/3756489775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 msgr2=0x56296a9e0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9e0540 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f200d9690 tx=0x7f0f20004a90 comp rx=0 tx=0).stop 2024-04-26T01:47:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- 172.21.15.52:0/3756489775 shutdown_connections 2024-04-26T01:47:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296a9ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 --2- 172.21.15.52:0/3756489775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9e0540 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- 172.21.15.52:0/3756489775 >> 172.21.15.52:0/3756489775 conn(0x56296a9c5d30 msgr2=0x56296a9d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- 172.21.15.52:0/3756489775 shutdown_connections 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- 172.21.15.52:0/3756489775 wait complete. 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.611+0000 7f0f37980740 1 -- start start 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 Processor -- start 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ae310 con 0x56296a9d4a00 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301af780 con 0x56296a9ca140 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301ae9e0 con 0x56296a9c9820 2024-04-26T01:47:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f37980740 10 client.0 _ll_register_callbacks cb 0x7f0f28003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f2e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.612+0000 7f0f2e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57920/0 (socket says 172.21.15.52:57920) 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 -- 172.21.15.52:0/965432047 learned_addr learned my addr 172.21.15.52:0/965432047 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33804/0 (socket says 172.21.15.52:33804) 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f25400640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 -- 172.21.15.52:0/965432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 msgr2=0x56296a9d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 -- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 msgr2=0x56296a9d8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 -- 172.21.15.52:0/965432047 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56296aa49980 con 0x56296a9d4a00 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2de00640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f2e800640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009fd0 tx=0x7f0f28005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8cec1b31912319 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f13400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.613+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28034070 con 0x56296a9d4a00 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0f2800fd00 con 0x56296a9d4a00 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f200d4450 con 0x56296a9d4a00 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0f2801e530 con 0x56296a9d4a00 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 10 client.15318 fetch_fsmap learned FSMap version 27 2024-04-26T01:47:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56296aa4ac70 con 0x56296a9d4a00 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f200d4450 con 0x56296a9d4a00 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0f28026040 con 0x56296a9d4a00 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 10 client.15318 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 10 client.15318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.614+0000 7f0f37980740 20 client.15318 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296aa4b400 0x56296aa4d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f37980740 4 client.15318 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56296a93d320 con 0x56296aa4b400 2024-04-26T01:47:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f2de00640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296aa4b400 0x56296aa4d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f2de00640 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296aa4b400 0x56296aa4d820 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2208c56dd390c05c server_cookie=195393c996c93060 in_seq=0 out_seq=0 2024-04-26T01:47:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.615+0000 7f0f13400640 10 client.15318 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.657+0000 7f0f13400640 1 -- 172.21.15.52:0/965432047 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56296a93d320 con 0x56296aa4b400 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.657+0000 7f0f13400640 10 client.15318 handle_command_reply: tid=0 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 client.15318 shutdown 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 msgr2=0x56296aa49410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009fd0 tx=0x7f0f28005c40 comp rx=0 tx=0).stop 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 shutdown_connections 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56296aa4b400 0x56296aa4d820 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56296a9d4a00 0x56296aa49410 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56296a9ca140 0x56296a9d8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 --2- 172.21.15.52:0/965432047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56296a9c9820 0x56296a9d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f13400640 0 client.15318 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 >> 172.21.15.52:0/965432047 conn(0x56296a9c5d30 msgr2=0x56296a9cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 shutdown_connections 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 1 -- 172.21.15.52:0/965432047 wait complete. 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.658+0000 7f0f37980740 20 client.15318 trim_cache size 0 max 16384 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:47:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 msgr2=0x7f0f140a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 0x7f0f140a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005fd0 tx=0x7f0f28009500 comp rx=0 tx=0).stop 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 msgr2=0x7f0f301986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e9f0 tx=0x7f0f2000edf0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 shutdown_connections 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f140a3e40 0x7f0f140a6300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f3019cf60 0x7f0f3019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30107270 0x7f0f30198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 --2- 172.21.15.52:0/1174284452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f301021a0 0x7f0f301986a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 >> 172.21.15.52:0/1174284452 conn(0x7f0f300793e0 msgr2=0x7f0f300ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 shutdown_connections 2024-04-26T01:47:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:01.659+0000 7f0f34c00640 1 -- 172.21.15.52:0/1174284452 wait complete. 2024-04-26T01:47:02.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 25235 7.37 MB/sec warmup 21 sec latency 52.997 ms 2024-04-26T01:47:03.058 INFO:tasks.workunit.client.0.smithi052.stdout: 1 25787 7.07 MB/sec warmup 22 sec latency 21.094 ms 2024-04-26T01:47:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:03 smithi183 ceph-mon[32752]: pgmap v64: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 844 KiB/s wr, 196 op/s 2024-04-26T01:47:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:03 smithi052 ceph-mon[27851]: pgmap v64: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 844 KiB/s wr, 196 op/s 2024-04-26T01:47:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:03 smithi191 ceph-mon[32745]: pgmap v64: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 844 KiB/s wr, 196 op/s 2024-04-26T01:47:04.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 26058 6.82 MB/sec warmup 23 sec latency 703.048 ms 2024-04-26T01:47:05.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 26058 6.54 MB/sec warmup 24 sec latency 1703.190 ms 2024-04-26T01:47:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:05 smithi183 ceph-mon[32752]: pgmap v65: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 383 KiB/s wr, 79 op/s 2024-04-26T01:47:05.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:05 smithi052 ceph-mon[27851]: pgmap v65: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 383 KiB/s wr, 79 op/s 2024-04-26T01:47:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:05 smithi191 ceph-mon[32745]: pgmap v65: 97 pgs: 97 active+clean; 13 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 383 KiB/s wr, 79 op/s 2024-04-26T01:47:06.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 26058 6.28 MB/sec warmup 25 sec latency 2703.326 ms 2024-04-26T01:47:07.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 26058 6.03 MB/sec warmup 26 sec latency 3703.480 ms 2024-04-26T01:47:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:07 smithi052 ceph-mon[27851]: pgmap v66: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 596 KiB/s wr, 146 op/s 2024-04-26T01:47:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:07 smithi191 ceph-mon[32745]: pgmap v66: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 596 KiB/s wr, 146 op/s 2024-04-26T01:47:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:07 smithi183 ceph-mon[32752]: pgmap v66: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 596 KiB/s wr, 146 op/s 2024-04-26T01:47:08.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 26409 5.93 MB/sec warmup 27 sec latency 4476.385 ms 2024-04-26T01:47:09.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 28235 6.05 MB/sec warmup 28 sec latency 12.784 ms 2024-04-26T01:47:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:09 smithi052 ceph-mon[27851]: pgmap v67: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:09.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:09 smithi191 ceph-mon[32745]: pgmap v67: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:09 smithi183 ceph-mon[32752]: pgmap v67: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:10.059 INFO:tasks.workunit.client.0.smithi052.stdout: 1 28794 6.00 MB/sec warmup 29 sec latency 21.261 ms 2024-04-26T01:47:11.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 29288 5.82 MB/sec warmup 30 sec latency 20.968 ms 2024-04-26T01:47:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:11 smithi052 ceph-mon[27851]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:11.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:11 smithi191 ceph-mon[32745]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:11 smithi183 ceph-mon[32752]: pgmap v68: 97 pgs: 97 active+clean; 11 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 424 KiB/s wr, 118 op/s 2024-04-26T01:47:12.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 29485 5.67 MB/sec warmup 31 sec latency 755.927 ms 2024-04-26T01:47:12.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:12 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:12.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:12 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:12.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:12 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:47:13.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 29802 5.59 MB/sec warmup 32 sec latency 1562.263 ms 2024-04-26T01:47:13.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:13 smithi052 ceph-mon[27851]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-04-26T01:47:13.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:13 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:13 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:13 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:47:13.576 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:13 smithi183 ceph-mon[32752]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-04-26T01:47:13.576 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:13 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:13.576 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:13 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:13.576 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:13 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:47:13.651 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:13 smithi191 ceph-mon[32745]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-04-26T01:47:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:13 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:47:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:13 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:47:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:13 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:47:14.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 31686 5.71 MB/sec warmup 33 sec latency 22.048 ms 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 -- 172.21.15.52:0/736168330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d006a9f0 msgr2=0x7f96d006add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 --2- 172.21.15.52:0/736168330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d006a9f0 0x7f96d006add0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009ca0 tx=0x7f96c402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 -- 172.21.15.52:0/736168330 shutdown_connections 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 --2- 172.21.15.52:0/736168330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d0107260 0x7f96d010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 --2- 172.21.15.52:0/736168330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d006b3a0 0x7f96d0106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 --2- 172.21.15.52:0/736168330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d006a9f0 0x7f96d006add0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 -- 172.21.15.52:0/736168330 >> 172.21.15.52:0/736168330 conn(0x7f96d0076b80 msgr2=0x7f96d0076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 -- 172.21.15.52:0/736168330 shutdown_connections 2024-04-26T01:47:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.812+0000 7f96d7800640 1 -- 172.21.15.52:0/736168330 wait complete. 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.813+0000 7f96d7800640 1 Processor -- start 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.813+0000 7f96d7800640 1 -- start start 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d0107260 0x7f96d01a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 0x7f96d019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01121f0 con 0x7f96d006a9f0 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96d0112070 con 0x7f96d0107260 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d7800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f96d0112370 con 0x7f96d019a680 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39786/0 (socket says 172.21.15.52:39786) 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d4e00640 1 -- 172.21.15.52:0/3534323468 learned_addr learned my addr 172.21.15.52:0/3534323468 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96d5800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 0x7f96d019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.814+0000 7f96cfe00640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d0107260 0x7f96d01a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 -- 172.21.15.52:0/3534323468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 msgr2=0x7f96d019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 0x7f96d019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 -- 172.21.15.52:0/3534323468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d0107260 msgr2=0x7f96d01a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d0107260 0x7f96d01a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 -- 172.21.15.52:0/3534323468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d019b280 con 0x7f96d006a9f0 2024-04-26T01:47:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d5800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 0x7f96d019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96d4e00640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96c4005e30 tx=0x7f96c4031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96cd600640 1 -- 172.21.15.52:0/3534323468 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4003b70 con 0x7f96d006a9f0 2024-04-26T01:47:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96cd600640 1 -- 172.21.15.52:0/3534323468 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c4003d10 con 0x7f96d006a9f0 2024-04-26T01:47:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.815+0000 7f96cd600640 1 -- 172.21.15.52:0/3534323468 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c402fa30 con 0x7f96d006a9f0 2024-04-26T01:47:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.816+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d019b510 con 0x7f96d006a9f0 2024-04-26T01:47:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.816+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d01a7660 con 0x7f96d006a9f0 2024-04-26T01:47:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.819+0000 7f96cd600640 1 -- 172.21.15.52:0/3534323468 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f96c402fbd0 con 0x7f96d006a9f0 2024-04-26T01:47:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.820+0000 7f96da7af740 1 Processor -- start 2024-04-26T01:47:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.820+0000 7f96da7af740 1 -- start start 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.820+0000 7f96da7af740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca28c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.820+0000 7f96da7af740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca29140 0x5651aca334c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96da7af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96da7af740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5651ac98e900 con 0x5651aca29140 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96da7af740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5651ac934ef0 con 0x5651aca28820 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96da7af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5651ac98a8c0 con 0x5651aca33a00 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cd600640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 0x7f96b80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96d5800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca28c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:49612/0 (socket says 172.21.15.52:49612) 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 -- 172.21.15.52:0/2190266171 learned_addr learned my addr 172.21.15.52:0/2190266171 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96d4e00640 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca29140 0x5651aca334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cd600640 1 -- 172.21.15.52:0/3534323468 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f96c40f1a20 con 0x7f96d006a9f0 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 -- 172.21.15.52:0/2190266171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 msgr2=0x5651aca28c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca28c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96d5800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 0x7f96b80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 -- 172.21.15.52:0/2190266171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca29140 msgr2=0x5651aca334c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca29140 0x5651aca334c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.821+0000 7f96cfe00640 1 -- 172.21.15.52:0/2190266171 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651ac6a1210 con 0x5651aca33a00 2024-04-26T01:47:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.822+0000 7f96cfe00640 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c60 tx=0x7f96bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afa90c88486ddf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.822+0000 7f96c2000640 1 -- 172.21.15.52:0/2190266171 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc004230 con 0x5651aca33a00 2024-04-26T01:47:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.822+0000 7f96c2000640 1 -- 172.21.15.52:0/2190266171 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96bc0043d0 con 0x5651aca33a00 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.822+0000 7f96c2000640 1 -- 172.21.15.52:0/2190266171 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc004cf0 con 0x5651aca33a00 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 -- 172.21.15.52:0/2190266171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 msgr2=0x5651aca35df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c60 tx=0x7f96bc011810 comp rx=0 tx=0).stop 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96d5800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 0x7f96b80a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96c8009fd0 tx=0x7f96c8004730 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 -- 172.21.15.52:0/2190266171 shutdown_connections 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca35df0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca29140 0x5651aca334c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 --2- 172.21.15.52:0/2190266171 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca28c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 -- 172.21.15.52:0/2190266171 >> 172.21.15.52:0/2190266171 conn(0x5651aca24d30 msgr2=0x5651aca25110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 -- 172.21.15.52:0/2190266171 shutdown_connections 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.823+0000 7f96da7af740 1 -- 172.21.15.52:0/2190266171 wait complete. 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 -- start start 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 Processor -- start 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca30e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca33a00 0x5651aca313c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5651aca36630 con 0x5651aca33a00 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5651ac948310 con 0x5651aca28820 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.824+0000 7f96da7af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5651aca367b0 con 0x5651aca29140 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca30e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96da7af740 10 client.0 _ll_register_callbacks cb 0x5651ac916cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:39762/0 (socket says 172.21.15.52:39762) 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 -- 172.21.15.52:0/2738650514 learned_addr learned my addr 172.21.15.52:0/2738650514 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d5800640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca33a00 0x5651aca313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 -- 172.21.15.52:0/2738650514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 msgr2=0x5651aca30e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca30e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 -- 172.21.15.52:0/2738650514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca33a00 msgr2=0x5651aca313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca33a00 0x5651aca313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 -- 172.21.15.52:0/2738650514 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651ac76d620 con 0x5651aca28820 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96d4e00640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f96c40ef2b0 tx=0x7f96c408d380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc7371cd4c9e8085 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.825+0000 7f96b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c40be1a0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f96c40ef3c0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96bc0043d0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f96c40ef5c0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 10 client.25412 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5651aca36630 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96bc0043d0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f96c408f0d0 con 0x5651aca28820 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 10 client.25412 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 10 client.25412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.826+0000 7f96da7af740 20 client.25412 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651aca43ce0 0x5651aca46120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96da7af740 4 client.25412 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651ac9a28b0 con 0x5651aca43ce0 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96cfe00640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651aca43ce0 0x5651aca46120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96cfe00640 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651aca43ce0 0x5651aca46120 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c5f3b01eebcc8d server_cookie=9a3eaf6029df7d28 in_seq=0 out_seq=0 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.827+0000 7f96b6a00640 10 client.25412 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96b6a00640 1 -- 172.21.15.52:0/2738650514 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651ac9a28b0 con 0x5651aca43ce0 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96b6a00640 10 client.25412 handle_command_reply: tid=0 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 client.25412 shutdown 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 msgr2=0x5651aca30940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f96c40ef2b0 tx=0x7f96c408d380 comp rx=0 tx=0).stop 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 shutdown_connections 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651aca43ce0 0x5651aca46120 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca33a00 0x5651aca313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.828+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca30e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96da7af740 1 --2- 172.21.15.52:0/2738650514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca28820 0x5651aca30940 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 >> 172.21.15.52:0/2738650514 conn(0x5651aca24d30 msgr2=0x5651aca2aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96b6a00640 0 client.25412 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 shutdown_connections 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96da7af740 1 -- 172.21.15.52:0/2738650514 wait complete. 2024-04-26T01:47:14.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.829+0000 7f96da7af740 20 client.25412 trim_cache size 0 max 16384 2024-04-26T01:47:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.846+0000 7f96da7af740 1 Processor -- start 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.846+0000 7f96da7af740 1 -- start start 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca3f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca33a00 0x5651aca4ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01b2470 con 0x5651aca28820 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01afd10 con 0x5651aca33a00 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96da7af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01adb60 con 0x5651aca29140 2024-04-26T01:47:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca3f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96d5800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca33a00 0x5651aca4ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca3f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39814/0 (socket says 172.21.15.52:39814) 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96d4e00640 1 -- 172.21.15.52:0/3123536739 learned_addr learned my addr 172.21.15.52:0/3123536739 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.847+0000 7f96cfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:49636/0 (socket says 172.21.15.52:49636) 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 -- 172.21.15.52:0/3123536739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca33a00 msgr2=0x5651aca4ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca33a00 0x5651aca4ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 -- 172.21.15.52:0/3123536739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 msgr2=0x5651aca3f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca3f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 -- 172.21.15.52:0/3123536739 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651aca256b0 con 0x5651aca29140 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96cfe00640 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c30 tx=0x7f96bc0096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a71822cfa6d6741 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96c1600640 1 -- 172.21.15.52:0/3123536739 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc01e040 con 0x5651aca29140 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96c1600640 1 -- 172.21.15.52:0/3123536739 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96bc01f040 con 0x5651aca29140 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.848+0000 7f96c1600640 1 -- 172.21.15.52:0/3123536739 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc026440 con 0x5651aca29140 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 -- 172.21.15.52:0/3123536739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 msgr2=0x5651aca2b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009c30 tx=0x7f96bc0096a0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 -- 172.21.15.52:0/3123536739 shutdown_connections 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca33a00 0x5651aca4ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca29140 0x5651aca2b1e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 --2- 172.21.15.52:0/3123536739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca3f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 -- 172.21.15.52:0/3123536739 >> 172.21.15.52:0/3123536739 conn(0x5651aca24d30 msgr2=0x5651aca23e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 -- 172.21.15.52:0/3123536739 shutdown_connections 2024-04-26T01:47:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.849+0000 7f96da7af740 1 -- 172.21.15.52:0/3123536739 wait complete. 2024-04-26T01:47:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.850+0000 7f96da7af740 1 -- start start 2024-04-26T01:47:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.850+0000 7f96da7af740 1 Processor -- start 2024-04-26T01:47:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.850+0000 7f96da7af740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.850+0000 7f96da7af740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 0x5651aca39910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96da7af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca39e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96da7af740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01ae1d0 con 0x5651aca28820 2024-04-26T01:47:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96da7af740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01af640 con 0x5651aca29140 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96da7af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f96d01ae8a0 con 0x5651aca33a00 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96da7af740 10 client.0 _ll_register_callbacks cb 0x7f96c4009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39816/0 (socket says 172.21.15.52:39816) 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d4e00640 1 -- 172.21.15.52:0/4279622543 learned_addr learned my addr 172.21.15.52:0/4279622543 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 0x5651aca39910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d5800640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca39e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 -- 172.21.15.52:0/4279622543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 msgr2=0x5651aca39e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca39e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 -- 172.21.15.52:0/4279622543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 msgr2=0x5651aca393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96cfe00640 1 -- 172.21.15.52:0/4279622543 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651acab1ac0 con 0x5651aca29140 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d5800640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.851+0000 7f96d4e00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:14.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96cfe00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 0x5651aca39910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5651aca3fa30 tx=0x7f96bc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad641e1c5a8d03b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0037f0 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f96bc003990 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96bc01f040 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f96bc004050 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96da7af740 10 client.25418 fetch_fsmap learned FSMap version 28 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.852+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5651acab2d50 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96bc01f040 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f96bc02d040 con 0x5651aca29140 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 10 client.25418 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 10 client.25418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 20 client.25418 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651acab3270 0x5651acab5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 4 client.25418 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:14.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651ac99c320 con 0x5651acab3270 2024-04-26T01:47:14.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.853+0000 7f96d4e00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651acab3270 0x5651acab5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:14.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.854+0000 7f96d4e00640 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651acab3270 0x5651acab5690 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbac549a6f1f5873 server_cookie=b202353315df3d86 in_seq=0 out_seq=0 2024-04-26T01:47:14.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.854+0000 7f96b6a00640 10 client.25418 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.879+0000 7f96b6a00640 1 -- 172.21.15.52:0/4279622543 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651ac99c320 con 0x5651acab3270 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.879+0000 7f96b6a00640 10 client.25418 handle_command_reply: tid=0 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.879+0000 7f96da7af740 1 client.25418 shutdown 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.879+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 msgr2=0x5651aca39910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.879+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 0x5651aca39910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5651aca3fa30 tx=0x7f96bc009f90 comp rx=0 tx=0).stop 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 shutdown_connections 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5651acab3270 0x5651acab5690 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5651aca33a00 0x5651aca39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5651aca29140 0x5651aca39910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96b6a00640 0 client.25418 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 --2- 172.21.15.52:0/4279622543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651aca28820 0x5651aca393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 >> 172.21.15.52:0/4279622543 conn(0x5651aca24d30 msgr2=0x5651aca23600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 shutdown_connections 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 1 -- 172.21.15.52:0/4279622543 wait complete. 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.880+0000 7f96da7af740 20 client.25418 trim_cache size 0 max 16384 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:47:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 msgr2=0x7f96b80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 0x7f96b80a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96c8009fd0 tx=0x7f96c8004730 comp rx=0 tx=0).stop 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 msgr2=0x7f96d01a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96c4005e30 tx=0x7f96c4031840 comp rx=0 tx=0).stop 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 shutdown_connections 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f96b80a3dc0 0x7f96b80a6280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96d019a680 0x7f96d019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96d0107260 0x7f96d01a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 --2- 172.21.15.52:0/3534323468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96d006a9f0 0x7f96d01a05c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 >> 172.21.15.52:0/3534323468 conn(0x7f96d0076b80 msgr2=0x7f96d010d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 shutdown_connections 2024-04-26T01:47:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:14.881+0000 7f96d7800640 1 -- 172.21.15.52:0/3534323468 wait complete. 2024-04-26T01:47:14.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:14 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:14.901 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:14 smithi191 ceph-mon[32745]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-04-26T01:47:14.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:14 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:14.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:14 smithi183 ceph-mon[32752]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-04-26T01:47:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:14 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:14 smithi052 ceph-mon[27851]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-04-26T01:47:15.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 32334 5.68 MB/sec warmup 34 sec latency 45.116 ms 2024-04-26T01:47:16.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 32829 5.53 MB/sec warmup 35 sec latency 23.579 ms 2024-04-26T01:47:17.060 INFO:tasks.workunit.client.0.smithi052.stdout: 1 34433 5.58 MB/sec warmup 36 sec latency 20.298 ms 2024-04-26T01:47:17.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:16 smithi052 ceph-mon[27851]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-26T01:47:17.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:16 smithi191 ceph-mon[32745]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-26T01:47:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:16 smithi183 ceph-mon[32752]: pgmap v71: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 4.0 MiB/s wr, 224 op/s 2024-04-26T01:47:18.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 35719 5.73 MB/sec warmup 37 sec latency 16.644 ms 2024-04-26T01:47:19.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 36012 5.58 MB/sec warmup 38 sec latency 18.646 ms 2024-04-26T01:47:19.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:18 smithi191 ceph-mon[32745]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:19.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:18 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:18 smithi183 ceph-mon[32752]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:18 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:18 smithi052 ceph-mon[27851]: pgmap v72: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:18 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:20.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 36933 5.57 MB/sec warmup 39 sec latency 23.004 ms 2024-04-26T01:47:21.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 38855 5.67 MB/sec warmup 40 sec latency 11.889 ms 2024-04-26T01:47:21.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:20 smithi191 ceph-mon[32745]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:20 smithi183 ceph-mon[32752]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:20 smithi052 ceph-mon[27851]: pgmap v73: 97 pgs: 97 active+clean; 30 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 3.8 MiB/s wr, 157 op/s 2024-04-26T01:47:22.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 39427 5.64 MB/sec warmup 41 sec latency 28.474 ms 2024-04-26T01:47:23.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 39929 5.52 MB/sec warmup 42 sec latency 20.325 ms 2024-04-26T01:47:23.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:22 smithi191 ceph-mon[32745]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-04-26T01:47:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:22 smithi183 ceph-mon[32752]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-04-26T01:47:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:22 smithi052 ceph-mon[27851]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 218 op/s 2024-04-26T01:47:24.061 INFO:tasks.workunit.client.0.smithi052.stdout: 1 41648 5.57 MB/sec warmup 43 sec latency 17.096 ms 2024-04-26T01:47:25.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 42837 5.68 MB/sec warmup 44 sec latency 19.890 ms 2024-04-26T01:47:25.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:24 smithi191 ceph-mon[32745]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 169 op/s 2024-04-26T01:47:25.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:24 smithi183 ceph-mon[32752]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 169 op/s 2024-04-26T01:47:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:24 smithi052 ceph-mon[27851]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 169 op/s 2024-04-26T01:47:26.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 43153 5.57 MB/sec warmup 45 sec latency 26.506 ms 2024-04-26T01:47:27.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 44178 5.56 MB/sec warmup 46 sec latency 21.997 ms 2024-04-26T01:47:27.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:26 smithi191 ceph-mon[32745]: pgmap v76: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-04-26T01:47:27.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:26 smithi183 ceph-mon[32752]: pgmap v76: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-04-26T01:47:27.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:26 smithi052 ceph-mon[27851]: pgmap v76: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 -- 172.21.15.52:0/615899464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 msgr2=0x7fbb9010ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 --2- 172.21.15.52:0/615899464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb9010ec60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009ca0 tx=0x7fbb8c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 -- 172.21.15.52:0/615899464 shutdown_connections 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 --2- 172.21.15.52:0/615899464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb9010ec60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 --2- 172.21.15.52:0/615899464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb90102970 0x7fbb901069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 --2- 172.21.15.52:0/615899464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 0x7fbb901023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 -- 172.21.15.52:0/615899464 >> 172.21.15.52:0/615899464 conn(0x7fbb900fd8a0 msgr2=0x7fbb900ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 -- 172.21.15.52:0/615899464 shutdown_connections 2024-04-26T01:47:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.041+0000 7fbb97200640 1 -- 172.21.15.52:0/615899464 wait complete. 2024-04-26T01:47:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.042+0000 7fbb97200640 1 Processor -- start 2024-04-26T01:47:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.042+0000 7fbb97200640 1 -- start start 2024-04-26T01:47:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 0x7fbb901a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 0x7fbb9019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90112050 con 0x7fbb9019a5b0 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90111ed0 con 0x7fbb90101fc0 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb97200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901121d0 con 0x7fbb90107090 2024-04-26T01:47:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37924/0 (socket says 172.21.15.52:37924) 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 -- 172.21.15.52:0/2437437687 learned_addr learned my addr 172.21.15.52:0/2437437687 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95e00640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 0x7fbb901a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb96800640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 0x7fbb9019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 -- 172.21.15.52:0/2437437687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 msgr2=0x7fbb901a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 0x7fbb901a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 -- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 msgr2=0x7fbb9019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 0x7fbb9019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.043+0000 7fbb95400640 1 -- 172.21.15.52:0/2437437687 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9019b0d0 con 0x7fbb90107090 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb96800640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 0x7fbb9019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb95400640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00e9c0 tx=0x7fbb7c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb8aa00640 1 -- 172.21.15.52:0/2437437687 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c00cd80 con 0x7fbb90107090 2024-04-26T01:47:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb8aa00640 1 -- 172.21.15.52:0/2437437687 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb7c0054d0 con 0x7fbb90107090 2024-04-26T01:47:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb8aa00640 1 -- 172.21.15.52:0/2437437687 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c010430 con 0x7fbb90107090 2024-04-26T01:47:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9019b3c0 con 0x7fbb90107090 2024-04-26T01:47:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.044+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb901a75c0 con 0x7fbb90107090 2024-04-26T01:47:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.048+0000 7fbb8aa00640 1 -- 172.21.15.52:0/2437437687 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbb7c004fd0 con 0x7fbb90107090 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.049+0000 7fbb9b662740 1 Processor -- start 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.049+0000 7fbb9b662740 1 -- start start 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a582d0 0x556350a62650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a64f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5563509bd900 con 0x556350a62b90 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556350963ef0 con 0x556350a579b0 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb9b662740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5563509b98c0 con 0x556350a582d0 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb8aa00640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 0x7fbb780a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41990/0 (socket says 172.21.15.52:41990) 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 -- 172.21.15.52:0/1891791563 learned_addr learned my addr 172.21.15.52:0/1891791563 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb95e00640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a582d0 0x556350a62650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb95400640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a64f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb8aa00640 1 -- 172.21.15.52:0/2437437687 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbb7c0d0ba0 con 0x7fbb90107090 2024-04-26T01:47:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 -- 172.21.15.52:0/1891791563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a582d0 msgr2=0x556350a62650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a582d0 0x556350a62650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 -- 172.21.15.52:0/1891791563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 msgr2=0x556350a64f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a64f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.050+0000 7fbb96800640 1 -- 172.21.15.52:0/1891791563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563506d0210 con 0x556350a579b0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb95400640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a64f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb96800640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 0x7fbb780a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb96800640 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02f540 tx=0x7fbb8c009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f5ea30b1eed565 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb83000640 1 -- 172.21.15.52:0/1891791563 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c003e40 con 0x556350a579b0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb83000640 1 -- 172.21.15.52:0/1891791563 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb8c036040 con 0x556350a579b0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.051+0000 7fbb83000640 1 -- 172.21.15.52:0/1891791563 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c02f990 con 0x556350a579b0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 -- 172.21.15.52:0/1891791563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 msgr2=0x556350a57d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02f540 tx=0x7fbb8c009910 comp rx=0 tx=0).stop 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb96800640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 0x7fbb780a6250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c04c000 tx=0x7fbb8c049000 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 -- 172.21.15.52:0/1891791563 shutdown_connections 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a64f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a582d0 0x556350a62650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1891791563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a579b0 0x556350a57d90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 -- 172.21.15.52:0/1891791563 >> 172.21.15.52:0/1891791563 conn(0x556350a53ec0 msgr2=0x556350a542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 -- 172.21.15.52:0/1891791563 shutdown_connections 2024-04-26T01:47:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.052+0000 7fbb9b662740 1 -- 172.21.15.52:0/1891791563 wait complete. 2024-04-26T01:47:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.053+0000 7fbb9b662740 1 -- start start 2024-04-26T01:47:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.053+0000 7fbb9b662740 1 Processor -- start 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.053+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a605e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556350a65640 con 0x556350a62b90 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556350977310 con 0x556350a582d0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556350a657c0 con 0x556350a579b0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb9b662740 10 client.0 _ll_register_callbacks cb 0x556350945cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42004/0 (socket says 172.21.15.52:42004) 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 -- 172.21.15.52:0/327234996 learned_addr learned my addr 172.21.15.52:0/327234996 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37936/0 (socket says 172.21.15.52:37936) 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb96800640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 -- 172.21.15.52:0/327234996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 msgr2=0x556350a5fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 -- 172.21.15.52:0/327234996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 msgr2=0x556350a605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95400640 1 -- 172.21.15.52:0/327234996 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55635079c620 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.054+0000 7fbb95e00640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb95400640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c018000 tx=0x7fbb7c017f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d598f088814ecc8b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c0046c0 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb8c036040 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbb7c0ce540 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb7c0ce740 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb9b662740 10 client.25430 fetch_fsmap learned FSMap version 29 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556350a71950 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb8c036040 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.055+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbb7c0972c0 con 0x556350a582d0 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 10 client.25430 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 10 client.25430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 20 client.25430 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350a72dc0 0x556350a75210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 4 client.25430 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563509d18b0 con 0x556350a72dc0 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.056+0000 7fbb95e00640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350a72dc0 0x556350a75210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.057+0000 7fbb95e00640 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350a72dc0 0x556350a75210 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a657fe891e0d9f server_cookie=5072270faafd9218 in_seq=0 out_seq=0 2024-04-26T01:47:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.057+0000 7fbb77400640 10 client.25430 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.057+0000 7fbb77400640 1 -- 172.21.15.52:0/327234996 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563509d18b0 con 0x556350a72dc0 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.057+0000 7fbb77400640 10 client.25430 handle_command_reply: tid=0 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 client.25430 shutdown 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 msgr2=0x556350a600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c018000 tx=0x7fbb7c017f90 comp rx=0 tx=0).stop 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 shutdown_connections 2024-04-26T01:47:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350a72dc0 0x556350a75210 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a582d0 0x556350a600a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 --2- 172.21.15.52:0/327234996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a5fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 >> 172.21.15.52:0/327234996 conn(0x556350a53ec0 msgr2=0x556350a59bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.058+0000 7fbb77400640 0 client.25430 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.059+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 shutdown_connections 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.059+0000 7fbb9b662740 1 -- 172.21.15.52:0/327234996 wait complete. 2024-04-26T01:47:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.059+0000 7fbb9b662740 20 client.25430 trim_cache size 0 max 16384 2024-04-26T01:47:28.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 46109 5.70 MB/sec warmup 47 sec latency 13.839 ms 2024-04-26T01:47:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 Processor -- start 2024-04-26T01:47:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 -- start start 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a6e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a59a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901b24a0 con 0x556350a62b90 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901afd10 con 0x556350a623a0 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.077+0000 7fbb9b662740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901adb60 con 0x556350a579b0 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb95400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a59a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42008/0 (socket says 172.21.15.52:42008) 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 -- 172.21.15.52:0/2260094428 learned_addr learned my addr 172.21.15.52:0/2260094428 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb95e00640 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a6e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 -- 172.21.15.52:0/2260094428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 msgr2=0x556350a6e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a6e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 -- 172.21.15.52:0/2260094428 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 msgr2=0x556350a59a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a59a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb96800640 1 -- 172.21.15.52:0/2260094428 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556350a5eb10 con 0x556350a623a0 2024-04-26T01:47:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.078+0000 7fbb95400640 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a59a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.079+0000 7fbb96800640 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02f540 tx=0x7fbb8c052000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8922bff06c6eed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.079+0000 7fbb82600640 1 -- 172.21.15.52:0/2260094428 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c062070 con 0x556350a623a0 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.079+0000 7fbb82600640 1 -- 172.21.15.52:0/2260094428 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb8c003dc0 con 0x556350a623a0 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.079+0000 7fbb82600640 1 -- 172.21.15.52:0/2260094428 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0042a0 con 0x556350a623a0 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 -- 172.21.15.52:0/2260094428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 msgr2=0x556350a7ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c02f540 tx=0x7fbb8c052000 comp rx=0 tx=0).stop 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 -- 172.21.15.52:0/2260094428 shutdown_connections 2024-04-26T01:47:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a623a0 0x556350a7ab20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a62b90 0x556350a59a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 --2- 172.21.15.52:0/2260094428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a6e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 -- 172.21.15.52:0/2260094428 >> 172.21.15.52:0/2260094428 conn(0x556350a53ec0 msgr2=0x556350a59360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 -- 172.21.15.52:0/2260094428 shutdown_connections 2024-04-26T01:47:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.080+0000 7fbb9b662740 1 -- 172.21.15.52:0/2260094428 wait complete. 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.081+0000 7fbb9b662740 1 -- start start 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.081+0000 7fbb9b662740 1 Processor -- start 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.081+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.081+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 0x556350a68b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb9b662740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb9b662740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901ae1d0 con 0x556350a623a0 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb9b662740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901af640 con 0x556350a62b90 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb9b662740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbb901ae8a0 con 0x556350a579b0 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb9b662740 10 client.0 _ll_register_callbacks cb 0x7fbb7c0986f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 0x556350a68b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37948/0 (socket says 172.21.15.52:37948) 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95e00640 1 -- 172.21.15.52:0/1794774899 learned_addr learned my addr 172.21.15.52:0/1794774899 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42018/0 (socket says 172.21.15.52:42018) 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 -- 172.21.15.52:0/1794774899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 msgr2=0x556350a68650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 -- 172.21.15.52:0/1794774899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 msgr2=0x556350a68b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 0x556350a68b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb96800640 1 -- 172.21.15.52:0/1794774899 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556350ad8500 con 0x556350a62b90 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95e00640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.082+0000 7fbb95400640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 0x556350a68b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb96800640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009ca0 tx=0x7fbb8c031bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca7f0ebd8d82d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb77400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c062040 con 0x556350a62b90 2024-04-26T01:47:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fbb8c004040 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb8c003dc0 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb8c004240 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb9b662740 10 client.25436 fetch_fsmap learned FSMap version 29 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556350ad97f0 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.083+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb8c003dc0 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbb8c05c030 con 0x556350a62b90 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 10 client.25436 fetch_fsmap finished waiting for FSMap version 29 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 10 client.25436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 20 client.25436 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350ad9a80 0x556350adbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 4 client.25436 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563509cb320 con 0x556350ad9a80 2024-04-26T01:47:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.084+0000 7fbb95e00640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350ad9a80 0x556350adbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.085+0000 7fbb95e00640 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350ad9a80 0x556350adbe70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0d599f8eb8cb626 server_cookie=9d4e6cf39a6ec76f in_seq=0 out_seq=0 2024-04-26T01:47:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.085+0000 7fbb77400640 10 client.25436 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.117+0000 7fbb77400640 1 -- 172.21.15.52:0/1794774899 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563509cb320 con 0x556350ad9a80 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.117+0000 7fbb77400640 10 client.25436 handle_command_reply: tid=0 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.117+0000 7fbb9b662740 1 client.25436 shutdown 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 msgr2=0x556350a690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009ca0 tx=0x7fbb8c031bf0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 shutdown_connections 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556350ad9a80 0x556350adbe70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556350a62b90 0x556350a690d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556350a623a0 0x556350a68b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 --2- 172.21.15.52:0/1794774899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556350a579b0 0x556350a68650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb77400640 0 client.25436 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 >> 172.21.15.52:0/1794774899 conn(0x556350a53ec0 msgr2=0x556350a59bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 shutdown_connections 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 1 -- 172.21.15.52:0/1794774899 wait complete. 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.118+0000 7fbb9b662740 20 client.25436 trim_cache size 0 max 16384 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:47:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:47:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 msgr2=0x7fbb780a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 0x7fbb780a6250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c04c000 tx=0x7fbb8c049000 comp rx=0 tx=0).stop 2024-04-26T01:47:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 msgr2=0x7fbb901a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00e9c0 tx=0x7fbb7c00ee80 comp rx=0 tx=0).stop 2024-04-26T01:47:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 shutdown_connections 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbb780a3d90 0x7fbb780a6250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbb9019a5b0 0x7fbb9019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb90107090 0x7fbb901a0940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 --2- 172.21.15.52:0/2437437687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbb90101fc0 0x7fbb901a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 >> 172.21.15.52:0/2437437687 conn(0x7fbb900fd8a0 msgr2=0x7fbb900fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 shutdown_connections 2024-04-26T01:47:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:28.119+0000 7fbb97200640 1 -- 172.21.15.52:0/2437437687 wait complete. 2024-04-26T01:47:29.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 46567 5.62 MB/sec warmup 48 sec latency 30.194 ms 2024-04-26T01:47:29.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:29 smithi191 ceph-mon[32745]: pgmap v77: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:29.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:29 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:29.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:29 smithi183 ceph-mon[32752]: pgmap v77: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:29.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:29 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:29 smithi052 ceph-mon[27851]: pgmap v77: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:29 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:47:29.541 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:47:29.542 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:47:29.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:47:30.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:30.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 46870 5.51 MB/sec warmup 49 sec latency 553.864 ms 2024-04-26T01:47:31.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 46970 5.40 MB/sec warmup 50 sec latency 1435.765 ms 2024-04-26T01:47:31.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:31 smithi191 ceph-mon[32745]: pgmap v78: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:31.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:31 smithi183 ceph-mon[32752]: pgmap v78: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:31.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:31 smithi052 ceph-mon[27851]: pgmap v78: 97 pgs: 97 active+clean; 36 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 614 KiB/s wr, 114 op/s 2024-04-26T01:47:32.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 47061 5.30 MB/sec warmup 51 sec latency 916.480 ms 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 -- 172.21.15.52:0/4184261169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324101930 msgr2=0x7f1324101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 --2- 172.21.15.52:0/4184261169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324101930 0x7f1324101db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1310009990 tx=0x7f1310009ca0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 -- 172.21.15.52:0/4184261169 shutdown_connections 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 --2- 172.21.15.52:0/4184261169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13241022f0 0x7f132410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 --2- 172.21.15.52:0/4184261169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1324107980 0x7f1324107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 --2- 172.21.15.52:0/4184261169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324101930 0x7f1324101db0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 -- 172.21.15.52:0/4184261169 >> 172.21.15.52:0/4184261169 conn(0x7f13240788b0 msgr2=0x7f1324078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.665+0000 7f132c600700 1 -- 172.21.15.52:0/4184261169 shutdown_connections 2024-04-26T01:47:32.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.666+0000 7f132c600700 1 -- 172.21.15.52:0/4184261169 wait complete. 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 Processor -- start 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 -- start start 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 0x7f1324108d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 0x7f1324109270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1324111b20 con 0x7f1324101930 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13241119a0 con 0x7f13241022f0 2024-04-26T01:47:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.667+0000 7f132c600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1324111ca0 con 0x7f1324107980 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39310/0 (socket says 172.21.15.52:39310) 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 -- 172.21.15.52:0/990363894 learned_addr learned my addr 172.21.15.52:0/990363894 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329200700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 0x7f1324108d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f132a600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 0x7f1324109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 -- 172.21.15.52:0/990363894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 msgr2=0x7f1324109270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 0x7f1324109270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 -- 172.21.15.52:0/990363894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 msgr2=0x7f1324108d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 0x7f1324108d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329c00700 1 -- 172.21.15.52:0/990363894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13241aba00 con 0x7f1324101930 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f132a600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 0x7f1324109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.668+0000 7f1329200700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 0x7f1324108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f1329c00700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f131800d950 tx=0x7f131800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318016070 con 0x7f1324101930 2024-04-26T01:47:32.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f131800fd40 con 0x7f1324101930 2024-04-26T01:47:32.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13241abcf0 con 0x7f1324101930 2024-04-26T01:47:32.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131800e3b0 con 0x7f1324101930 2024-04-26T01:47:32.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.669+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13241ac1d0 con 0x7f1324101930 2024-04-26T01:47:32.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.670+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1324103050 con 0x7f1324101930 2024-04-26T01:47:32.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.672+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1318019070 con 0x7f1324101930 2024-04-26T01:47:32.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.674+0000 7f1322000700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 0x7f130c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:32.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.674+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f13180d0190 con 0x7f1324101930 2024-04-26T01:47:32.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.674+0000 7f1329200700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 0x7f130c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:32.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.675+0000 7f1329200700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 0x7f130c0a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1324109fb0 tx=0x7f131000b260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:32.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.680+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f131809c860 con 0x7f1324101930 2024-04-26T01:47:32.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.970+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1324109d20 con 0x7f1324101930 2024-04-26T01:47:32.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.972+0000 7f1322000700 1 -- 172.21.15.52:0/990363894 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11889 (secure 0 0 0) 0x7f131809c160 con 0x7f1324101930 2024-04-26T01:47:32.970 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:47:32.970 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:47:28.026495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:47:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 msgr2=0x7f130c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 0x7f130c0a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1324109fb0 tx=0x7f131000b260 comp rx=0 tx=0).stop 2024-04-26T01:47:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 msgr2=0x7f13241087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f131800d950 tx=0x7f131800dc60 comp rx=0 tx=0).stop 2024-04-26T01:47:32.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 shutdown_connections 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f130c0a3db0 0x7f130c0a6270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1324107980 0x7f1324109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1324101930 0x7f13241087f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 --2- 172.21.15.52:0/990363894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13241022f0 0x7f1324108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 >> 172.21.15.52:0/990363894 conn(0x7f13240788b0 msgr2=0x7f132406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 shutdown_connections 2024-04-26T01:47:32.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:32.975+0000 7f132c600700 1 -- 172.21.15.52:0/990363894 wait complete. 2024-04-26T01:47:32.975 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-04-26T01:47:33.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 47185 5.22 MB/sec warmup 52 sec latency 1669.939 ms 2024-04-26T01:47:33.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:33 smithi052 ceph-mon[27851]: pgmap v79: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 871 KiB/s wr, 177 op/s 2024-04-26T01:47:33.362 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:33 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/990363894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:33 smithi191 ceph-mon[32745]: pgmap v79: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 871 KiB/s wr, 177 op/s 2024-04-26T01:47:33.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:33 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/990363894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:33 smithi183 ceph-mon[32752]: pgmap v79: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 871 KiB/s wr, 177 op/s 2024-04-26T01:47:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:33 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/990363894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:47:33.912 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:47:33.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:47:34.062 INFO:tasks.workunit.client.0.smithi052.stdout: 1 48864 5.29 MB/sec warmup 53 sec latency 18.749 ms 2024-04-26T01:47:34.394 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:47:35.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50009 5.34 MB/sec warmup 54 sec latency 23.044 ms 2024-04-26T01:47:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:35 smithi052 ceph-mon[27851]: pgmap v80: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 528 KiB/s wr, 116 op/s 2024-04-26T01:47:35.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:35 smithi191 ceph-mon[32745]: pgmap v80: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 528 KiB/s wr, 116 op/s 2024-04-26T01:47:35.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:35 smithi183 ceph-mon[32752]: pgmap v80: 97 pgs: 97 active+clean; 38 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 528 KiB/s wr, 116 op/s 2024-04-26T01:47:36.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50411 5.25 MB/sec warmup 55 sec latency 81.514 ms 2024-04-26T01:47:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 -- 172.21.15.52:0/1529851459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00075590 msgr2=0x7f6d0010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 --2- 172.21.15.52:0/1529851459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00075590 0x7f6d0010ef60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009990 tx=0x7f6cf0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 -- 172.21.15.52:0/1529851459 shutdown_connections 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 --2- 172.21.15.52:0/1529851459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d00111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 --2- 172.21.15.52:0/1529851459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00074c70 0x7f6d00075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 --2- 172.21.15.52:0/1529851459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00075590 0x7f6d0010ef60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 -- 172.21.15.52:0/1529851459 >> 172.21.15.52:0/1529851459 conn(0x7f6d00079b10 msgr2=0x7f6d00079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 -- 172.21.15.52:0/1529851459 shutdown_connections 2024-04-26T01:47:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.693+0000 7f6d08800700 1 -- 172.21.15.52:0/1529851459 wait complete. 2024-04-26T01:47:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.695+0000 7f6d08800700 1 Processor -- start 2024-04-26T01:47:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.695+0000 7f6d08800700 1 -- start start 2024-04-26T01:47:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 0x7f6d0019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d0019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00114e40 con 0x7f6d0010f4a0 2024-04-26T01:47:36.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00114cc0 con 0x7f6d00074c70 2024-04-26T01:47:36.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d08800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00114fc0 con 0x7f6d00075590 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41274/0 (socket says 172.21.15.52:41274) 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 -- 172.21.15.52:0/69113423 learned_addr learned my addr 172.21.15.52:0/69113423 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05400700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 0x7f6d0019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d06800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d0019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 -- 172.21.15.52:0/69113423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 msgr2=0x7f6d0019b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 0x7f6d0019b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 -- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 msgr2=0x7f6d0019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d0019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05e00700 1 -- 172.21.15.52:0/69113423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d001a02a0 con 0x7f6d00074c70 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6d06800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d0019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.696+0000 7f6d05400700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 0x7f6d0019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6d05e00700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e9a0 tx=0x7f6cf400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6cfaa00700 1 -- 172.21.15.52:0/69113423 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400cbd0 con 0x7f6d00074c70 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6cfaa00700 1 -- 172.21.15.52:0/69113423 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf400cd70 con 0x7f6d00074c70 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d001a0590 con 0x7f6d00074c70 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6cfaa00700 1 -- 172.21.15.52:0/69113423 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4018680 con 0x7f6d00074c70 2024-04-26T01:47:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.697+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d001a7e10 con 0x7f6d00074c70 2024-04-26T01:47:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.702+0000 7f6cfaa00700 1 -- 172.21.15.52:0/69113423 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6cf4010970 con 0x7f6d00074c70 2024-04-26T01:47:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.703+0000 7f6d131d3b80 1 Processor -- start 2024-04-26T01:47:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.703+0000 7f6d131d3b80 1 -- start start 2024-04-26T01:47:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1eba9c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1eba9e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e1eb9e4020 con 0x55e1eba90dd0 2024-04-26T01:47:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e1eb98a8c0 con 0x55e1eba9c560 2024-04-26T01:47:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e1eb9dffe0 con 0x55e1eba916f0 2024-04-26T01:47:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6cfaa00700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 0x7f6ce80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d05400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1eba9e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6cfaa00700 1 -- 172.21.15.52:0/69113423 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f6cf40d0b00 con 0x7f6d00074c70 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d05400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1eba9e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41286/0 (socket says 172.21.15.52:41286) 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d05e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1eba9c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d05400700 1 -- 172.21.15.52:0/1019287628 learned_addr learned my addr 172.21.15.52:0/1019287628 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d06800700 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.704+0000 7f6d06800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 0x7f6ce80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 -- 172.21.15.52:0/1019287628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 msgr2=0x55e1eba9e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1eba9e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 -- 172.21.15.52:0/1019287628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 msgr2=0x55e1eba911b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba911b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 -- 172.21.15.52:0/1019287628 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1eb980810 con 0x55e1eba916f0 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d05e00700 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1eba9c020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40ce220 tx=0x7f6cf406c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924663bdcd657082 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6cef400700 1 -- 172.21.15.52:0/1019287628 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf409d170 con 0x55e1eba916f0 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6d06800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 0x7f6ce80a6390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc012820 tx=0x7f6cfc00b040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6cef400700 1 -- 172.21.15.52:0/1019287628 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf4067580 con 0x55e1eba916f0 2024-04-26T01:47:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.705+0000 7f6cef400700 1 -- 172.21.15.52:0/1019287628 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf409cf30 con 0x55e1eba916f0 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 -- 172.21.15.52:0/1019287628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 msgr2=0x55e1eba9c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1eba9c020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40ce220 tx=0x7f6cf406c350 comp rx=0 tx=0).stop 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 -- 172.21.15.52:0/1019287628 shutdown_connections 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1eba9e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/1019287628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1eba9c020 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 -- 172.21.15.52:0/1019287628 >> 172.21.15.52:0/1019287628 conn(0x55e1eb9c00e0 msgr2=0x55e1eba8d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.706+0000 7f6d131d3b80 1 -- 172.21.15.52:0/1019287628 shutdown_connections 2024-04-26T01:47:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.707+0000 7f6d131d3b80 1 -- 172.21.15.52:0/1019287628 wait complete. 2024-04-26T01:47:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.707+0000 7f6d131d3b80 1 -- start start 2024-04-26T01:47:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.707+0000 7f6d131d3b80 1 Processor -- start 2024-04-26T01:47:36.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba9bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba9c560 0x55e1eba994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e1eba9f050 con 0x55e1eba90dd0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e1eb99d9f0 con 0x55e1eba916f0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e1eba9f1d0 con 0x55e1eba9c560 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d131d3b80 10 client.0 _ll_register_callbacks cb 0x55e1eb97c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d05400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d06800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba9c560 0x55e1eba994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d05400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41300/0 (socket says 172.21.15.52:41300) 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d05400700 1 -- 172.21.15.52:0/65969076 learned_addr learned my addr 172.21.15.52:0/65969076 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.708+0000 7f6d05e00700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba9bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 -- 172.21.15.52:0/65969076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba9c560 msgr2=0x55e1eba994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba9c560 0x55e1eba994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 -- 172.21.15.52:0/65969076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 msgr2=0x55e1eba9bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba9bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 -- 172.21.15.52:0/65969076 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1eb931a60 con 0x55e1eba916f0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d05400700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000be50 tx=0x7f6cf000be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668f9f70241c17d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6ce3e00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000faf0 con 0x55e1eba916f0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f6cf000fc90 con 0x55e1eba916f0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.709+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf4067580 con 0x55e1eba916f0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.710+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6cf0005070 con 0x55e1eba916f0 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.710+0000 7f6d131d3b80 10 client.25454 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:36.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.710+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1eb97ac80 con 0x55e1eba916f0 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.710+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf4067580 con 0x55e1eba916f0 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.710+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f6cf00038c0 con 0x55e1eba916f0 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 10 client.25454 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 10 client.25454 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 20 client.25454 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebab44e0 0x55e1ebab6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 4 client.25454 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1eb9f1810 con 0x55e1ebab44e0 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.711+0000 7f6d05e00700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebab44e0 0x55e1ebab6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.712+0000 7f6d05e00700 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebab44e0 0x55e1ebab6930 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3be9e70b02d82aef server_cookie=9ee68599e502db0d in_seq=0 out_seq=0 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.712+0000 7f6ce3e00700 10 client.25454 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.712+0000 7f6ce3e00700 1 -- 172.21.15.52:0/65969076 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1eb9f1810 con 0x55e1ebab44e0 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.712+0000 7f6ce3e00700 10 client.25454 handle_command_reply: tid=0 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 client.25454 shutdown 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 msgr2=0x55e1eba98f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000be50 tx=0x7f6cf000be80 comp rx=0 tx=0).stop 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 shutdown_connections 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebab44e0 0x55e1ebab6930 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba9c560 0x55e1eba994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1eba9bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/65969076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba916f0 0x55e1eba98f60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6ce3e00700 0 client.25454 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 >> 172.21.15.52:0/65969076 conn(0x55e1eb9c00e0 msgr2=0x55e1eba8ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 shutdown_connections 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 1 -- 172.21.15.52:0/65969076 wait complete. 2024-04-26T01:47:36.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.713+0000 7f6d131d3b80 20 client.25454 trim_cache size 0 max 16384 2024-04-26T01:47:36.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 Processor -- start 2024-04-26T01:47:36.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 -- start start 2024-04-26T01:47:36.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1ebaacd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaad250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001ad880 con 0x55e1eba90dd0 2024-04-26T01:47:36.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001ab080 con 0x55e1eba9c560 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.735+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00106be0 con 0x55e1eba916f0 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59936/0 (socket says 172.21.15.52:59936) 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 -- 172.21.15.52:0/2893549810 learned_addr learned my addr 172.21.15.52:0/2893549810 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d06800700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05400700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1ebaacd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 -- 172.21.15.52:0/2893549810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 msgr2=0x55e1ebaacd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1ebaacd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 -- 172.21.15.52:0/2893549810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 msgr2=0x55e1ebaad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaad250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d05e00700 1 -- 172.21.15.52:0/2893549810 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1eba974d0 con 0x55e1eba90dd0 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.736+0000 7f6d06800700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:36.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.737+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6cf409c770 tx=0x7f6cf406c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e319431f95f0d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.737+0000 7f6ceea00700 1 -- 172.21.15.52:0/2893549810 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40a56e0 con 0x55e1eba90dd0 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.737+0000 7f6ceea00700 1 -- 172.21.15.52:0/2893549810 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf409ccd0 con 0x55e1eba90dd0 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.737+0000 7f6ceea00700 1 -- 172.21.15.52:0/2893549810 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf409c120 con 0x55e1eba90dd0 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2893549810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 msgr2=0x55e1ebab0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6cf409c770 tx=0x7f6cf406c350 comp rx=0 tx=0).stop 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2893549810 shutdown_connections 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba90dd0 0x55e1ebab0810 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2893549810 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba916f0 0x55e1ebaacd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2893549810 >> 172.21.15.52:0/2893549810 conn(0x55e1eb9c00e0 msgr2=0x55e1eba8c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2893549810 shutdown_connections 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2893549810 wait complete. 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.738+0000 7f6d131d3b80 1 -- start start 2024-04-26T01:47:36.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 Processor -- start 2024-04-26T01:47:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba90dd0 0x55e1ebaac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaa7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001a9540 con 0x55e1eba916f0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001aa9b0 con 0x55e1eba9c560 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d001a9c10 con 0x55e1eba90dd0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.739+0000 7f6d131d3b80 10 client.0 _ll_register_callbacks cb 0x55e1eb97c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59950/0 (socket says 172.21.15.52:59950) 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d06800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaa7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 -- 172.21.15.52:0/2444582321 learned_addr learned my addr 172.21.15.52:0/2444582321 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba90dd0 0x55e1ebaac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 -- 172.21.15.52:0/2444582321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba90dd0 msgr2=0x55e1ebaac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba90dd0 0x55e1ebaac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 -- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 msgr2=0x55e1ebaa7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaa7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.740+0000 7f6d05400700 1 -- 172.21.15.52:0/2444582321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ebaa8690 con 0x55e1eba916f0 2024-04-26T01:47:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6d05400700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009990 tx=0x7f6cf000b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871ffb58ecdf88e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6ce3e00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf003b060 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f6cf0005620 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf409ccd0 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6cf0003ad0 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6d131d3b80 10 client.15378 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1ebb1c5b0 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.741+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf409ccd0 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f6cf000f8e0 con 0x55e1eba916f0 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 10 client.15378 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 10 client.15378 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 20 client.15378 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebb1cdc0 0x55e1ebb1f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 4 client.15378 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:47:36.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x55e1eb9eb8b0 con 0x55e1ebb1cdc0 2024-04-26T01:47:36.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.742+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebb1cdc0 0x55e1ebb1f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:36.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.743+0000 7f6d05e00700 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebb1cdc0 0x55e1ebb1f1e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7bf48ecbf9fc94d server_cookie=ded47cd7a552137f in_seq=0 out_seq=0 2024-04-26T01:47:36.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.743+0000 7f6ce3e00700 10 client.15378 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:36.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.805+0000 7f6ce3e00700 1 -- 172.21.15.52:0/2444582321 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55e1eb9eb8b0 con 0x55e1ebb1cdc0 2024-04-26T01:47:36.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.805+0000 7f6ce3e00700 10 client.15378 handle_command_reply: tid=0 2024-04-26T01:47:36.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.805+0000 7f6d131d3b80 1 client.15378 shutdown 2024-04-26T01:47:36.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 msgr2=0x55e1ebaacf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009990 tx=0x7f6cf000b610 comp rx=0 tx=0).stop 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 shutdown_connections 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e1ebb1cdc0 0x55e1ebb1f1e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e1eba9c560 0x55e1ebaa7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e1eba90dd0 0x55e1ebaac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 --2- 172.21.15.52:0/2444582321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e1eba916f0 0x55e1ebaacf00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.806+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 >> 172.21.15.52:0/2444582321 conn(0x55e1eb9c00e0 msgr2=0x55e1eba8cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.807+0000 7f6ce3e00700 0 client.15378 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.807+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 shutdown_connections 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.807+0000 7f6d131d3b80 1 -- 172.21.15.52:0/2444582321 wait complete. 2024-04-26T01:47:36.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.807+0000 7f6d131d3b80 20 client.15378 trim_cache size 0 max 16384 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 30, 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 6, 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "73d09ee": { 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 6, 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:47:36.806 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 90.0, 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:47:36.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:47:36.808 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:47:36.808 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:47:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 msgr2=0x7f6ce80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 0x7f6ce80a6390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc012820 tx=0x7f6cfc00b040 comp rx=0 tx=0).stop 2024-04-26T01:47:36.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 msgr2=0x7f6d0019b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e9a0 tx=0x7f6cf400ed60 comp rx=0 tx=0).stop 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 shutdown_connections 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ce80a3ed0 0x7f6ce80a6390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d0010f4a0 0x7f6d0019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d00074c70 0x7f6d0019b290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 --2- 172.21.15.52:0/69113423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d00075590 0x7f6d0019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 >> 172.21.15.52:0/69113423 conn(0x7f6d00079b10 msgr2=0x7f6d000ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.808+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 shutdown_connections 2024-04-26T01:47:36.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:47:36.809+0000 7f6d08800700 1 -- 172.21.15.52:0/69113423 wait complete. 2024-04-26T01:47:37.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 5.16 MB/sec warmup 56 sec latency 789.063 ms 2024-04-26T01:47:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:37 smithi052 ceph-mon[27851]: pgmap v81: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 726 KiB/s wr, 153 op/s 2024-04-26T01:47:37.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:37 smithi191 ceph-mon[32745]: pgmap v81: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 726 KiB/s wr, 153 op/s 2024-04-26T01:47:37.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:37 smithi183 ceph-mon[32752]: pgmap v81: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 726 KiB/s wr, 153 op/s 2024-04-26T01:47:37.845 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '73d09ee', quiesce took 8.3 seconds. Will release after: 30.0 2024-04-26T01:47:38.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 5.07 MB/sec warmup 57 sec latency 1265.163 ms 2024-04-26T01:47:39.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.98 MB/sec warmup 58 sec latency 2265.274 ms 2024-04-26T01:47:39.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:39 smithi191 ceph-mon[32745]: pgmap v82: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:39.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:39 smithi183 ceph-mon[32752]: pgmap v82: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:39.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:39 smithi052 ceph-mon[27851]: pgmap v82: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:40.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.90 MB/sec warmup 59 sec latency 3265.382 ms 2024-04-26T01:47:41.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.82 MB/sec warmup 60 sec latency 4265.497 ms 2024-04-26T01:47:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 -- 172.21.15.52:0/3493338531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 msgr2=0x7f32f81032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3493338531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81032f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ca0 tx=0x7f32e802a950 comp rx=0 tx=0).stop 2024-04-26T01:47:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 -- 172.21.15.52:0/3493338531 shutdown_connections 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3493338531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3493338531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3493338531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81032f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 -- 172.21.15.52:0/3493338531 >> 172.21.15.52:0/3493338531 conn(0x7f32f80fc770 msgr2=0x7f32f80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 -- 172.21.15.52:0/3493338531 shutdown_connections 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.265+0000 7f32fdc00640 1 -- 172.21.15.52:0/3493338531 wait complete. 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 Processor -- start 2024-04-26T01:47:41.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 -- start start 2024-04-26T01:47:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f81a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8112010 con 0x7f32f8102f10 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8111e90 con 0x7f32f810c330 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.267+0000 7f32fdc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8112190 con 0x7f32f8103830 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32ef400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f81a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41332/0 (socket says 172.21.15.52:41332) 2024-04-26T01:47:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 -- 172.21.15.52:0/3691410867 learned_addr learned my addr 172.21.15.52:0/3691410867 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 -- 172.21.15.52:0/3691410867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 msgr2=0x7f32f81a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f81a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 -- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 msgr2=0x7f32f81a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 -- 172.21.15.52:0/3691410867 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f819aef0 con 0x7f32f810c330 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32ef400640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f81a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.268+0000 7f32fd200640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32f000e970 tx=0x7f32f000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.269+0000 7f32f5600640 1 -- 172.21.15.52:0/3691410867 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f000ccc0 con 0x7f32f810c330 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.269+0000 7f32f5600640 1 -- 172.21.15.52:0/3691410867 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f000ce60 con 0x7f32f810c330 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.269+0000 7f32f5600640 1 -- 172.21.15.52:0/3691410867 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0010430 con 0x7f32f810c330 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.269+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f819b1e0 con 0x7f32f810c330 2024-04-26T01:47:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.269+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f81a74f0 con 0x7f32f810c330 2024-04-26T01:47:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.272+0000 7f3302066740 1 Processor -- start 2024-04-26T01:47:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.273+0000 7f32f5600640 1 -- 172.21.15.52:0/3691410867 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f32f0004ff0 con 0x7f32f810c330 2024-04-26T01:47:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.273+0000 7f3302066740 1 -- start start 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56522c640900 con 0x56522c6da9b0 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56522c5e6ef0 con 0x56522c6e5b90 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f3302066740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56522c63c8c0 con 0x56522c6db2d0 2024-04-26T01:47:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59976/0 (socket says 172.21.15.52:59976) 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 -- 172.21.15.52:0/2760002733 learned_addr learned my addr 172.21.15.52:0/2760002733 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32f7e00640 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 -- 172.21.15.52:0/2760002733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 msgr2=0x56522c6e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 -- 172.21.15.52:0/2760002733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 msgr2=0x56522c6e7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.274+0000 7f32ef400640 1 -- 172.21.15.52:0/2760002733 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56522c353210 con 0x56522c6da9b0 2024-04-26T01:47:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32ef400640 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f32e0013bc0 tx=0x7f32e0009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbc5089a1a8b3a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32f5600640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 0x7f32dc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32ed600640 1 -- 172.21.15.52:0/2760002733 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0013cb0 con 0x56522c6da9b0 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32ed600640 1 -- 172.21.15.52:0/2760002733 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e0013e50 con 0x56522c6da9b0 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32ed600640 1 -- 172.21.15.52:0/2760002733 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0003ba0 con 0x56522c6da9b0 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32f5600640 1 -- 172.21.15.52:0/3691410867 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f32f00d0b10 con 0x7f32f810c330 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.275+0000 7f32f7e00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 0x7f32dc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 -- 172.21.15.52:0/2760002733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 msgr2=0x56522c6dad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f32e0013bc0 tx=0x7f32e0009b40 comp rx=0 tx=0).stop 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 -- 172.21.15.52:0/2760002733 shutdown_connections 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 --2- 172.21.15.52:0/2760002733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6dad90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 -- 172.21.15.52:0/2760002733 >> 172.21.15.52:0/2760002733 conn(0x56522c6d6ec0 msgr2=0x56522c6d72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 -- 172.21.15.52:0/2760002733 shutdown_connections 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f3302066740 1 -- 172.21.15.52:0/2760002733 wait complete. 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.276+0000 7f32f7e00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 0x7f32dc0a6250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f32e802af40 tx=0x7f32e8005d80 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 -- start start 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 Processor -- start 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56522c6e87c0 con 0x56522c6da9b0 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56522c5fa310 con 0x56522c6e5b90 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.277+0000 7f3302066740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56522c6e8940 con 0x56522c6db2d0 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f3302066740 10 client.0 _ll_register_callbacks cb 0x56522c5c8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59988/0 (socket says 172.21.15.52:59988) 2024-04-26T01:47:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 -- 172.21.15.52:0/4281131991 learned_addr learned my addr 172.21.15.52:0/4281131991 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32fd200640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32ef400640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 -- 172.21.15.52:0/4281131991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 msgr2=0x56522c6e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 -- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 msgr2=0x56522c6e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32f7e00640 1 -- 172.21.15.52:0/4281131991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56522c41f620 con 0x56522c6da9b0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.278+0000 7f32fd200640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f32f7e00640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f32e8035000 tx=0x7f32e80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806fa746c2e1bd2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f32e6000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8055060 con 0x56522c6da9b0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f32e8003f70 con 0x56522c6da9b0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e0013e50 con 0x56522c6da9b0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f32e8034040 con 0x56522c6da9b0 2024-04-26T01:47:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f3302066740 10 client.15390 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.279+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56522c6f49e0 con 0x56522c6da9b0 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e0013e50 con 0x56522c6da9b0 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f32e805a900 con 0x56522c6da9b0 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 10 client.15390 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 10 client.15390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 20 client.15390 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c6f5f10 0x56522c6f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 4 client.15390 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56522c6548b0 con 0x56522c6f5f10 2024-04-26T01:47:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.280+0000 7f32ef400640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c6f5f10 0x56522c6f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.281+0000 7f32ef400640 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c6f5f10 0x56522c6f83b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26255e93263339eb server_cookie=1e2dd25d2be50823 in_seq=0 out_seq=0 2024-04-26T01:47:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.281+0000 7f32e6000640 10 client.15390 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f32e6000640 1 -- 172.21.15.52:0/4281131991 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56522c6548b0 con 0x56522c6f5f10 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f32e6000640 10 client.15390 handle_command_reply: tid=0 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 client.15390 shutdown 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 msgr2=0x56522c6e2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f32e8035000 tx=0x7f32e80095c0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 shutdown_connections 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c6f5f10 0x56522c6f83b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c6e3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f32e6000640 0 client.15390 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 --2- 172.21.15.52:0/4281131991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6e2af0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.282+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 >> 172.21.15.52:0/4281131991 conn(0x56522c6d6ec0 msgr2=0x56522c6dcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.283+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 shutdown_connections 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.283+0000 7f3302066740 1 -- 172.21.15.52:0/4281131991 wait complete. 2024-04-26T01:47:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.283+0000 7f3302066740 20 client.15390 trim_cache size 0 max 16384 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 Processor -- start 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 -- start start 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6f1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6e5b90 0x56522c6fdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b2510 con 0x56522c6da9b0 2024-04-26T01:47:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81afd50 con 0x56522c6db2d0 2024-04-26T01:47:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.299+0000 7f3302066740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81adba0 con 0x56522c6e5b90 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41354/0 (socket says 172.21.15.52:41354) 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 -- 172.21.15.52:0/2999170943 learned_addr learned my addr 172.21.15.52:0/2999170943 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32fd200640 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6e5b90 0x56522c6fdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 -- 172.21.15.52:0/2999170943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6e5b90 msgr2=0x56522c6fdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6e5b90 0x56522c6fdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 -- 172.21.15.52:0/2999170943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 msgr2=0x56522c6f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.300+0000 7f32ef400640 1 -- 172.21.15.52:0/2999170943 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56522c6e3a50 con 0x56522c6db2d0 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f32ef400640 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x56522c6e2350 tx=0x7f32e0003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d52f61945ea49d97 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f32ecc00640 1 -- 172.21.15.52:0/2999170943 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e002f070 con 0x56522c6db2d0 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f32ecc00640 1 -- 172.21.15.52:0/2999170943 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e0017040 con 0x56522c6db2d0 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f32ecc00640 1 -- 172.21.15.52:0/2999170943 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0005600 con 0x56522c6db2d0 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f3302066740 1 -- 172.21.15.52:0/2999170943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 msgr2=0x56522c6dd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.301+0000 7f3302066740 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x56522c6e2350 tx=0x7f32e0003700 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 -- 172.21.15.52:0/2999170943 shutdown_connections 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6e5b90 0x56522c6fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6db2d0 0x56522c6dd370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 --2- 172.21.15.52:0/2999170943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c6f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 -- 172.21.15.52:0/2999170943 >> 172.21.15.52:0/2999170943 conn(0x56522c6d6ec0 msgr2=0x56522c6e3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 -- 172.21.15.52:0/2999170943 shutdown_connections 2024-04-26T01:47:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 -- 172.21.15.52:0/2999170943 wait complete. 2024-04-26T01:47:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 -- start start 2024-04-26T01:47:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.302+0000 7f3302066740 1 Processor -- start 2024-04-26T01:47:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c753d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81ae210 con 0x56522c6da9b0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81af680 con 0x56522c6e5b90 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81ae8e0 con 0x56522c6db2d0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f3302066740 10 client.0 _ll_register_callbacks cb 0x7f32e8003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59998/0 (socket says 172.21.15.52:59998) 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 -- 172.21.15.52:0/2642819501 learned_addr learned my addr 172.21.15.52:0/2642819501 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32ef400640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c753d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32fd200640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 -- 172.21.15.52:0/2642819501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 msgr2=0x56522c753d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c753d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 -- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 msgr2=0x56522c6eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.303+0000 7f32f7e00640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32f7e00640 1 -- 172.21.15.52:0/2642819501 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56522c6eb700 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32f7e00640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32e8003db0 tx=0x7f32e805abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93cae4238b983fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32e6000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8055050 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f32e80457a0 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e0017040 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f32e806b050 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f3302066740 10 client.15396 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.304+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56522c75c050 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e0017040 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f32e8062070 con 0x56522c6da9b0 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 10 client.15396 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 20 client.15396 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c75c7d0 0x56522c75ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 4 client.15396 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56522c64e320 con 0x56522c75c7d0 2024-04-26T01:47:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.305+0000 7f32ef400640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c75c7d0 0x56522c75ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.306+0000 7f32ef400640 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c75c7d0 0x56522c75ebf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df7a08c30bc608bc server_cookie=162e25b786a425f2 in_seq=0 out_seq=0 2024-04-26T01:47:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.306+0000 7f32e6000640 10 client.15396 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f32e6000640 1 -- 172.21.15.52:0/2642819501 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56522c64e320 con 0x56522c75c7d0 2024-04-26T01:47:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f32e6000640 10 client.15396 handle_command_reply: tid=0 2024-04-26T01:47:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 client.15396 shutdown 2024-04-26T01:47:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 msgr2=0x56522c753840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32e8003db0 tx=0x7f32e805abb0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 shutdown_connections 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56522c75c7d0 0x56522c75ebf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.334+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56522c6e5b90 0x56522c6eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56522c6db2d0 0x56522c753d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 1 --2- 172.21.15.52:0/2642819501 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56522c6da9b0 0x56522c753840 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f32e6000640 0 client.15396 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 >> 172.21.15.52:0/2642819501 conn(0x56522c6d6ec0 msgr2=0x56522c6d5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 shutdown_connections 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 1 -- 172.21.15.52:0/2642819501 wait complete. 2024-04-26T01:47:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.335+0000 7f3302066740 20 client.15396 trim_cache size 0 max 16384 2024-04-26T01:47:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:47:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:47:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:47:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 msgr2=0x7f32dc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 0x7f32dc0a6250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f32e802af40 tx=0x7f32e8005d80 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 msgr2=0x7f32f819a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32f000e970 tx=0x7f32f000ed70 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 shutdown_connections 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f32dc0a3d90 0x7f32dc0a6250 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32f810c330 0x7f32f819a630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32f8103830 0x7f32f81a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 --2- 172.21.15.52:0/3691410867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f32f8102f10 0x7f32f81a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 >> 172.21.15.52:0/3691410867 conn(0x7f32f80fc770 msgr2=0x7f32f8100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 shutdown_connections 2024-04-26T01:47:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:41.336+0000 7f32fdc00640 1 -- 172.21.15.52:0/3691410867 wait complete. 2024-04-26T01:47:41.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:41 smithi191 ceph-mon[32745]: pgmap v83: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:41 smithi183 ceph-mon[32752]: pgmap v83: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:41.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:41 smithi052 ceph-mon[27851]: pgmap v83: 97 pgs: 97 active+clean; 40 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 455 KiB/s wr, 100 op/s 2024-04-26T01:47:42.063 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.74 MB/sec warmup 61 sec latency 5265.674 ms 2024-04-26T01:47:43.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.66 MB/sec warmup 62 sec latency 6265.786 ms 2024-04-26T01:47:43.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:43 smithi191 ceph-mon[32745]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 490 KiB/s wr, 111 op/s 2024-04-26T01:47:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:43 smithi183 ceph-mon[32752]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 490 KiB/s wr, 111 op/s 2024-04-26T01:47:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:43 smithi052 ceph-mon[27851]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 490 KiB/s wr, 111 op/s 2024-04-26T01:47:44.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.59 MB/sec warmup 63 sec latency 7265.928 ms 2024-04-26T01:47:45.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.51 MB/sec warmup 64 sec latency 8266.042 ms 2024-04-26T01:47:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:45 smithi191 ceph-mon[32745]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 233 KiB/s wr, 48 op/s 2024-04-26T01:47:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:45 smithi183 ceph-mon[32752]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 233 KiB/s wr, 48 op/s 2024-04-26T01:47:45.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:45 smithi052 ceph-mon[27851]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 233 KiB/s wr, 48 op/s 2024-04-26T01:47:46.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.44 MB/sec warmup 65 sec latency 9266.204 ms 2024-04-26T01:47:47.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.38 MB/sec warmup 66 sec latency 10266.370 ms 2024-04-26T01:47:47.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:47 smithi191 ceph-mon[32745]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-04-26T01:47:47.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:47 smithi183 ceph-mon[32752]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-04-26T01:47:47.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:47 smithi052 ceph-mon[27851]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-04-26T01:47:48.064 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.31 MB/sec warmup 67 sec latency 11266.526 ms 2024-04-26T01:47:49.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.25 MB/sec warmup 68 sec latency 12266.693 ms 2024-04-26T01:47:49.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:49 smithi191 ceph-mon[32745]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:49 smithi183 ceph-mon[32752]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:49.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:49 smithi052 ceph-mon[27851]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:50.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.19 MB/sec warmup 69 sec latency 13267.205 ms 2024-04-26T01:47:51.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.13 MB/sec warmup 70 sec latency 14267.351 ms 2024-04-26T01:47:51.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:51 smithi191 ceph-mon[32745]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:51.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:51 smithi183 ceph-mon[32752]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:51 smithi052 ceph-mon[27851]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-04-26T01:47:52.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.07 MB/sec warmup 71 sec latency 15267.481 ms 2024-04-26T01:47:53.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 4.01 MB/sec warmup 72 sec latency 16267.644 ms 2024-04-26T01:47:53.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:53 smithi191 ceph-mon[32745]: pgmap v89: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-04-26T01:47:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:53 smithi183 ceph-mon[32752]: pgmap v89: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-04-26T01:47:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:53 smithi052 ceph-mon[27851]: pgmap v89: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-04-26T01:47:54.065 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.96 MB/sec warmup 73 sec latency 17267.787 ms 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 -- 172.21.15.52:0/2928588271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc104bb0 msgr2=0x7f7bdc108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 --2- 172.21.15.52:0/2928588271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc104bb0 0x7f7bdc108bf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009900 tx=0x7f7bc802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 -- 172.21.15.52:0/2928588271 shutdown_connections 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 --2- 172.21.15.52:0/2928588271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 --2- 172.21.15.52:0/2928588271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc104bb0 0x7f7bdc108bf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.483+0000 7f7be2800640 1 --2- 172.21.15.52:0/2928588271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc104200 0x7f7bdc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.484+0000 7f7be2800640 1 -- 172.21.15.52:0/2928588271 >> 172.21.15.52:0/2928588271 conn(0x7f7bdc0ffa60 msgr2=0x7f7bdc101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.484+0000 7f7be2800640 1 -- 172.21.15.52:0/2928588271 shutdown_connections 2024-04-26T01:47:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.484+0000 7f7be2800640 1 -- 172.21.15.52:0/2928588271 wait complete. 2024-04-26T01:47:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.485+0000 7f7be2800640 1 Processor -- start 2024-04-26T01:47:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.485+0000 7f7be2800640 1 -- start start 2024-04-26T01:47:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc104200 0x7f7bdc1a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 0x7f7bdc19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc114240 con 0x7f7bdc1092d0 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1140c0 con 0x7f7bdc104200 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be2800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1143c0 con 0x7f7bdc19a5e0 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be0a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be0a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41044/0 (socket says 172.21.15.52:41044) 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be0a00640 1 -- 172.21.15.52:0/2806687151 learned_addr learned my addr 172.21.15.52:0/2806687151 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be1e00640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 0x7f7bdc19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.486+0000 7f7be1400640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc104200 0x7f7bdc1a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 -- 172.21.15.52:0/2806687151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 msgr2=0x7f7bdc19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 0x7f7bdc19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 -- 172.21.15.52:0/2806687151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc104200 msgr2=0x7f7bdc1a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc104200 0x7f7bdc1a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 -- 172.21.15.52:0/2806687151 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bdc19b100 con 0x7f7bdc1092d0 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be1e00640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 0x7f7bdc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be0a00640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8005ab0 tx=0x7f7bc8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7bda000640 1 -- 172.21.15.52:0/2806687151 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc803f070 con 0x7f7bdc1092d0 2024-04-26T01:47:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bdc19b390 con 0x7f7bdc1092d0 2024-04-26T01:47:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7bda000640 1 -- 172.21.15.52:0/2806687151 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc8031b50 con 0x7f7bdc1092d0 2024-04-26T01:47:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.487+0000 7f7bda000640 1 -- 172.21.15.52:0/2806687151 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8035030 con 0x7f7bdc1092d0 2024-04-26T01:47:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.488+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bdc1a7540 con 0x7f7bdc1092d0 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.491+0000 7f7bda000640 1 -- 172.21.15.52:0/2806687151 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7bc8042070 con 0x7f7bdc1092d0 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.491+0000 7f7be6c36740 1 Processor -- start 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.491+0000 7f7be6c36740 1 -- start start 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x557423386650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557423386b90 0x557423388f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5574232e1900 con 0x55742337b9b0 2024-04-26T01:47:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557423287ef0 con 0x557423386b90 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be6c36740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5574232dd8c0 con 0x55742337c2d0 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41052/0 (socket says 172.21.15.52:41052) 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be1e00640 1 -- 172.21.15.52:0/1003379923 learned_addr learned my addr 172.21.15.52:0/1003379923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be1400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x557423386650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.492+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557423386b90 0x557423388f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 -- 172.21.15.52:0/1003379923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 msgr2=0x557423386650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x557423386650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 -- 172.21.15.52:0/1003379923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557423386b90 msgr2=0x557423388f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557423386b90 0x557423388f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 -- 172.21.15.52:0/1003379923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557422ff4210 con 0x55742337b9b0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1400640 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x557423386650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7bda000640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 0x7f7bc40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7bda000640 1 -- 172.21.15.52:0/2806687151 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7bc80f0de0 con 0x7f7bdc1092d0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1e00640 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4011440 tx=0x7f7bd4011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4041bbfc36349a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7be1400640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 0x7f7bc40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7bce600640 1 -- 172.21.15.52:0/1003379923 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4004020 con 0x55742337b9b0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.493+0000 7f7bce600640 1 -- 172.21.15.52:0/1003379923 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd40041c0 con 0x55742337b9b0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7bce600640 1 -- 172.21.15.52:0/1003379923 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd400f7c0 con 0x55742337b9b0 2024-04-26T01:47:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 -- 172.21.15.52:0/1003379923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 msgr2=0x55742337bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be1400640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 0x7f7bc40a61d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55742337cf50 tx=0x7f7bd0009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4011440 tx=0x7f7bd4011900 comp rx=0 tx=0).stop 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 -- 172.21.15.52:0/1003379923 shutdown_connections 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557423386b90 0x557423388f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x557423386650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1003379923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x55742337bd90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.494+0000 7f7be6c36740 1 -- 172.21.15.52:0/1003379923 >> 172.21.15.52:0/1003379923 conn(0x557423377ec0 msgr2=0x5574233782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.495+0000 7f7be6c36740 1 -- 172.21.15.52:0/1003379923 shutdown_connections 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.495+0000 7f7be6c36740 1 -- 172.21.15.52:0/1003379923 wait complete. 2024-04-26T01:47:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.495+0000 7f7be6c36740 1 -- start start 2024-04-26T01:47:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.495+0000 7f7be6c36740 1 Processor -- start 2024-04-26T01:47:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x557423384060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557423386b90 0x5574233845a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5574233897c0 con 0x55742337b9b0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55742329b310 con 0x55742337c2d0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557423389940 con 0x557423386b90 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be6c36740 10 client.0 _ll_register_callbacks cb 0x557423269cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be1400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be1400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41056/0 (socket says 172.21.15.52:41056) 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be1400640 1 -- 172.21.15.52:0/1006763309 learned_addr learned my addr 172.21.15.52:0/1006763309 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be1e00640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557423386b90 0x5574233845a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.496+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x557423384060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 -- 172.21.15.52:0/1006763309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557423386b90 msgr2=0x5574233845a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557423386b90 0x5574233845a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 -- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 msgr2=0x557423384060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x557423384060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 -- 172.21.15.52:0/1006763309 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574230c0620 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be1400640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000fc90 tx=0x7f7bd0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5dfff6ed2a2065 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7bc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00044c0 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7bd0004660 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.497+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd40041c0 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7bd0028440 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 10 client.15408 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557423395a80 con 0x55742337b9b0 2024-04-26T01:47:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd40041c0 con 0x55742337b9b0 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f7bd0026020 con 0x55742337b9b0 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 10 client.15408 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 10 client.15408 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 20 client.15408 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557423396f40 0x557423399330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 4 client.15408 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.498+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574232f58b0 con 0x557423396f40 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.499+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557423396f40 0x557423399330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.499+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557423396f40 0x557423399330 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4460390781009f67 server_cookie=95c0092b0e360c94 in_seq=0 out_seq=0 2024-04-26T01:47:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.499+0000 7f7bc2a00640 10 client.15408 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1006763309 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574232f58b0 con 0x557423396f40 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7bc2a00640 10 client.15408 handle_command_reply: tid=0 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 client.15408 shutdown 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 msgr2=0x557423383b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000fc90 tx=0x7f7bd0004ab0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 shutdown_connections 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557423396f40 0x557423399330 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557423386b90 0x5574233845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x557423384060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1006763309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55742337b9b0 0x557423383b20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 >> 172.21.15.52:0/1006763309 conn(0x557423377ec0 msgr2=0x55742337dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.500+0000 7f7bc2a00640 0 client.15408 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.501+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 shutdown_connections 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.501+0000 7f7be6c36740 1 -- 172.21.15.52:0/1006763309 wait complete. 2024-04-26T01:47:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.501+0000 7f7be6c36740 20 client.15408 trim_cache size 0 max 16384 2024-04-26T01:47:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 Processor -- start 2024-04-26T01:47:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 -- start start 2024-04-26T01:47:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x55742337e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1b2470 con 0x557423386b90 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1afcb0 con 0x55742337c2d0 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.517+0000 7f7be6c36740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1adb20 con 0x55742337b9b0 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40372/0 (socket says 172.21.15.52:40372) 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 -- 172.21.15.52:0/438659676 learned_addr learned my addr 172.21.15.52:0/438659676 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be0a00640 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x55742337e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 -- 172.21.15.52:0/438659676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 msgr2=0x55742337e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x55742337e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 -- 172.21.15.52:0/438659676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 msgr2=0x5574233841f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233841f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 -- 172.21.15.52:0/438659676 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742337cc80 con 0x55742337b9b0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.518+0000 7f7be1400640 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000fc60 tx=0x7f7bd0031950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b851c3e721f2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7bcdc00640 1 -- 172.21.15.52:0/438659676 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0031af0 con 0x55742337b9b0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7bcdc00640 1 -- 172.21.15.52:0/438659676 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd00048c0 con 0x55742337b9b0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7bcdc00640 1 -- 172.21.15.52:0/438659676 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0028940 con 0x55742337b9b0 2024-04-26T01:47:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 -- 172.21.15.52:0/438659676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 msgr2=0x557423392680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000fc60 tx=0x7f7bd0031950 comp rx=0 tx=0).stop 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 -- 172.21.15.52:0/438659676 shutdown_connections 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233841f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337c2d0 0x55742337e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 --2- 172.21.15.52:0/438659676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337b9b0 0x557423392680 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.519+0000 7f7be6c36740 1 -- 172.21.15.52:0/438659676 >> 172.21.15.52:0/438659676 conn(0x557423377ec0 msgr2=0x557423383ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.520+0000 7f7be6c36740 1 -- 172.21.15.52:0/438659676 shutdown_connections 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.520+0000 7f7be6c36740 1 -- 172.21.15.52:0/438659676 wait complete. 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.520+0000 7f7be6c36740 1 -- start start 2024-04-26T01:47:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.520+0000 7f7be6c36740 1 Processor -- start 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 0x55742338bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233fb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1ae170 con 0x557423386b90 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1af5e0 con 0x55742337b9b0 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc1ae840 con 0x55742337c2d0 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be6c36740 10 client.0 _ll_register_callbacks cb 0x7f7bd000faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233fb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40380/0 (socket says 172.21.15.52:40380) 2024-04-26T01:47:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 -- 172.21.15.52:0/1462657036 learned_addr learned my addr 172.21.15.52:0/1462657036 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be1400640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 0x55742338bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 -- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 msgr2=0x55742338bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 0x55742338bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 -- 172.21.15.52:0/1462657036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 msgr2=0x5574233fb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.521+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233fb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be0a00640 1 -- 172.21.15.52:0/1462657036 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574233fba40 con 0x55742337c2d0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be1400640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 0x55742338bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be1e00640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233fb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be0a00640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80098d0 tx=0x7f7bc80dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d896986bab8a3ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7bc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80d6060 con 0x55742337c2d0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd00048c0 con 0x55742337c2d0 2024-04-26T01:47:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f7bc80d6200 con 0x55742337c2d0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7bc8049040 con 0x55742337c2d0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be6c36740 10 client.25507 fetch_fsmap learned FSMap version 30 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.522+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5574233fcdb0 con 0x55742337c2d0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd00048c0 con 0x55742337c2d0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f7bd0011d20 con 0x55742337c2d0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 10 client.25507 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 10 client.25507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 20 client.25507 populate_metadata read hostname 'smithi052' 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5574233fd4c0 0x5574233ff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 4 client.25507 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55742337f720 con 0x5574233fd4c0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.523+0000 7f7be1400640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5574233fd4c0 0x5574233ff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.524+0000 7f7be1400640 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5574233fd4c0 0x5574233ff8e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d13bfa762ed9b1 server_cookie=a757700e1da046e3 in_seq=0 out_seq=0 2024-04-26T01:47:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.524+0000 7f7bc2a00640 10 client.25507 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7bc2a00640 1 -- 172.21.15.52:0/1462657036 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55742337f720 con 0x5574233fd4c0 2024-04-26T01:47:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7bc2a00640 10 client.25507 handle_command_reply: tid=0 2024-04-26T01:47:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 client.25507 shutdown 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 msgr2=0x55742338c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80098d0 tx=0x7f7bc80dd040 comp rx=0 tx=0).stop 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 shutdown_connections 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5574233fd4c0 0x5574233ff8e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557423386b90 0x5574233fb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55742337c2d0 0x55742338c290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 --2- 172.21.15.52:0/1462657036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55742337b9b0 0x55742338bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7bc2a00640 0 client.25507 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 >> 172.21.15.52:0/1462657036 conn(0x557423377ec0 msgr2=0x557423383a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 shutdown_connections 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.546+0000 7f7be6c36740 1 -- 172.21.15.52:0/1462657036 wait complete. 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.547+0000 7f7be6c36740 20 client.25507 trim_cache size 0 max 16384 2024-04-26T01:47:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:47:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:47:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:47:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:47:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.547+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 msgr2=0x7f7bc40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.547+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 0x7f7bc40a61d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55742337cf50 tx=0x7f7bd0009500 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.547+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 msgr2=0x7f7bdc1a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.547+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8005ab0 tx=0x7f7bc8003b90 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 shutdown_connections 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f7bc40a3d10 0x7f7bc40a61d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7bdc19a5e0 0x7f7bdc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7bdc1092d0 0x7f7bdc1a0920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 --2- 172.21.15.52:0/2806687151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bdc104200 0x7f7bdc1a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 >> 172.21.15.52:0/2806687151 conn(0x7f7bdc0ffa60 msgr2=0x7f7bdc101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 shutdown_connections 2024-04-26T01:47:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:47:54.548+0000 7f7be2800640 1 -- 172.21.15.52:0/2806687151 wait complete. 2024-04-26T01:47:55.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.90 MB/sec warmup 74 sec latency 18267.932 ms 2024-04-26T01:47:55.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:55 smithi191 ceph-mon[32745]: pgmap v90: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-04-26T01:47:55.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:55 smithi183 ceph-mon[32752]: pgmap v90: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-04-26T01:47:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:55 smithi052 ceph-mon[27851]: pgmap v90: 97 pgs: 97 active+clean; 33 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-04-26T01:47:56.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.85 MB/sec warmup 75 sec latency 19268.077 ms 2024-04-26T01:47:57.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.80 MB/sec warmup 76 sec latency 20268.209 ms 2024-04-26T01:47:57.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:57 smithi191 ceph-mon[32745]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-04-26T01:47:57.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:57 smithi183 ceph-mon[32752]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-04-26T01:47:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:57 smithi052 ceph-mon[27851]: pgmap v91: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-04-26T01:47:58.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.75 MB/sec warmup 77 sec latency 21268.354 ms 2024-04-26T01:47:59.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.70 MB/sec warmup 78 sec latency 22268.488 ms 2024-04-26T01:47:59.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:47:59 smithi191 ceph-mon[32745]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:47:59.403 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:47:59 smithi052 ceph-mon[27851]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:47:59.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:47:59 smithi183 ceph-mon[32752]: pgmap v92: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:48:00.066 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.66 MB/sec warmup 79 sec latency 23268.621 ms 2024-04-26T01:48:01.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.61 MB/sec warmup 80 sec latency 24268.761 ms 2024-04-26T01:48:01.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:01 smithi191 ceph-mon[32745]: pgmap v93: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:48:01.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:01 smithi183 ceph-mon[32752]: pgmap v93: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:48:01.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:01 smithi052 ceph-mon[27851]: pgmap v93: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2024-04-26T01:48:02.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.57 MB/sec warmup 81 sec latency 25268.869 ms 2024-04-26T01:48:03.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.52 MB/sec warmup 82 sec latency 26269.011 ms 2024-04-26T01:48:03.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:03 smithi191 ceph-mon[32745]: pgmap v94: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-26T01:48:03.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:03 smithi183 ceph-mon[32752]: pgmap v94: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-26T01:48:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:03 smithi052 ceph-mon[27851]: pgmap v94: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-26T01:48:04.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.48 MB/sec warmup 83 sec latency 27269.177 ms 2024-04-26T01:48:05.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.44 MB/sec warmup 84 sec latency 28269.326 ms 2024-04-26T01:48:05.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:05 smithi191 ceph-mon[32745]: pgmap v95: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-04-26T01:48:05.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:05 smithi183 ceph-mon[32752]: pgmap v95: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-04-26T01:48:05.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:05 smithi052 ceph-mon[27851]: pgmap v95: 97 pgs: 97 active+clean; 34 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 43 KiB/s wr, 18 op/s 2024-04-26T01:48:06.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.40 MB/sec warmup 85 sec latency 29269.492 ms 2024-04-26T01:48:07.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.36 MB/sec warmup 86 sec latency 30269.668 ms 2024-04-26T01:48:07.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:07 smithi191 ceph-mon[32745]: pgmap v96: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-04-26T01:48:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:07 smithi183 ceph-mon[32752]: pgmap v96: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-04-26T01:48:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:07 smithi052 ceph-mon[27851]: pgmap v96: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 32 op/s 2024-04-26T01:48:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 -- 172.21.15.52:0/4261845189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108e10 msgr2=0x7f72e4115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 --2- 172.21.15.52:0/4261845189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108e10 0x7f72e4115250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009cd0 tx=0x7f72dc02a8e0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 -- 172.21.15.52:0/4261845189 shutdown_connections 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 --2- 172.21.15.52:0/4261845189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108e10 0x7f72e4115250 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 --2- 172.21.15.52:0/4261845189 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108490 0x7f72e41088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 --2- 172.21.15.52:0/4261845189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 -- 172.21.15.52:0/4261845189 >> 172.21.15.52:0/4261845189 conn(0x7f72e4077d30 msgr2=0x7f72e4106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.693+0000 7f72e9e00640 1 -- 172.21.15.52:0/4261845189 shutdown_connections 2024-04-26T01:48:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.694+0000 7f72e9e00640 1 -- 172.21.15.52:0/4261845189 wait complete. 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.694+0000 7f72e9e00640 1 Processor -- start 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 -- start start 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 0x7f72e41a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41185e0 con 0x7f72e4108e10 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72e4118460 con 0x7f72e410e420 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72e4118760 con 0x7f72e4108490 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 0x7f72e41a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 0x7f72e41a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:38634/0 (socket says 172.21.15.52:38634) 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57408/0 (socket says 172.21.15.52:57408) 2024-04-26T01:48:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e8a00640 1 -- 172.21.15.52:0/1136760229 learned_addr learned my addr 172.21.15.52:0/1136760229 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.695+0000 7f72e9400640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 -- 172.21.15.52:0/1136760229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 msgr2=0x7f72e41a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 0x7f72e41a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 -- 172.21.15.52:0/1136760229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 msgr2=0x7f72e419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 -- 172.21.15.52:0/1136760229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e419f0c0 con 0x7f72e4108e10 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e9400640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e3e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9f0 tx=0x7f72d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e1600640 1 -- 172.21.15.52:0/1136760229 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d000ccc0 con 0x7f72e4108e10 2024-04-26T01:48:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e1600640 1 -- 172.21.15.52:0/1136760229 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d00108e0 con 0x7f72e4108e10 2024-04-26T01:48:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e419f3b0 con 0x7f72e4108e10 2024-04-26T01:48:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.696+0000 7f72e1600640 1 -- 172.21.15.52:0/1136760229 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0018880 con 0x7f72e4108e10 2024-04-26T01:48:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.697+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e41ab650 con 0x7f72e4108e10 2024-04-26T01:48:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72ee36e740 1 Processor -- start 2024-04-26T01:48:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72e1600640 1 -- 172.21.15.52:0/1136760229 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f72d0010430 con 0x7f72e4108e10 2024-04-26T01:48:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72ee36e740 1 -- start start 2024-04-26T01:48:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d27d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 0x5632a9d32650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.700+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72ee36e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5632a9c8d900 con 0x5632a9d279b0 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72ee36e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5632a9c33ef0 con 0x5632a9d282d0 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72ee36e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5632a9c898c0 con 0x5632a9d32b90 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e9400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d27d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 0x5632a9d32650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e9400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d27d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57410/0 (socket says 172.21.15.52:57410) 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e9400640 1 -- 172.21.15.52:0/3539755349 learned_addr learned my addr 172.21.15.52:0/3539755349 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e3e00640 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 -- 172.21.15.52:0/3539755349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 msgr2=0x5632a9d34f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d34f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 -- 172.21.15.52:0/3539755349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 msgr2=0x5632a9d27d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d27d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.701+0000 7f72e8a00640 1 -- 172.21.15.52:0/3539755349 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632a99a0210 con 0x5632a9d282d0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72e1600640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 0x7f72cc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 0x5632a9d32650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009cf0 tx=0x7f72d8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6521fd1fa3ad4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72e1600640 1 -- 172.21.15.52:0/1136760229 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f72d00d0b30 con 0x7f72e4108e10 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72d5c00640 1 -- 172.21.15.52:0/3539755349 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d801e070 con 0x5632a9d282d0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72e9400640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 0x7f72cc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72d5c00640 1 -- 172.21.15.52:0/3539755349 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d8019ae0 con 0x5632a9d282d0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.702+0000 7f72d5c00640 1 -- 172.21.15.52:0/3539755349 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d800f890 con 0x5632a9d282d0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- 172.21.15.52:0/3539755349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 msgr2=0x5632a9d32650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72e9400640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 0x7f72cc0a61d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72dc004af0 tx=0x7f72dc03d040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 0x5632a9d32650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009cf0 tx=0x7f72d8011810 comp rx=0 tx=0).stop 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- 172.21.15.52:0/3539755349 shutdown_connections 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d282d0 0x5632a9d32650 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3539755349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d27d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- 172.21.15.52:0/3539755349 >> 172.21.15.52:0/3539755349 conn(0x5632a9d23ec0 msgr2=0x5632a9d242a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- 172.21.15.52:0/3539755349 shutdown_connections 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- 172.21.15.52:0/3539755349 wait complete. 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.703+0000 7f72ee36e740 1 -- start start 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 Processor -- start 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d279b0 0x5632a9d2fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d32b90 0x5632a9d30510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5632a9d357c0 con 0x5632a9d282d0 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5632a9c47310 con 0x5632a9d32b90 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5632a9d35940 con 0x5632a9d279b0 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72ee36e740 10 client.0 _ll_register_callbacks cb 0x5632a9c15cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57420/0 (socket says 172.21.15.52:57420) 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.704+0000 7f72e3e00640 1 -- 172.21.15.52:0/3567139449 learned_addr learned my addr 172.21.15.52:0/3567139449 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e9400640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d32b90 0x5632a9d30510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d279b0 0x5632a9d2fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 -- 172.21.15.52:0/3567139449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d279b0 msgr2=0x5632a9d2fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d279b0 0x5632a9d2fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 -- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d32b90 msgr2=0x5632a9d30510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d32b90 0x5632a9d30510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 -- 172.21.15.52:0/3567139449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632a9a6c620 con 0x5632a9d282d0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72e3e00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72d009c850 tx=0x7f72d0010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85668f6c0b0c0d99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72ca000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.705+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d00a0e50 con 0x5632a9d282d0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f72d0010a90 con 0x5632a9d282d0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d8019ae0 con 0x5632a9d282d0 2024-04-26T01:48:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f72d00a5710 con 0x5632a9d282d0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ee36e740 10 client.15426 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5632a9d419d0 con 0x5632a9d282d0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d8019ae0 con 0x5632a9d282d0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f72d00b9080 con 0x5632a9d282d0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ee36e740 10 client.15426 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.706+0000 7f72ee36e740 10 client.15426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.707+0000 7f72ee36e740 20 client.15426 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.707+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9d42e90 0x5632a9d452a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.707+0000 7f72ee36e740 4 client.15426 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.707+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632a9ca18b0 con 0x5632a9d42e90 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.707+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9d42e90 0x5632a9d452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.708+0000 7f72e8a00640 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9d42e90 0x5632a9d452a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46fd64cd2fd559ac server_cookie=d01d3a099d290935 in_seq=0 out_seq=0 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.708+0000 7f72ca000640 10 client.15426 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.708+0000 7f72ca000640 1 -- 172.21.15.52:0/3567139449 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632a9ca18b0 con 0x5632a9d42e90 2024-04-26T01:48:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.708+0000 7f72ca000640 10 client.15426 handle_command_reply: tid=0 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.708+0000 7f72ee36e740 1 client.15426 shutdown 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 msgr2=0x5632a9d2ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72d009c850 tx=0x7f72d0010720 comp rx=0 tx=0).stop 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 shutdown_connections 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9d42e90 0x5632a9d452a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d32b90 0x5632a9d30510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d282d0 0x5632a9d2ffd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 --2- 172.21.15.52:0/3567139449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d279b0 0x5632a9d2fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ca000640 0 client.15426 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 >> 172.21.15.52:0/3567139449 conn(0x5632a9d23ec0 msgr2=0x5632a9d29c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 shutdown_connections 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 1 -- 172.21.15.52:0/3567139449 wait complete. 2024-04-26T01:48:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.709+0000 7f72ee36e740 20 client.15426 trim_cache size 0 max 16384 2024-04-26T01:48:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.725+0000 7f72ee36e740 1 Processor -- start 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 -- start start 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d30160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d4abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b6890 con 0x5632a9d279b0 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b40d0 con 0x5632a9d306a0 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72ee36e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b1f20 con 0x5632a9d32b90 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57426/0 (socket says 172.21.15.52:57426) 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72e8a00640 1 -- 172.21.15.52:0/1307805595 learned_addr learned my addr 172.21.15.52:0/1307805595 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72e9400640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d4abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.726+0000 7f72e3e00640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d30160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 -- 172.21.15.52:0/1307805595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 msgr2=0x5632a9d30160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d30160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 -- 172.21.15.52:0/1307805595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 msgr2=0x5632a9d4abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d4abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 -- 172.21.15.52:0/1307805595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632a9d2e020 con 0x5632a9d279b0 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e3e00640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d30160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72e8a00640 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009ab0 tx=0x7f72d8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5192fc8bccb862 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72d5200640 1 -- 172.21.15.52:0/1307805595 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d801e040 con 0x5632a9d279b0 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72d5200640 1 -- 172.21.15.52:0/1307805595 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d801f040 con 0x5632a9d279b0 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.727+0000 7f72d5200640 1 -- 172.21.15.52:0/1307805595 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d802b250 con 0x5632a9d279b0 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- 172.21.15.52:0/1307805595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 msgr2=0x5632a9d3e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009ab0 tx=0x7f72d8005f90 comp rx=0 tx=0).stop 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- 172.21.15.52:0/1307805595 shutdown_connections 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d4abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d30160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 --2- 172.21.15.52:0/1307805595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3e680 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- 172.21.15.52:0/1307805595 >> 172.21.15.52:0/1307805595 conn(0x5632a9d23ec0 msgr2=0x5632a9d2fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- 172.21.15.52:0/1307805595 shutdown_connections 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- 172.21.15.52:0/1307805595 wait complete. 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.728+0000 7f72ee36e740 1 -- start start 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 Processor -- start 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b2590 con 0x5632a9d279b0 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b3a00 con 0x5632a9d306a0 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b2c60 con 0x5632a9d32b90 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.729+0000 7f72ee36e740 10 client.0 _ll_register_callbacks cb 0x7f72d009c760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e8a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:39334/0 (socket says 172.21.15.52:39334) 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 -- 172.21.15.52:0/4142108707 learned_addr learned my addr 172.21.15.52:0/4142108707 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e9400640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 -- 172.21.15.52:0/4142108707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 msgr2=0x5632a9d395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d395c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 -- 172.21.15.52:0/4142108707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 msgr2=0x5632a9d3ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 -- 172.21.15.52:0/4142108707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632a9d39b80 con 0x5632a9d306a0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72e3e00640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9c0 tx=0x7f72d009cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa179646dcbb9831 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72ca000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.730+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d00b6020 con 0x5632a9d306a0 2024-04-26T01:48:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d801f040 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f72d00988a0 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f72d00a0e50 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 10 client.25514 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5632a9daa240 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d801f040 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f72d00b9070 con 0x5632a9d306a0 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 10 client.25514 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 10 client.25514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 20 client.25514 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9daa8b0 0x5632a9daccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 4 client.25514 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.731+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632a9c9b320 con 0x5632a9daa8b0 2024-04-26T01:48:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.732+0000 7f72e8a00640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9daa8b0 0x5632a9daccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.732+0000 7f72e8a00640 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9daa8b0 0x5632a9daccd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d4ce1a7efb2e454 server_cookie=596602a50ad894fe in_seq=0 out_seq=0 2024-04-26T01:48:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.732+0000 7f72ca000640 10 client.25514 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.748+0000 7f72ca000640 1 -- 172.21.15.52:0/4142108707 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632a9c9b320 con 0x5632a9daa8b0 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.748+0000 7f72ca000640 10 client.25514 handle_command_reply: tid=0 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 client.25514 shutdown 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 msgr2=0x5632a9d39080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9c0 tx=0x7f72d009cb50 comp rx=0 tx=0).stop 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 shutdown_connections 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632a9daa8b0 0x5632a9daccd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632a9d32b90 0x5632a9d395c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632a9d306a0 0x5632a9d39080 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 --2- 172.21.15.52:0/4142108707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632a9d279b0 0x5632a9d3ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ca000640 0 client.25514 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 >> 172.21.15.52:0/4142108707 conn(0x5632a9d23ec0 msgr2=0x5632a9d293b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 shutdown_connections 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 1 -- 172.21.15.52:0/4142108707 wait complete. 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.749+0000 7f72ee36e740 20 client.25514 trim_cache size 0 max 16384 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:48:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:48:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:48:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:48:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 msgr2=0x7f72cc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 0x7f72cc0a61d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72dc004af0 tx=0x7f72dc03d040 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 msgr2=0x7f72e41a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9f0 tx=0x7f72d000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 shutdown_connections 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f72cc0a3d10 0x7f72cc0a61d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72e410e420 0x7f72e419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f72e4108e10 0x7f72e41a4cd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 --2- 172.21.15.52:0/1136760229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e4108490 0x7f72e41a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.750+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 >> 172.21.15.52:0/1136760229 conn(0x7f72e4077d30 msgr2=0x7f72e4078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.751+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 shutdown_connections 2024-04-26T01:48:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:07.751+0000 7f72e9e00640 1 -- 172.21.15.52:0/1136760229 wait complete. 2024-04-26T01:48:07.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:48:08.067 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.32 MB/sec warmup 87 sec latency 31269.762 ms 2024-04-26T01:48:08.326 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:09.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.28 MB/sec warmup 88 sec latency 32269.928 ms 2024-04-26T01:48:09.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:09 smithi191 ceph-mon[32745]: pgmap v97: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:09.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:09 smithi183 ceph-mon[32752]: pgmap v97: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:09 smithi052 ceph-mon[27851]: pgmap v97: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:10.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.25 MB/sec warmup 89 sec latency 33270.076 ms 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 -- 172.21.15.52:0/4055168789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 msgr2=0x7f70a41052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 --2- 172.21.15.52:0/4055168789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a41052c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7098009990 tx=0x7f7098009ca0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 -- 172.21.15.52:0/4055168789 shutdown_connections 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 --2- 172.21.15.52:0/4055168789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4109fc0 0x7f70a4111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 --2- 172.21.15.52:0/4055168789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a41052c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 --2- 172.21.15.52:0/4055168789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4105890 0x7f70a41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 -- 172.21.15.52:0/4055168789 >> 172.21.15.52:0/4055168789 conn(0x7f70a4100770 msgr2=0x7f70a4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 -- 172.21.15.52:0/4055168789 shutdown_connections 2024-04-26T01:48:10.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.565+0000 7f70ab600700 1 -- 172.21.15.52:0/4055168789 wait complete. 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.566+0000 7f70ab600700 1 Processor -- start 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 -- start start 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4105890 0x7f70a419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4109fc0 0x7f70a419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4114e60 con 0x7f70a4109fc0 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4114ce0 con 0x7f70a4104ee0 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.567+0000 7f70ab600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4114fe0 con 0x7f70a4105890 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:39352/0 (socket says 172.21.15.52:39352) 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 -- 172.21.15.52:0/2219441280 learned_addr learned my addr 172.21.15.52:0/2219441280 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a3e00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4105890 0x7f70a419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a9600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4109fc0 0x7f70a419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 -- 172.21.15.52:0/2219441280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4105890 msgr2=0x7f70a419b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4105890 0x7f70a419b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 -- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4109fc0 msgr2=0x7f70a419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4109fc0 0x7f70a419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 -- 172.21.15.52:0/2219441280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41a0340 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.568+0000 7f70a8c00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70980057c0 tx=0x7f7098005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.569+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098026070 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.569+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7098021410 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.569+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098017650 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.569+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a41a05d0 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.569+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41a7eb0 con 0x7f70a4104ee0 2024-04-26T01:48:10.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.570+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a41063c0 con 0x7f70a4104ee0 2024-04-26T01:48:10.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.572+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7098029070 con 0x7f70a4104ee0 2024-04-26T01:48:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.574+0000 7f70a1600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 0x7f708c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.574+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f70980d1470 con 0x7f70a4104ee0 2024-04-26T01:48:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.574+0000 7f70a3e00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 0x7f708c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:10.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.575+0000 7f70a3e00700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 0x7f708c0a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a4101c70 tx=0x7f7090009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:10.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.576+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7098013070 con 0x7f70a4104ee0 2024-04-26T01:48:10.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.854+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70a419c690 con 0x7f70a4104ee0 2024-04-26T01:48:10.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.855+0000 7f70a1600700 1 -- 172.21.15.52:0/2219441280 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11889 (secure 0 0 0) 0x7f709809dc00 con 0x7f70a4104ee0 2024-04-26T01:48:10.854 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:48:10.854 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:47:28.026495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 msgr2=0x7f708c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 0x7f708c0a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a4101c70 tx=0x7f7090009260 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 msgr2=0x7f70a419b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70980057c0 tx=0x7f7098005ef0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 shutdown_connections 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f708c0a3e30 0x7f708c0a62f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70a4109fc0 0x7f70a419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.858+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70a4104ee0 0x7f70a419b330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.859+0000 7f70ab600700 1 --2- 172.21.15.52:0/2219441280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70a4105890 0x7f70a419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.859+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 >> 172.21.15.52:0/2219441280 conn(0x7f70a4100770 msgr2=0x7f70a4102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:10.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.859+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 shutdown_connections 2024-04-26T01:48:10.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:10.859+0000 7f70ab600700 1 -- 172.21.15.52:0/2219441280 wait complete. 2024-04-26T01:48:10.858 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-04-26T01:48:11.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.21 MB/sec warmup 90 sec latency 34270.187 ms 2024-04-26T01:48:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:11 smithi191 ceph-mon[32745]: pgmap v98: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:11 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2219441280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:11 smithi183 ceph-mon[32752]: pgmap v98: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:11 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2219441280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:11 smithi052 ceph-mon[27851]: pgmap v98: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-04-26T01:48:11.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:11 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2219441280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:11.762 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 73d09ee --release --await' 2024-04-26T01:48:11.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 73d09ee --release --await 2024-04-26T01:48:12.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.17 MB/sec warmup 91 sec latency 35270.294 ms 2024-04-26T01:48:12.253 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:13.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.14 MB/sec warmup 92 sec latency 36270.413 ms 2024-04-26T01:48:13.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:13 smithi191 ceph-mon[32745]: pgmap v99: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 27 op/s 2024-04-26T01:48:13.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:13 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:13.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:13 smithi183 ceph-mon[32752]: pgmap v99: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 27 op/s 2024-04-26T01:48:13.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:13 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:13.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:13 smithi052 ceph-mon[27851]: pgmap v99: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 27 op/s 2024-04-26T01:48:13.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:13 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:48:14.068 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50412 3.11 MB/sec warmup 93 sec latency 37270.524 ms 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 -- 172.21.15.52:0/3361995788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58077890 msgr2=0x7fdb581133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 --2- 172.21.15.52:0/3361995788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58077890 0x7fdb581133c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009990 tx=0x7fdb44009ca0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 -- 172.21.15.52:0/3361995788 shutdown_connections 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 --2- 172.21.15.52:0/3361995788 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58113900 0x7fdb58115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 --2- 172.21.15.52:0/3361995788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58076f70 0x7fdb58077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 --2- 172.21.15.52:0/3361995788 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58077890 0x7fdb581133c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 -- 172.21.15.52:0/3361995788 >> 172.21.15.52:0/3361995788 conn(0x7fdb58100750 msgr2=0x7fdb58102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 -- 172.21.15.52:0/3361995788 shutdown_connections 2024-04-26T01:48:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.529+0000 7fdb60400700 1 -- 172.21.15.52:0/3361995788 wait complete. 2024-04-26T01:48:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.530+0000 7fdb60400700 1 Processor -- start 2024-04-26T01:48:14.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.530+0000 7fdb60400700 1 -- start start 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 0x7fdb581a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58113900 0x7fdb5819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581192d0 con 0x7fdb58077890 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdb58119150 con 0x7fdb58076f70 2024-04-26T01:48:14.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb60400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdb58119450 con 0x7fdb58113900 2024-04-26T01:48:14.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 0x7fdb581a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.531+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 0x7fdb581a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38284/0 (socket says 172.21.15.52:38284) 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37698/0 (socket says 172.21.15.52:37698) 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5d000700 1 -- 172.21.15.52:0/689014488 learned_addr learned my addr 172.21.15.52:0/689014488 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5e400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58113900 0x7fdb5819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 -- 172.21.15.52:0/689014488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58113900 msgr2=0x7fdb5819f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58113900 0x7fdb5819f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 -- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 msgr2=0x7fdb581a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 0x7fdb581a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 -- 172.21.15.52:0/689014488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5819fd50 con 0x7fdb58076f70 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.532+0000 7fdb5da00700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00e9f0 tx=0x7fdb4c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.534+0000 7fdb56000700 1 -- 172.21.15.52:0/689014488 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00c9c0 con 0x7fdb58076f70 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.534+0000 7fdb56000700 1 -- 172.21.15.52:0/689014488 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb4c00cb60 con 0x7fdb58076f70 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.534+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb581a0040 con 0x7fdb58076f70 2024-04-26T01:48:14.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.534+0000 7fdb56000700 1 -- 172.21.15.52:0/689014488 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c018680 con 0x7fdb58076f70 2024-04-26T01:48:14.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.534+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb581ac2b0 con 0x7fdb58076f70 2024-04-26T01:48:14.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.538+0000 7fdb56000700 1 -- 172.21.15.52:0/689014488 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdb4c010430 con 0x7fdb58076f70 2024-04-26T01:48:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 Processor -- start 2024-04-26T01:48:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 -- start start 2024-04-26T01:48:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558d84cc4f90 con 0x558d84d72660 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558d84c6b830 con 0x558d84d71d40 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.539+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558d84cc0f50 con 0x558d84d7d4d0 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37706/0 (socket says 172.21.15.52:37706) 2024-04-26T01:48:14.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5e400700 1 -- 172.21.15.52:0/1590891636 learned_addr learned my addr 172.21.15.52:0/1590891636 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb56000700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 0x7fdb400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 -- 172.21.15.52:0/1590891636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 msgr2=0x558d84d72120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 -- 172.21.15.52:0/1590891636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 msgr2=0x558d84d7cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5d000700 1 -- 172.21.15.52:0/1590891636 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d84c61780 con 0x558d84d7d4d0 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb5e400700 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.540+0000 7fdb56000700 1 -- 172.21.15.52:0/689014488 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fdb4c0d0990 con 0x7fdb58076f70 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.541+0000 7fdb5e400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 0x7fdb400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.541+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7f8c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb440096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7746a4b9a0e25b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.541+0000 7fdb4aa00700 1 -- 172.21.15.52:0/1590891636 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44027070 con 0x558d84d7d4d0 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.541+0000 7fdb4aa00700 1 -- 172.21.15.52:0/1590891636 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb440056a0 con 0x558d84d7d4d0 2024-04-26T01:48:14.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.541+0000 7fdb4aa00700 1 -- 172.21.15.52:0/1590891636 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44003a60 con 0x558d84d7d4d0 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1590891636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 msgr2=0x558d84d7f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb5e400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 0x7fdb400a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb50007620 tx=0x7fdb50009040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7f8c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb440096a0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1590891636 shutdown_connections 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7f8c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d71d40 0x558d84d72120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1590891636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1590891636 >> 172.21.15.52:0/1590891636 conn(0x558d84ca1050 msgr2=0x558d84d6e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1590891636 shutdown_connections 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.542+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1590891636 wait complete. 2024-04-26T01:48:14.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.543+0000 7fdb6ad85b80 1 -- start start 2024-04-26T01:48:14.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.543+0000 7fdb6ad85b80 1 Processor -- start 2024-04-26T01:48:14.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.543+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d7cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.543+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.543+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558d84d7ff60 con 0x558d84d71d40 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558d84c7e960 con 0x558d84d72660 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558d84d800e0 con 0x558d84d7d4d0 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb6ad85b80 10 client.0 _ll_register_callbacks cb 0x558d84c5d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d7cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37718/0 (socket says 172.21.15.52:37718) 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 -- 172.21.15.52:0/1424923238 learned_addr learned my addr 172.21.15.52:0/1424923238 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5e400700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 -- 172.21.15.52:0/1424923238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 msgr2=0x558d84d7a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 -- 172.21.15.52:0/1424923238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 msgr2=0x558d84d7cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d7cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5d000700 1 -- 172.21.15.52:0/1424923238 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d84c129d0 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.544+0000 7fdb5e400700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400bfe0 tx=0x7fdb4400be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e64264dc8a515e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb3f400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44030020 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fdb44003a90 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb440056a0 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdb4403b050 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb6ad85b80 10 client.25532 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.545+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558d84c5bbf0 con 0x558d84d72660 2024-04-26T01:48:14.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb440056a0 con 0x558d84d72660 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fdb44021070 con 0x558d84d72660 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 10 client.25532 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 10 client.25532 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 20 client.25532 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84d8ca90 0x558d84d8ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 4 client.25532 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.546+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d84cd2780 con 0x558d84d8ca90 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.547+0000 7fdb5da00700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84d8ca90 0x558d84d8ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.547+0000 7fdb5da00700 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84d8ca90 0x558d84d8ee80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3027db2d50ae5859 server_cookie=5f4aa46bbdb83390 in_seq=0 out_seq=0 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.547+0000 7fdb3f400700 10 client.25532 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb3f400700 1 -- 172.21.15.52:0/1424923238 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d84cd2780 con 0x558d84d8ca90 2024-04-26T01:48:14.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb3f400700 10 client.25532 handle_command_reply: tid=0 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 client.25532 shutdown 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 msgr2=0x558d84d79ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400bfe0 tx=0x7fdb4400be00 comp rx=0 tx=0).stop 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 shutdown_connections 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84d8ca90 0x558d84d8ee80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d7d4d0 0x558d84d7a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d7cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1424923238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d72660 0x558d84d79ea0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb3f400700 0 client.25532 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 >> 172.21.15.52:0/1424923238 conn(0x558d84ca1050 msgr2=0x558d84d6de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 shutdown_connections 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.548+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1424923238 wait complete. 2024-04-26T01:48:14.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.549+0000 7fdb6ad85b80 20 client.25532 trim_cache size 0 max 16384 2024-04-26T01:48:14.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 Processor -- start 2024-04-26T01:48:14.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 -- start start 2024-04-26T01:48:14.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84d85390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b6130 con 0x558d84d72660 2024-04-26T01:48:14.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b3930 con 0x558d84d7d4d0 2024-04-26T01:48:14.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.573+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b1780 con 0x558d84d71d40 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58576/0 (socket says 172.21.15.52:58576) 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84d85390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 -- 172.21.15.52:0/1805967969 learned_addr learned my addr 172.21.15.52:0/1805967969 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5d000700 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 -- 172.21.15.52:0/1805967969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 msgr2=0x558d84d85390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84d85390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 -- 172.21.15.52:0/1805967969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 msgr2=0x558d84d7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.574+0000 7fdb5da00700 1 -- 172.21.15.52:0/1805967969 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d84d7fcf0 con 0x558d84d71d40 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.575+0000 7fdb5da00700 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0992d0 tx=0x7fdb4c0d2e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=536b2170ef15d9d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.575+0000 7fdb4a000700 1 -- 172.21.15.52:0/1805967969 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c09c900 con 0x558d84d71d40 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.575+0000 7fdb4a000700 1 -- 172.21.15.52:0/1805967969 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb4c0a0cb0 con 0x558d84d71d40 2024-04-26T01:48:14.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.575+0000 7fdb4a000700 1 -- 172.21.15.52:0/1805967969 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c09cb60 con 0x558d84d71d40 2024-04-26T01:48:14.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1805967969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 msgr2=0x558d84d88e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0992d0 tx=0x7fdb4c0d2e50 comp rx=0 tx=0).stop 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1805967969 shutdown_connections 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84d85390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d71d40 0x558d84d88e90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/1805967969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d72660 0x558d84d7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1805967969 >> 172.21.15.52:0/1805967969 conn(0x558d84ca1050 msgr2=0x558d84d7b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1805967969 shutdown_connections 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/1805967969 wait complete. 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.576+0000 7fdb6ad85b80 1 -- start start 2024-04-26T01:48:14.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 Processor -- start 2024-04-26T01:48:14.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 0x558d84d80b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84deccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b1df0 con 0x558d84d71d40 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b3260 con 0x558d84d7d4d0 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdb581b24c0 con 0x558d84d72660 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb6ad85b80 10 client.0 _ll_register_callbacks cb 0x558d84c5d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb5da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d805e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38308/0 (socket says 172.21.15.52:38308) 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb5da00700 1 -- 172.21.15.52:0/2865923137 learned_addr learned my addr 172.21.15.52:0/2865923137 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.577+0000 7fdb5d000700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 0x558d84d80b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5e400700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84deccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 -- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 msgr2=0x558d84deccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84deccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 -- 172.21.15.52:0/2865923137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 msgr2=0x558d84d805e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:14.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d805e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 -- 172.21.15.52:0/2865923137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d84ded330 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5e400700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84deccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb5d000700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 0x558d84d80b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb44009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbd711fe5a633cef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb3f400700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.578+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44044060 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fdb44003890 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb4c0a0cb0 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdb44030030 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 10 client.25549 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558d84df4710 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb4c0a0cb0 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fdb44021040 con 0x558d84d72660 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 10 client.25549 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 10 client.25549 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:48:14.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 20 client.25549 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84df4d90 0x558d84df71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 4 client.25549 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "73d09ee", "release": true, "await": true}] 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.579+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "73d09ee", "release": true, "await": true}) -- 0x558d84ccc820 con 0x558d84df4d90 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.580+0000 7fdb5da00700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84df4d90 0x558d84df71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.580+0000 7fdb3f400700 10 client.25549 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:14.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.580+0000 7fdb5da00700 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84df4d90 0x558d84df71b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cef879b9fa20c28e server_cookie=b9f256aac2695a5b in_seq=0 out_seq=0 2024-04-26T01:48:14.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.584+0000 7fdb3f400700 1 -- 172.21.15.52:0/2865923137 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x558d84ccc820 con 0x558d84df4d90 2024-04-26T01:48:14.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.584+0000 7fdb3f400700 10 client.25549 handle_command_reply: tid=0 2024-04-26T01:48:14.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.584+0000 7fdb6ad85b80 1 client.25549 shutdown 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.584+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 msgr2=0x558d84d80b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.584+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 0x558d84d80b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006010 tx=0x7fdb44009290 comp rx=0 tx=0).stop 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 shutdown_connections 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558d84df4d90 0x558d84df71b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558d84d7d4d0 0x558d84deccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d84d71d40 0x558d84d805e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 --2- 172.21.15.52:0/2865923137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d84d72660 0x558d84d80b20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 >> 172.21.15.52:0/2865923137 conn(0x558d84ca1050 msgr2=0x558d84d7bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb3f400700 0 client.25549 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:14.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 shutdown_connections 2024-04-26T01:48:14.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 1 -- 172.21.15.52:0/2865923137 wait complete. 2024-04-26T01:48:14.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.585+0000 7fdb6ad85b80 20 client.25549 trim_cache size 0 max 16384 2024-04-26T01:48:14.584 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:48:14.584 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 30, 2024-04-26T01:48:14.584 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:48:14.584 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 8, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "73d09ee": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 8, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 90.0, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:48:14.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:14.586 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 msgr2=0x7fdb400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 0x7fdb400a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb50007620 tx=0x7fdb50009040 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 msgr2=0x7fdb581a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00e9f0 tx=0x7fdb4c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 shutdown_connections 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdb400a3db0 0x7fdb400a6270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdb58113900 0x7fdb5819f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdb58076f70 0x7fdb581a53e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 --2- 172.21.15.52:0/689014488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdb58077890 0x7fdb581a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 >> 172.21.15.52:0/689014488 conn(0x7fdb58100750 msgr2=0x7fdb58102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 shutdown_connections 2024-04-26T01:48:14.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:14.588+0000 7fdb60400700 1 -- 172.21.15.52:0/689014488 wait complete. 2024-04-26T01:48:15.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 50922 3.09 MB/sec warmup 94 sec latency 37790.713 ms 2024-04-26T01:48:15.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:15 smithi191 ceph-mon[32745]: pgmap v100: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-04-26T01:48:15.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:15 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:15.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:15 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:15.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:15 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:48:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:15 smithi183 ceph-mon[32752]: pgmap v100: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-04-26T01:48:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:15 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:15 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:15 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:48:15.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:15 smithi052 ceph-mon[27851]: pgmap v100: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 20 op/s 2024-04-26T01:48:15.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:15 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:48:15.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:15 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:48:15.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:15 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:48:15.680 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '73d09ee', seconds elapsed: 46.1 2024-04-26T01:48:15.680 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-26T01:48:15.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:48:16.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 52618 3.17 MB/sec warmup 95 sec latency 19.061 ms 2024-04-26T01:48:16.190 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:17.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53546 3.20 MB/sec warmup 96 sec latency 29.046 ms 2024-04-26T01:48:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:17 smithi183 ceph-mon[32752]: pgmap v101: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 142 KiB/s wr, 40 op/s 2024-04-26T01:48:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:17 smithi052 ceph-mon[27851]: pgmap v101: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 142 KiB/s wr, 40 op/s 2024-04-26T01:48:17.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:17 smithi191 ceph-mon[32745]: pgmap v101: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 142 KiB/s wr, 40 op/s 2024-04-26T01:48:18.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.17 MB/sec warmup 97 sec latency 286.210 ms 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 -- 172.21.15.52:0/2478299411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b38108a30 msgr2=0x7f6b38108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/2478299411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b38108a30 0x7f6b38108e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b0a0 tx=0x7f6b2800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 -- 172.21.15.52:0/2478299411 shutdown_connections 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/2478299411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806b0f0 0x7f6b3810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/2478299411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b38108a30 0x7f6b38108e10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/2478299411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b3806a750 0x7f6b3806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 -- 172.21.15.52:0/2478299411 >> 172.21.15.52:0/2478299411 conn(0x7f6b380788f0 msgr2=0x7f6b38078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 -- 172.21.15.52:0/2478299411 shutdown_connections 2024-04-26T01:48:18.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.545+0000 7f6b3ec00700 1 -- 172.21.15.52:0/2478299411 wait complete. 2024-04-26T01:48:18.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.547+0000 7f6b3ec00700 1 Processor -- start 2024-04-26T01:48:18.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.547+0000 7f6b3ec00700 1 -- start start 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.547+0000 7f6b3ec00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3ec00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3ec00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b38108a30 0x7f6b3819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3ec00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6b38112bb0 con 0x7f6b38108a30 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3ec00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6b38112a30 con 0x7f6b3806b0f0 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3ec00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6b38112d30 con 0x7f6b3806a750 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b3cc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b38108a30 0x7f6b3819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37748/0 (socket says 172.21.15.52:37748) 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37400700 1 -- 172.21.15.52:0/834002879 learned_addr learned my addr 172.21.15.52:0/834002879 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:18.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58610/0 (socket says 172.21.15.52:58610) 2024-04-26T01:48:18.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 -- 172.21.15.52:0/834002879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 msgr2=0x7f6b3819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 -- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b38108a30 msgr2=0x7f6b3819fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.548+0000 7f6b37e00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b38108a30 0x7f6b3819fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b37e00700 1 -- 172.21.15.52:0/834002879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b381a02f0 con 0x7f6b3806a750 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b37400700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b37e00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b070 tx=0x7f6b28004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b28021070 con 0x7f6b3806a750 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b28007e30 con 0x7f6b3806a750 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b28004560 con 0x7f6b3806a750 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.549+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b381a0580 con 0x7f6b3806a750 2024-04-26T01:48:18.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.550+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b381a7e60 con 0x7f6b3806a750 2024-04-26T01:48:18.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.551+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b38104170 con 0x7f6b3806a750 2024-04-26T01:48:18.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.553+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6b28004700 con 0x7f6b3806a750 2024-04-26T01:48:18.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.554+0000 7f6b34c00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 0x7f6b1c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:18.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.554+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f6b2800f020 con 0x7f6b3806a750 2024-04-26T01:48:18.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.555+0000 7f6b37400700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 0x7f6b1c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:18.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.555+0000 7f6b37400700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 0x7f6b1c0a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6b38078f60 tx=0x7f6b24009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:18.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.557+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b2802a020 con 0x7f6b3806a750 2024-04-26T01:48:18.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.847+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b3802df80 con 0x7f6b3806a750 2024-04-26T01:48:18.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.849+0000 7f6b34c00700 1 -- 172.21.15.52:0/834002879 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11889 (secure 0 0 0) 0x7f6b2809e810 con 0x7f6b3806a750 2024-04-26T01:48:18.847 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:48:18.847 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:47:28.026495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 msgr2=0x7f6b1c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 0x7f6b1c0a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6b38078f60 tx=0x7f6b24009260 comp rx=0 tx=0).stop 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 msgr2=0x7f6b3819b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800b070 tx=0x7f6b28004d00 comp rx=0 tx=0).stop 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 shutdown_connections 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6b1c0a3db0 0x7f6b1c0a6270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6b38108a30 0x7f6b3819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b3806a750 0x7f6b3819b2e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 --2- 172.21.15.52:0/834002879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6b3806b0f0 0x7f6b3819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:18.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.852+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 >> 172.21.15.52:0/834002879 conn(0x7f6b380788f0 msgr2=0x7f6b3810de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:18.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.853+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 shutdown_connections 2024-04-26T01:48:18.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:18.853+0000 7f6b3ec00700 1 -- 172.21.15.52:0/834002879 wait complete. 2024-04-26T01:48:18.852 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-04-26T01:48:19.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.14 MB/sec warmup 98 sec latency 1286.312 ms 2024-04-26T01:48:19.275 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:19 smithi052 ceph-mon[27851]: pgmap v102: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:19.275 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:19 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/834002879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:19 smithi183 ceph-mon[32752]: pgmap v102: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:19 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/834002879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:19 smithi191 ceph-mon[32745]: pgmap v102: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:19 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/834002879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:48:19.741 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0' 2024-04-26T01:48:19.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 90.0 --await-for 42.0 2024-04-26T01:48:20.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.11 MB/sec warmup 99 sec latency 2286.377 ms 2024-04-26T01:48:20.180 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:48:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 -- 172.21.15.52:0/2395270796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4970102b80 msgr2=0x7f4970106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 --2- 172.21.15.52:0/2395270796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4970102b80 0x7f4970106bc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f495c009900 tx=0x7f495c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 -- 172.21.15.52:0/2395270796 shutdown_connections 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 --2- 172.21.15.52:0/2395270796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49701072a0 0x7f497010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 --2- 172.21.15.52:0/2395270796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4970102b80 0x7f4970106bc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 --2- 172.21.15.52:0/2395270796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 -- 172.21.15.52:0/2395270796 >> 172.21.15.52:0/2395270796 conn(0x7f49700793e0 msgr2=0x7f49700797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 -- 172.21.15.52:0/2395270796 shutdown_connections 2024-04-26T01:48:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.893+0000 7f4975000640 1 -- 172.21.15.52:0/2395270796 wait complete. 2024-04-26T01:48:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.894+0000 7f4975000640 1 Processor -- start 2024-04-26T01:48:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 -- start start 2024-04-26T01:48:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497019a6f0 0x7f497019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4970112210 con 0x7f49701072a0 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4970112090 con 0x7f49701021d0 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f4975000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4970112390 con 0x7f497019a6f0 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f496d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f496d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38342/0 (socket says 172.21.15.52:38342) 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.895+0000 7f496d400640 1 -- 172.21.15.52:0/1840695271 learned_addr learned my addr 172.21.15.52:0/1840695271 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496de00640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496e800640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497019a6f0 0x7f497019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 -- 172.21.15.52:0/1840695271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497019a6f0 msgr2=0x7f497019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497019a6f0 0x7f497019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 -- 172.21.15.52:0/1840695271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 msgr2=0x7f49701a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 -- 172.21.15.52:0/1840695271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497019b2f0 con 0x7f49701072a0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496de00640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.896+0000 7f496d400640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f495c005ab0 tx=0x7f495c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.897+0000 7f4966a00640 1 -- 172.21.15.52:0/1840695271 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c03f070 con 0x7f49701072a0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.897+0000 7f4966a00640 1 -- 172.21.15.52:0/1840695271 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495c031b50 con 0x7f49701072a0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.897+0000 7f4966a00640 1 -- 172.21.15.52:0/1840695271 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c035030 con 0x7f49701072a0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.897+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497019b580 con 0x7f49701072a0 2024-04-26T01:48:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.897+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49701a7670 con 0x7f49701072a0 2024-04-26T01:48:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.900+0000 7f4977d7d740 1 Processor -- start 2024-04-26T01:48:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.900+0000 7f4966a00640 1 -- 172.21.15.52:0/1840695271 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f495c042070 con 0x7f49701072a0 2024-04-26T01:48:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.900+0000 7f4977d7d740 1 -- start start 2024-04-26T01:48:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 0x55f46ae014c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46ae03df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f46ad5c900 con 0x55f46ae01a00 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f46ad02ef0 con 0x55f46adf6820 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f4977d7d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f46ad588c0 con 0x55f46adf7140 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f496d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46ae03df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f496e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f496de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 0x55f46ae014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f496e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37788/0 (socket says 172.21.15.52:37788) 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.901+0000 7f496e800640 1 -- 172.21.15.52:0/3183122453 learned_addr learned my addr 172.21.15.52:0/3183122453 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 -- 172.21.15.52:0/3183122453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 msgr2=0x55f46ae014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 0x55f46ae014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 -- 172.21.15.52:0/3183122453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 msgr2=0x55f46ae03df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46ae03df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 -- 172.21.15.52:0/3183122453 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f46aa6f210 con 0x55f46adf6820 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496de00640 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 0x55f46ae014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f496e800640 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4968009f40 tx=0x7f49680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=467927445e1e748a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f4966a00640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 0x7f49540a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f495b400640 1 -- 172.21.15.52:0/3183122453 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496801e070 con 0x55f46adf6820 2024-04-26T01:48:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f495b400640 1 -- 172.21.15.52:0/3183122453 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4968003c30 con 0x55f46adf6820 2024-04-26T01:48:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f495b400640 1 -- 172.21.15.52:0/3183122453 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496800f7b0 con 0x55f46adf6820 2024-04-26T01:48:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.902+0000 7f4966a00640 1 -- 172.21.15.52:0/1840695271 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f495c0f0de0 con 0x7f49701072a0 2024-04-26T01:48:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f496de00640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 0x7f49540a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.905 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 -- 172.21.15.52:0/3183122453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 msgr2=0x55f46adf6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.905 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4968009f40 tx=0x7f49680118d0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.905 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 -- 172.21.15.52:0/3183122453 shutdown_connections 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f496de00640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 0x7f49540a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55f46adf7dc0 tx=0x7f4960005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46ae03df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf7140 0x55f46ae014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3183122453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46adf6c00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 -- 172.21.15.52:0/3183122453 >> 172.21.15.52:0/3183122453 conn(0x55f46adf2d30 msgr2=0x55f46adf3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 -- 172.21.15.52:0/3183122453 shutdown_connections 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.903+0000 7f4977d7d740 1 -- 172.21.15.52:0/3183122453 wait complete. 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.904+0000 7f4977d7d740 1 -- start start 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.904+0000 7f4977d7d740 1 Processor -- start 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 0x55f46adfea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adfef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46adff480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.906 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f46ae04630 con 0x55f46ae01a00 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f46ad16310 con 0x55f46adf7140 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f46ae047b0 con 0x55f46adf6820 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f4977d7d740 10 client.0 _ll_register_callbacks cb 0x55f46ace4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 0x55f46adfea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46adff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adfef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 0x55f46adfea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58640/0 (socket says 172.21.15.52:58640) 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46adff480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38366/0 (socket says 172.21.15.52:38366) 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496de00640 1 -- 172.21.15.52:0/1279093566 learned_addr learned my addr 172.21.15.52:0/1279093566 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 -- 172.21.15.52:0/1279093566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 msgr2=0x55f46adfea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 0x55f46adfea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 -- 172.21.15.52:0/1279093566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 msgr2=0x55f46adff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.907 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46adff480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.905+0000 7f496d400640 1 -- 172.21.15.52:0/1279093566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f46ab3b620 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f496d400640 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adfef40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f495c0ee500 tx=0x7f495c003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e9b4e621202da16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f494fe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c049070 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f495c03b460 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4968003c30 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f495c0bc760 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f4977d7d740 10 client.25556 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f46ae108f0 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.906+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4968003c30 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f495c0ee780 con 0x55f46adf7140 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 10 client.25556 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 10 client.25556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 20 client.25556 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:20.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae11e40 0x55f46ae142a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 4 client.25556 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f46ad708b0 con 0x55f46ae11e40 2024-04-26T01:48:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.907+0000 7f496de00640 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae11e40 0x55f46ae142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.908+0000 7f496de00640 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae11e40 0x55f46ae142a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3714f96c27b35aaa server_cookie=9f971541718f18ad in_seq=0 out_seq=0 2024-04-26T01:48:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.908+0000 7f494fe00640 10 client.25556 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f494fe00640 1 -- 172.21.15.52:0/1279093566 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f46ad708b0 con 0x55f46ae11e40 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f494fe00640 10 client.25556 handle_command_reply: tid=0 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 client.25556 shutdown 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 msgr2=0x55f46adfef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adfef40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f495c0ee500 tx=0x7f495c003f90 comp rx=0 tx=0).stop 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 shutdown_connections 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae11e40 0x55f46ae142a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46ae01a00 0x55f46adff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f494fe00640 0 client.25556 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adfef40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1279093566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46adf6820 0x55f46adfea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 >> 172.21.15.52:0/1279093566 conn(0x55f46adf2d30 msgr2=0x55f46adf8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 shutdown_connections 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 1 -- 172.21.15.52:0/1279093566 wait complete. 2024-04-26T01:48:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.909+0000 7f4977d7d740 20 client.25556 trim_cache size 0 max 16384 2024-04-26T01:48:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.926+0000 7f4977d7d740 1 Processor -- start 2024-04-26T01:48:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.926+0000 7f4977d7d740 1 -- start start 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adf91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46adff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f49701b2600 con 0x55f46adf6820 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f49701afe40 con 0x55f46adf7140 2024-04-26T01:48:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f4977d7d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49701adc90 con 0x55f46ae01a00 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38378/0 (socket says 172.21.15.52:38378) 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 -- 172.21.15.52:0/1692663872 learned_addr learned my addr 172.21.15.52:0/1692663872 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496e800640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46adff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496d400640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adf91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 -- 172.21.15.52:0/1692663872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 msgr2=0x55f46adff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46adff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 -- 172.21.15.52:0/1692663872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 msgr2=0x55f46adf91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adf91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.927+0000 7f496de00640 1 -- 172.21.15.52:0/1692663872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f46ae14920 con 0x55f46adf6820 2024-04-26T01:48:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.928+0000 7f496d400640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adf91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.928+0000 7f496de00640 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4960012e50 tx=0x7f496001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ec7b8aa795519c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.928+0000 7f495aa00640 1 -- 172.21.15.52:0/1692663872 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496000fd50 con 0x55f46adf6820 2024-04-26T01:48:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.928+0000 7f495aa00640 1 -- 172.21.15.52:0/1692663872 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496001dc90 con 0x55f46adf6820 2024-04-26T01:48:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.928+0000 7f495aa00640 1 -- 172.21.15.52:0/1692663872 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496002b8b0 con 0x55f46adf6820 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 -- 172.21.15.52:0/1692663872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 msgr2=0x55f46adf2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4960012e50 tx=0x7f496001b000 comp rx=0 tx=0).stop 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 -- 172.21.15.52:0/1692663872 shutdown_connections 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46adff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf7140 0x55f46adf91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 --2- 172.21.15.52:0/1692663872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf6820 0x55f46adf2610 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 -- 172.21.15.52:0/1692663872 >> 172.21.15.52:0/1692663872 conn(0x55f46adf2d30 msgr2=0x55f46adfe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 -- 172.21.15.52:0/1692663872 shutdown_connections 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.929+0000 7f4977d7d740 1 -- 172.21.15.52:0/1692663872 wait complete. 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 -- start start 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 Processor -- start 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46ae06ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46ae6f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f49701ae300 con 0x55f46adf7140 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f49701af770 con 0x55f46adf6820 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49701ae9d0 con 0x55f46ae01a00 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f4977d7d740 10 client.0 _ll_register_callbacks cb 0x7f495c009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.930+0000 7f496d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38384/0 (socket says 172.21.15.52:38384) 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 -- 172.21.15.52:0/3848243403 learned_addr learned my addr 172.21.15.52:0/3848243403 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496e800640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46ae6f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496de00640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46ae06ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 -- 172.21.15.52:0/3848243403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 msgr2=0x55f46ae6f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46ae6f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 -- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 msgr2=0x55f46ae06ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46ae06ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 -- 172.21.15.52:0/3848243403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f46ae6fa20 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f496d400640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f495c031440 tx=0x7f495c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fe519655460dca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f494fe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.931+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c049040 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f495c0c5730 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f496001dc90 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f495c0c1100 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 10 client.15480 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f46ae77fe0 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496001dc90 con 0x55f46adf7140 2024-04-26T01:48:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f495c0dd160 con 0x55f46adf7140 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 10 client.15480 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 20 client.15480 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae78620 0x55f46ae7aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 4 client.15480 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.932+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f46ad6a320 con 0x55f46ae78620 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.933+0000 7f496de00640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae78620 0x55f46ae7aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.933+0000 7f496de00640 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae78620 0x55f46ae7aa40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac47dc8a92d55ef server_cookie=9644b4ef1874ffc4 in_seq=0 out_seq=0 2024-04-26T01:48:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:20.933+0000 7f494fe00640 10 client.15480 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:21.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.08 MB/sec warmup 100 sec latency 3286.464 ms 2024-04-26T01:48:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:21 smithi183 ceph-mon[32752]: pgmap v103: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:21 smithi052 ceph-mon[27851]: pgmap v103: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:21 smithi191 ceph-mon[32745]: pgmap v103: 97 pgs: 97 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 140 KiB/s wr, 26 op/s 2024-04-26T01:48:22.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.05 MB/sec warmup 101 sec latency 4286.615 ms 2024-04-26T01:48:22.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.675+0000 7f610fe00700 1 -- 172.21.15.52:0/2526370086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108077890 msgr2=0x7f61081133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:22.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 --2- 172.21.15.52:0/2526370086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108077890 0x7f61081133c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009990 tx=0x7f60f4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 -- 172.21.15.52:0/2526370086 shutdown_connections 2024-04-26T01:48:22.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 --2- 172.21.15.52:0/2526370086 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 0x7f6108115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 --2- 172.21.15.52:0/2526370086 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108076f70 0x7f6108077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 --2- 172.21.15.52:0/2526370086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108077890 0x7f61081133c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 -- 172.21.15.52:0/2526370086 >> 172.21.15.52:0/2526370086 conn(0x7f6108100750 msgr2=0x7f6108102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:22.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 -- 172.21.15.52:0/2526370086 shutdown_connections 2024-04-26T01:48:22.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.676+0000 7f610fe00700 1 -- 172.21.15.52:0/2526370086 wait complete. 2024-04-26T01:48:22.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.677+0000 7f610fe00700 1 Processor -- start 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.677+0000 7f610fe00700 1 -- start start 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108077890 0x7f610819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 0x7f61081a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f61081192d0 con 0x7f6108076f70 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6108119150 con 0x7f6108113900 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610fe00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6108119450 con 0x7f6108077890 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38410/0 (socket says 172.21.15.52:38410) 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 -- 172.21.15.52:0/3182235708 learned_addr learned my addr 172.21.15.52:0/3182235708 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610ca00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108077890 0x7f610819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610de00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 0x7f61081a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 -- 172.21.15.52:0/3182235708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108077890 msgr2=0x7f610819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108077890 0x7f610819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 -- 172.21.15.52:0/3182235708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 msgr2=0x7f61081a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.678+0000 7f610d400700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 0x7f61081a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f610d400700 1 -- 172.21.15.52:0/3182235708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61081a46f0 con 0x7f6108076f70 2024-04-26T01:48:22.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f610d400700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00eb30 tx=0x7f60fc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:22.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f6106000700 1 -- 172.21.15.52:0/3182235708 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc00cc90 con 0x7f6108076f70 2024-04-26T01:48:22.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f6106000700 1 -- 172.21.15.52:0/3182235708 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60fc0044d0 con 0x7f6108076f70 2024-04-26T01:48:22.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61081a49e0 con 0x7f6108076f70 2024-04-26T01:48:22.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f6106000700 1 -- 172.21.15.52:0/3182235708 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc010430 con 0x7f6108076f70 2024-04-26T01:48:22.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.679+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081ac260 con 0x7f6108076f70 2024-04-26T01:48:22.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.683+0000 7f6106000700 1 -- 172.21.15.52:0/3182235708 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f60fc003ea0 con 0x7f6108076f70 2024-04-26T01:48:22.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.684+0000 7f611a6c4b80 1 Processor -- start 2024-04-26T01:48:22.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.684+0000 7f6106000700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 0x7f60f00a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.684+0000 7f6106000700 1 -- 172.21.15.52:0/3182235708 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f60fc0d0ab0 con 0x7f6108076f70 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.684+0000 7f611a6c4b80 1 -- start start 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f610ca00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 0x7f60f00a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d18ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 0x56502d18d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56502d0d3020 con 0x56502d1802d0 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56502d0798c0 con 0x56502d18b180 2024-04-26T01:48:22.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56502d0cefe0 con 0x56502d17f9b0 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f610d400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f610d400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58678/0 (socket says 172.21.15.52:58678) 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.685+0000 7f610d400700 1 -- 172.21.15.52:0/1397447090 learned_addr learned my addr 172.21.15.52:0/1397447090 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610ca00700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d18ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610de00700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 0x56502d18d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610ca00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 0x7f60f00a6310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6108104150 tx=0x7f60f400b260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 -- 172.21.15.52:0/1397447090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 msgr2=0x56502d18d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 0x56502d18d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 -- 172.21.15.52:0/1397447090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 msgr2=0x56502d18ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d18ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 -- 172.21.15.52:0/1397447090 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56502d06f810 con 0x56502d17f9b0 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610ca00700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d18ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610de00700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 0x56502d18d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.686+0000 7f610d400700 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0a5160 tx=0x7f60fc09c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8fdaeef49de399 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.687+0000 7f60faa00700 1 -- 172.21.15.52:0/1397447090 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc09d120 con 0x56502d17f9b0 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.687+0000 7f60faa00700 1 -- 172.21.15.52:0/1397447090 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60fc09ca20 con 0x56502d17f9b0 2024-04-26T01:48:22.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.687+0000 7f60faa00700 1 -- 172.21.15.52:0/1397447090 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc004190 con 0x56502d17f9b0 2024-04-26T01:48:22.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.687+0000 7f611a6c4b80 1 -- 172.21.15.52:0/1397447090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 msgr2=0x56502d17fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:22.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.687+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0a5160 tx=0x7f60fc09c5e0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 -- 172.21.15.52:0/1397447090 shutdown_connections 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d18b180 0x56502d18d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d17fd90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/1397447090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d18ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 -- 172.21.15.52:0/1397447090 >> 172.21.15.52:0/1397447090 conn(0x56502d0ae730 msgr2=0x56502d17be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 -- 172.21.15.52:0/1397447090 shutdown_connections 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 -- 172.21.15.52:0/1397447090 wait complete. 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.688+0000 7f611a6c4b80 1 -- start start 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 Processor -- start 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 0x56502d18a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d187c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d1881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56502d18dca0 con 0x56502d1802d0 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56502d08c9f0 con 0x56502d17f9b0 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56502d18de20 con 0x56502d18b180 2024-04-26T01:48:22.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.689+0000 7f611a6c4b80 10 client.0 _ll_register_callbacks cb 0x56502d06b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:22.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d187c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610ca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d187c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38432/0 (socket says 172.21.15.52:38432) 2024-04-26T01:48:22.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610ca00700 1 -- 172.21.15.52:0/3008032625 learned_addr learned my addr 172.21.15.52:0/3008032625 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:22.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 0x56502d18a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610de00700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d1881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 -- 172.21.15.52:0/3008032625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 msgr2=0x56502d1881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d1881d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 -- 172.21.15.52:0/3008032625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 msgr2=0x56502d187c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d187c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 -- 172.21.15.52:0/3008032625 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56502d020a60 con 0x56502d17f9b0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f610d400700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 0x56502d18a990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00c5b0 tx=0x7f60fc00cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ba7dfce4645923 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f60ef400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.690+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc067060 con 0x56502d17f9b0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f60fc004190 con 0x56502d17f9b0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60fc09ca20 con 0x56502d17f9b0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60fc0a0dd0 con 0x56502d17f9b0 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f611a6c4b80 10 client.25574 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:22.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56502d069c80 con 0x56502d17f9b0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60fc09ca20 con 0x56502d17f9b0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f60fc0bb070 con 0x56502d17f9b0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f611a6c4b80 10 client.25574 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.691+0000 7f611a6c4b80 10 client.25574 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.692+0000 7f611a6c4b80 20 client.25574 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.692+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d19a8f0 0x56502d19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.692+0000 7f611a6c4b80 4 client.25574 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.692+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56502d0e0810 con 0x56502d19a8f0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.692+0000 7f610ca00700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d19a8f0 0x56502d19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.693+0000 7f610ca00700 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d19a8f0 0x56502d19cce0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a526641ce2563c54 server_cookie=94517949109fcdc8 in_seq=0 out_seq=0 2024-04-26T01:48:22.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:22.693+0000 7f60ef400700 10 client.25574 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:23.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 3.02 MB/sec warmup 102 sec latency 5286.745 ms 2024-04-26T01:48:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:23 smithi183 ceph-mon[32752]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 544 KiB/s wr, 56 op/s 2024-04-26T01:48:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:23 smithi052 ceph-mon[27851]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 544 KiB/s wr, 56 op/s 2024-04-26T01:48:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:23 smithi191 ceph-mon[32745]: pgmap v104: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 544 KiB/s wr, 56 op/s 2024-04-26T01:48:24.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.99 MB/sec warmup 103 sec latency 6286.891 ms 2024-04-26T01:48:25.069 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.96 MB/sec warmup 104 sec latency 7286.985 ms 2024-04-26T01:48:25.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:25 smithi183 ceph-mon[32752]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 536 KiB/s wr, 50 op/s 2024-04-26T01:48:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:25 smithi052 ceph-mon[27851]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 536 KiB/s wr, 50 op/s 2024-04-26T01:48:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:25 smithi191 ceph-mon[32745]: pgmap v105: 97 pgs: 97 active+clean; 39 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 536 KiB/s wr, 50 op/s 2024-04-26T01:48:26.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.93 MB/sec warmup 105 sec latency 8287.130 ms 2024-04-26T01:48:27.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.90 MB/sec warmup 106 sec latency 9287.233 ms 2024-04-26T01:48:27.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:27 smithi052 ceph-mon[27851]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 564 KiB/s wr, 65 op/s 2024-04-26T01:48:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:27 smithi191 ceph-mon[32745]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 564 KiB/s wr, 65 op/s 2024-04-26T01:48:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:27 smithi183 ceph-mon[32752]: pgmap v106: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 564 KiB/s wr, 65 op/s 2024-04-26T01:48:28.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.88 MB/sec warmup 107 sec latency 10287.385 ms 2024-04-26T01:48:29.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.85 MB/sec warmup 108 sec latency 11287.533 ms 2024-04-26T01:48:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:29 smithi052 ceph-mon[27851]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:29 smithi191 ceph-mon[32745]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:29.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:29 smithi183 ceph-mon[32752]: pgmap v107: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:30.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53763 2.82 MB/sec warmup 109 sec latency 12287.664 ms 2024-04-26T01:48:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.978+0000 7f494fe00640 1 -- 172.21.15.52:0/3848243403 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f46ad6a320 con 0x55f46ae78620 2024-04-26T01:48:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.978+0000 7f494fe00640 10 client.15480 handle_command_reply: tid=0 2024-04-26T01:48:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.978+0000 7f4977d7d740 1 client.15480 shutdown 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 msgr2=0x55f46ae07530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f495c031440 tx=0x7f495c009880 comp rx=0 tx=0).stop 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 shutdown_connections 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f46ae78620 0x55f46ae7aa40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f46ae01a00 0x55f46ae6f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f46adf7140 0x55f46ae07530 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 --2- 172.21.15.52:0/3848243403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f46adf6820 0x55f46ae06ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 >> 172.21.15.52:0/3848243403 conn(0x55f46adf2d30 msgr2=0x55f46adff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f494fe00640 0 client.15480 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:30.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f60ef400700 1 -- 172.21.15.52:0/3008032625 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56502d0e0810 con 0x56502d19a8f0 2024-04-26T01:48:30.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.979+0000 7f60ef400700 10 client.25574 handle_command_reply: tid=0 2024-04-26T01:48:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.980+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 shutdown_connections 2024-04-26T01:48:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.980+0000 7f4977d7d740 1 -- 172.21.15.52:0/3848243403 wait complete. 2024-04-26T01:48:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.980+0000 7f4977d7d740 20 client.15480 trim_cache size 0 max 16384 2024-04-26T01:48:30.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.980+0000 7f611a6c4b80 1 client.25574 shutdown 2024-04-26T01:48:30.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 msgr2=0x56502d18a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:30.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 0x56502d18a990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00c5b0 tx=0x7f60fc00cfa0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:48:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:48:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:48:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:48:30.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 shutdown_connections 2024-04-26T01:48:30.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d19a8f0 0x56502d19cce0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d1881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d17f9b0 0x56502d18a990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/3008032625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d1802d0 0x56502d187c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 >> 172.21.15.52:0/3008032625 conn(0x56502d0ae730 msgr2=0x56502d17ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f60ef400700 0 client.25574 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 shutdown_connections 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.981+0000 7f611a6c4b80 1 -- 172.21.15.52:0/3008032625 wait complete. 2024-04-26T01:48:30.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:30.982+0000 7f611a6c4b80 20 client.25574 trim_cache size 0 max 16384 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 msgr2=0x7f49540a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 0x7f49540a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55f46adf7dc0 tx=0x7f4960005f90 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 msgr2=0x7f49701a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f495c005ab0 tx=0x7f495c003b90 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 shutdown_connections 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f49540a3dc0 0x7f49540a6280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f497019a6f0 0x7f497019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f49701072a0 0x7f49701a0b70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 --2- 172.21.15.52:0/1840695271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49701021d0 0x7f49701a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.983+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 >> 172.21.15.52:0/1840695271 conn(0x7f49700793e0 msgr2=0x7f49700ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.984+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 shutdown_connections 2024-04-26T01:48:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:30.984+0000 7f4975000640 1 -- 172.21.15.52:0/1840695271 wait complete. 2024-04-26T01:48:31.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 Processor -- start 2024-04-26T01:48:31.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 -- start start 2024-04-26T01:48:31.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d19f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d19fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b6210 con 0x56502d17f9b0 2024-04-26T01:48:31.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b3a10 con 0x56502d1802d0 2024-04-26T01:48:31.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.019+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b1860 con 0x56502d18b180 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32792/0 (socket says 172.21.15.52:32792) 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610d400700 1 -- 172.21.15.52:0/2876024377 learned_addr learned my addr 172.21.15.52:0/2876024377 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610ca00700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610de00700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d19fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610ca00700 1 -- 172.21.15.52:0/2876024377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 msgr2=0x56502d19fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.020+0000 7f610ca00700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d19fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f610ca00700 1 -- 172.21.15.52:0/2876024377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 msgr2=0x56502d17bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f610ca00700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f610ca00700 1 -- 172.21.15.52:0/2876024377 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56502d17ffe0 con 0x56502d1802d0 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f610d400700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f610ca00700 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d19f930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x56502d069d10 tx=0x7f60f401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1cc4b944fbd47ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.021+0000 7f60fa000700 1 -- 172.21.15.52:0/2876024377 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f400be40 con 0x56502d1802d0 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.022+0000 7f60fa000700 1 -- 172.21.15.52:0/2876024377 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f4003b30 con 0x56502d1802d0 2024-04-26T01:48:31.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.022+0000 7f60fa000700 1 -- 172.21.15.52:0/2876024377 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4033850 con 0x56502d1802d0 2024-04-26T01:48:31.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2876024377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 msgr2=0x56502d19f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d19f930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x56502d069d10 tx=0x7f60f401f000 comp rx=0 tx=0).stop 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2876024377 shutdown_connections 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d18b180 0x56502d19fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d17f9b0 0x56502d17bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2876024377 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d19f930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2876024377 >> 172.21.15.52:0/2876024377 conn(0x56502d0ae730 msgr2=0x56502d1960b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2876024377 shutdown_connections 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.023+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2876024377 wait complete. 2024-04-26T01:48:31.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.024+0000 7f611a6c4b80 1 -- start start 2024-04-26T01:48:31.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.024+0000 7f611a6c4b80 1 Processor -- start 2024-04-26T01:48:31.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d1f8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d18b180 0x56502d1f9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b1ed0 con 0x56502d18b180 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b3340 con 0x56502d1802d0 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f61081b25a0 con 0x56502d17f9b0 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f611a6c4b80 10 client.0 _ll_register_callbacks cb 0x7f6100009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f610ca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f610ca00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49048/0 (socket says 172.21.15.52:49048) 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f610ca00700 1 -- 172.21.15.52:0/2144929444 learned_addr learned my addr 172.21.15.52:0/2144929444 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:31.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.025+0000 7f610de00700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d18b180 0x56502d1f9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 -- 172.21.15.52:0/2144929444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 msgr2=0x56502d1f8ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d1f8ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 -- 172.21.15.52:0/2144929444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d18b180 msgr2=0x56502d1f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d18b180 0x56502d1f9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 -- 172.21.15.52:0/2144929444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56502d1fa5b0 con 0x56502d1802d0 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f610ca00700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x56502d198580 tx=0x7f60f4005810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8f7d964abe5ff0f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:31.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.026+0000 7f60ef400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4040060 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f60f4003880 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f4003b30 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60f400be50 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f611a6c4b80 10 client.25580 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56502d2018f0 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.027+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f4003b30 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f60f400f800 con 0x56502d1802d0 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 10 client.25580 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 10 client.25580 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 20 client.25580 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d201f60 0x56502d204380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:31.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 4 client.25580 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}] 2024-04-26T01:48:31.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await_for": 42.0}) -- 0x56502d0da8b0 con 0x56502d201f60 2024-04-26T01:48:31.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.028+0000 7f610d400700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d201f60 0x56502d204380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:31.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.029+0000 7f610d400700 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d201f60 0x56502d204380 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e40f9f7beb3f370 server_cookie=4b4f86a776e94c4 in_seq=0 out_seq=0 2024-04-26T01:48:31.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.029+0000 7f60ef400700 10 client.25580 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:31.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.80 MB/sec warmup 110 sec latency 13230.797 ms 2024-04-26T01:48:31.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.096+0000 7f60ef400700 1 -- 172.21.15.52:0/2144929444 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56502d0da8b0 con 0x56502d201f60 2024-04-26T01:48:31.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.096+0000 7f60ef400700 10 client.25580 handle_command_reply: tid=0 2024-04-26T01:48:31.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 client.25580 shutdown 2024-04-26T01:48:31.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 msgr2=0x56502d1f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x56502d198580 tx=0x7f60f4005810 comp rx=0 tx=0).stop 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 shutdown_connections 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56502d201f60 0x56502d204380 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56502d18b180 0x56502d1f9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56502d17f9b0 0x56502d1f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 --2- 172.21.15.52:0/2144929444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56502d1802d0 0x56502d1f9530 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 >> 172.21.15.52:0/2144929444 conn(0x56502d0ae730 msgr2=0x56502d196940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.097+0000 7f60ef400700 0 client.25580 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.098+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 shutdown_connections 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.098+0000 7f611a6c4b80 1 -- 172.21.15.52:0/2144929444 wait complete. 2024-04-26T01:48:31.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.098+0000 7f611a6c4b80 20 client.25580 trim_cache size 0 max 16384 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 30, 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 10, 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "6a2a23e6": { 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 10, 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:48:31.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 90.0, 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:31.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:31.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:48:31.099 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 msgr2=0x7f60f00a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 0x7f60f00a6310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6108104150 tx=0x7f60f400b260 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 msgr2=0x7f610819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00eb30 tx=0x7f60fc00ee40 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 shutdown_connections 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f60f00a3e50 0x7f60f00a6310 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6108113900 0x7f61081a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6108076f70 0x7f610819f6e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 --2- 172.21.15.52:0/3182235708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6108077890 0x7f610819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 >> 172.21.15.52:0/3182235708 conn(0x7f6108100750 msgr2=0x7f6108102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 shutdown_connections 2024-04-26T01:48:31.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:48:31.099+0000 7f610fe00700 1 -- 172.21.15.52:0/3182235708 wait complete. 2024-04-26T01:48:31.454 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:31 smithi191 ceph-mon[32745]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:31.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:31 smithi052 ceph-mon[27851]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:31 smithi183 ceph-mon[32752]: pgmap v108: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 432 KiB/s wr, 44 op/s 2024-04-26T01:48:32.046 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6a2a23e6', quiesce took 16.4 seconds. Will release after: 30.0 2024-04-26T01:48:32.070 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.77 MB/sec warmup 111 sec latency 1019.227 ms 2024-04-26T01:48:33.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.75 MB/sec warmup 112 sec latency 2019.362 ms 2024-04-26T01:48:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:33 smithi052 ceph-mon[27851]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 433 KiB/s wr, 48 op/s 2024-04-26T01:48:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:33 smithi191 ceph-mon[32745]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 433 KiB/s wr, 48 op/s 2024-04-26T01:48:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:33 smithi183 ceph-mon[32752]: pgmap v109: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 433 KiB/s wr, 48 op/s 2024-04-26T01:48:34.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.72 MB/sec warmup 113 sec latency 3019.469 ms 2024-04-26T01:48:35.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.70 MB/sec warmup 114 sec latency 4019.595 ms 2024-04-26T01:48:35.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:35 smithi052 ceph-mon[27851]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 18 op/s 2024-04-26T01:48:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:35 smithi191 ceph-mon[32745]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 18 op/s 2024-04-26T01:48:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:35 smithi183 ceph-mon[32752]: pgmap v110: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 18 op/s 2024-04-26T01:48:36.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.68 MB/sec warmup 115 sec latency 5019.764 ms 2024-04-26T01:48:37.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.65 MB/sec warmup 116 sec latency 6019.912 ms 2024-04-26T01:48:37.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:37 smithi052 ceph-mon[27851]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-04-26T01:48:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:37 smithi191 ceph-mon[32745]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-04-26T01:48:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:37 smithi183 ceph-mon[32752]: pgmap v111: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 27 op/s 2024-04-26T01:48:38.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.63 MB/sec warmup 117 sec latency 7020.071 ms 2024-04-26T01:48:39.071 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.61 MB/sec warmup 118 sec latency 8020.215 ms 2024-04-26T01:48:39.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:39 smithi052 ceph-mon[27851]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:39 smithi191 ceph-mon[32745]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:39 smithi183 ceph-mon[32752]: pgmap v112: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:40.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 2.59 MB/sec warmup 119 sec latency 9020.353 ms 2024-04-26T01:48:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:41 smithi191 ceph-mon[32745]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:41 smithi183 ceph-mon[32752]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:41.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:41 smithi052 ceph-mon[27851]: pgmap v113: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-26T01:48:42.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 1 sec latency 11020.553 ms 2024-04-26T01:48:43.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 2 sec latency 12020.693 ms 2024-04-26T01:48:43.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:43 smithi183 ceph-mon[32752]: pgmap v114: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 16 op/s 2024-04-26T01:48:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:43 smithi191 ceph-mon[32745]: pgmap v114: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 16 op/s 2024-04-26T01:48:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:43 smithi052 ceph-mon[27851]: pgmap v114: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 16 op/s 2024-04-26T01:48:44.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 3 sec latency 13020.834 ms 2024-04-26T01:48:44.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.143+0000 7ff59a200640 1 -- 172.21.15.52:0/3344874750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff594104bb0 msgr2=0x7ff594108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.143+0000 7ff59a200640 1 --2- 172.21.15.52:0/3344874750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff594104bb0 0x7ff594108bf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff580009900 tx=0x7ff58002a410 comp rx=0 tx=0).stop 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 -- 172.21.15.52:0/3344874750 shutdown_connections 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 --2- 172.21.15.52:0/3344874750 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5941092d0 0x7ff594110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 --2- 172.21.15.52:0/3344874750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff594104bb0 0x7ff594108bf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 --2- 172.21.15.52:0/3344874750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 -- 172.21.15.52:0/3344874750 >> 172.21.15.52:0/3344874750 conn(0x7ff5940ffa60 msgr2=0x7ff594101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 -- 172.21.15.52:0/3344874750 shutdown_connections 2024-04-26T01:48:44.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.144+0000 7ff59a200640 1 -- 172.21.15.52:0/3344874750 wait complete. 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 Processor -- start 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 -- start start 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff59419a5e0 0x7ff59419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff594114240 con 0x7ff5941092d0 2024-04-26T01:48:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941140c0 con 0x7ff59419a5e0 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.145+0000 7ff59a200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941143c0 con 0x7ff594104200 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff593e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff593e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56342/0 (socket says 172.21.15.52:56342) 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff593e00640 1 -- 172.21.15.52:0/2428672244 learned_addr learned my addr 172.21.15.52:0/2428672244 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff599800640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff59419a5e0 0x7ff59419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 -- 172.21.15.52:0/2428672244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff59419a5e0 msgr2=0x7ff59419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff59419a5e0 0x7ff59419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 -- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 msgr2=0x7ff5941a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff598e00640 1 -- 172.21.15.52:0/2428672244 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59419b100 con 0x7ff594104200 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.146+0000 7ff593e00640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff598e00640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941a03e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff58800d6d0 tx=0x7ff58800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff591600640 1 -- 172.21.15.52:0/2428672244 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588017070 con 0x7ff594104200 2024-04-26T01:48:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59419b3f0 con 0x7ff594104200 2024-04-26T01:48:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff591600640 1 -- 172.21.15.52:0/2428672244 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff588009820 con 0x7ff594104200 2024-04-26T01:48:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff591600640 1 -- 172.21.15.52:0/2428672244 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588004c40 con 0x7ff594104200 2024-04-26T01:48:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.147+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5941a7640 con 0x7ff594104200 2024-04-26T01:48:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.150+0000 7ff59e616740 1 Processor -- start 2024-04-26T01:48:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.150+0000 7ff59e616740 1 -- start start 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff591600640 1 -- 172.21.15.52:0/2428672244 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff588005220 con 0x7ff594104200 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f8bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f96650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7ef1900 con 0x55d5d7f96b90 2024-04-26T01:48:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7e97ef0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff59e616740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7eed8c0 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff593e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff599800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f8bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff593e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56352/0 (socket says 172.21.15.52:56352) 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff593e00640 1 -- 172.21.15.52:0/2028544403 learned_addr learned my addr 172.21.15.52:0/2028544403 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.151+0000 7ff598e00640 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f96650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 -- 172.21.15.52:0/2028544403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 msgr2=0x55d5d7f8bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f8bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 -- 172.21.15.52:0/2028544403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 msgr2=0x55d5d7f96650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f96650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 -- 172.21.15.52:0/2028544403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d7c04210 con 0x55d5d7f96b90 2024-04-26T01:48:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff599800640 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f8bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff593e00640 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff58002aa00 tx=0x7ff580004520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4798db47ea9adb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff586000640 1 -- 172.21.15.52:0/2028544403 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580040070 con 0x55d5d7f96b90 2024-04-26T01:48:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff586000640 1 -- 172.21.15.52:0/2028544403 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5800047a0 con 0x55d5d7f96b90 2024-04-26T01:48:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff586000640 1 -- 172.21.15.52:0/2028544403 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580035040 con 0x55d5d7f96b90 2024-04-26T01:48:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.152+0000 7ff591600640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 0x7ff57c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff591600640 1 -- 172.21.15.52:0/2428672244 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7ff5880d0280 con 0x7ff594104200 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff599800640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 0x7ff57c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 -- 172.21.15.52:0/2028544403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 msgr2=0x55d5d7f98f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff58002aa00 tx=0x7ff580004520 comp rx=0 tx=0).stop 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 -- 172.21.15.52:0/2028544403 shutdown_connections 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f98f80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f96650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 --2- 172.21.15.52:0/2028544403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f8bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 -- 172.21.15.52:0/2028544403 >> 172.21.15.52:0/2028544403 conn(0x55d5d7f87ec0 msgr2=0x55d5d7f882a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 -- 172.21.15.52:0/2028544403 shutdown_connections 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff59e616740 1 -- 172.21.15.52:0/2028544403 wait complete. 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.153+0000 7ff599800640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 0x7ff57c0a61d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009fd0 tx=0x7ff58c009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.154+0000 7ff59e616740 1 -- start start 2024-04-26T01:48:44.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.154+0000 7ff59e616740 1 Processor -- start 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f93b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 0x55d5d7f945f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7f99760 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7eab310 con 0x55d5d7f96b90 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d5d7f998e0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff59e616740 10 client.0 _ll_register_callbacks cb 0x55d5d7e79cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff598e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f93b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff599800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 0x55d5d7f945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37448/0 (socket says 172.21.15.52:37448) 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 -- 172.21.15.52:0/1941513745 learned_addr learned my addr 172.21.15.52:0/1941513745 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:44.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 -- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 msgr2=0x55d5d7f945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 0x55d5d7f945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 -- 172.21.15.52:0/1941513745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 msgr2=0x55d5d7f93b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f93b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.155+0000 7ff593e00640 1 -- 172.21.15.52:0/1941513745 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d7cd0620 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff599800640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 0x55d5d7f945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:44.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff593e00640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff580000c00 tx=0x7ff580009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93e6df8fe525b2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff57aa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580040070 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7ff580005410 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5800047a0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff580056050 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff59e616740 10 client.25600 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.156+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d5d7f99760 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5800047a0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7ff5800314a0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 10 client.25600 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 10 client.25600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 20 client.25600 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d7fa6c80 0x55d5d7fa9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 4 client.25600 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.157+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5d7f058b0 con 0x55d5d7fa6c80 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.158+0000 7ff598e00640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d7fa6c80 0x55d5d7fa9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.158+0000 7ff598e00640 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d7fa6c80 0x55d5d7fa9070 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1495c7ec989a26a1 server_cookie=c59f21918e3148c in_seq=0 out_seq=0 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.158+0000 7ff57aa00640 10 client.25600 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff57aa00640 1 -- 172.21.15.52:0/1941513745 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5d7f058b0 con 0x55d5d7fa6c80 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff57aa00640 10 client.25600 handle_command_reply: tid=0 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 client.25600 shutdown 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 msgr2=0x55d5d7f940b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff580000c00 tx=0x7ff580009260 comp rx=0 tx=0).stop 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 shutdown_connections 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d7fa6c80 0x55d5d7fa9070 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f96b90 0x55d5d7f945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f940b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 --2- 172.21.15.52:0/1941513745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f93b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff57aa00640 0 client.25600 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.159+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 >> 172.21.15.52:0/1941513745 conn(0x55d5d7f87ec0 msgr2=0x55d5d7f8dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.160+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 shutdown_connections 2024-04-26T01:48:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.160+0000 7ff59e616740 1 -- 172.21.15.52:0/1941513745 wait complete. 2024-04-26T01:48:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.160+0000 7ff59e616740 20 client.25600 trim_cache size 0 max 16384 2024-04-26T01:48:44.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.176+0000 7ff59e616740 1 Processor -- start 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 -- start start 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7fae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f96b90 0x55d5d7faed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941b2470 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941afcb0 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff59e616740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941adb20 con 0x55d5d7f96b90 2024-04-26T01:48:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff598e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff598e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56370/0 (socket says 172.21.15.52:56370) 2024-04-26T01:48:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff598e00640 1 -- 172.21.15.52:0/1079606074 learned_addr learned my addr 172.21.15.52:0/1079606074 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff593e00640 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7fae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.177+0000 7ff599800640 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f96b90 0x55d5d7faed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 -- 172.21.15.52:0/1079606074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f96b90 msgr2=0x55d5d7faed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f96b90 0x55d5d7faed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 -- 172.21.15.52:0/1079606074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 msgr2=0x55d5d7fae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7fae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 -- 172.21.15.52:0/1079606074 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d7f8e9a0 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff598e00640 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff5880cda00 tx=0x7ff5880d26d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153e1184e6e9bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff585600640 1 -- 172.21.15.52:0/1079606074 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588020040 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff585600640 1 -- 172.21.15.52:0/1079606074 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff58809c2a0 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.178+0000 7ff585600640 1 -- 172.21.15.52:0/1079606074 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58809a3d0 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 -- 172.21.15.52:0/1079606074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 msgr2=0x55d5d7fa2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff5880cda00 tx=0x7ff5880d26d0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 -- 172.21.15.52:0/1079606074 shutdown_connections 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f96b90 0x55d5d7faed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7fae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 --2- 172.21.15.52:0/1079606074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7fa2620 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 -- 172.21.15.52:0/1079606074 >> 172.21.15.52:0/1079606074 conn(0x55d5d7f87ec0 msgr2=0x55d5d7f882a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 -- 172.21.15.52:0/1079606074 shutdown_connections 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.179+0000 7ff59e616740 1 -- 172.21.15.52:0/1079606074 wait complete. 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 -- start start 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 Processor -- start 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f9c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f9cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941ae170 con 0x55d5d7f96b90 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941af5e0 con 0x55d5d7f8b9b0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941ae840 con 0x55d5d7f8c2d0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.180+0000 7ff59e616740 10 client.0 _ll_register_callbacks cb 0x7ff5800043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56372/0 (socket says 172.21.15.52:56372) 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff598e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f9c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff593e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f9cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 -- 172.21.15.52:0/1315127259 learned_addr learned my addr 172.21.15.52:0/1315127259 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 -- 172.21.15.52:0/1315127259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 msgr2=0x55d5d7f9cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f9cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 -- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 msgr2=0x55d5d7f9c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f9c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff599800640 1 -- 172.21.15.52:0/1315127259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d800c2b0 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff593e00640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f9cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.181+0000 7ff598e00640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f9c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff599800640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff58c003f40 tx=0x7ff58c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7c209c34939eca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff57aa00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c02f060 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7ff58c00f610 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff58809c2a0 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff58c00f810 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff59e616740 10 client.15516 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d5d800d620 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.182+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff58809c2a0 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7ff58c029030 con 0x55d5d7f96b90 2024-04-26T01:48:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 10 client.15516 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 10 client.15516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 20 client.15516 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d800de30 0x55d5d8010250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 4 client.15516 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d5d7eff320 con 0x55d5d800de30 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.183+0000 7ff598e00640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d800de30 0x55d5d8010250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.184+0000 7ff598e00640 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d800de30 0x55d5d8010250 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9af46e31df0ddaf1 server_cookie=66957123a019903b in_seq=0 out_seq=0 2024-04-26T01:48:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.184+0000 7ff57aa00640 10 client.15516 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff57aa00640 1 -- 172.21.15.52:0/1315127259 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d5d7eff320 con 0x55d5d800de30 2024-04-26T01:48:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff57aa00640 10 client.15516 handle_command_reply: tid=0 2024-04-26T01:48:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 client.15516 shutdown 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 msgr2=0x55d5d7f9d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff58c003f40 tx=0x7ff58c004ab0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 shutdown_connections 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d5d800de30 0x55d5d8010250 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d5d7f96b90 0x55d5d7f9d240 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d5d7f8c2d0 0x55d5d7f9cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 --2- 172.21.15.52:0/1315127259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d5d7f8b9b0 0x55d5d7f9c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 >> 172.21.15.52:0/1315127259 conn(0x55d5d7f87ec0 msgr2=0x55d5d7f8d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff57aa00640 0 client.15516 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 shutdown_connections 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 1 -- 172.21.15.52:0/1315127259 wait complete. 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.205+0000 7ff59e616740 20 client.15516 trim_cache size 0 max 16384 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:48:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:48:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 msgr2=0x7ff57c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 0x7ff57c0a61d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009fd0 tx=0x7ff58c009500 comp rx=0 tx=0).stop 2024-04-26T01:48:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 msgr2=0x7ff5941a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941a03e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff58800d6d0 tx=0x7ff58800db90 comp rx=0 tx=0).stop 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 shutdown_connections 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff57c0a3d10 0x7ff57c0a61d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff59419a5e0 0x7ff59419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5941092d0 0x7ff5941a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 --2- 172.21.15.52:0/2428672244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff594104200 0x7ff5941a03e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 >> 172.21.15.52:0/2428672244 conn(0x7ff5940ffa60 msgr2=0x7ff594101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 shutdown_connections 2024-04-26T01:48:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:44.206+0000 7ff59a200640 1 -- 172.21.15.52:0/2428672244 wait complete. 2024-04-26T01:48:45.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 4 sec latency 14020.969 ms 2024-04-26T01:48:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:45 smithi191 ceph-mon[32745]: pgmap v115: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-26T01:48:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:45 smithi183 ceph-mon[32752]: pgmap v115: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-26T01:48:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:45 smithi052 ceph-mon[27851]: pgmap v115: 97 pgs: 97 active+clean; 39 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-26T01:48:46.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 5 sec latency 15021.111 ms 2024-04-26T01:48:47.072 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 6 sec latency 16021.260 ms 2024-04-26T01:48:47.514 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:47 smithi052 ceph-mon[27851]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-26T01:48:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:47 smithi191 ceph-mon[32745]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-26T01:48:47.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:47 smithi183 ceph-mon[32752]: pgmap v116: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-26T01:48:48.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 7 sec latency 17021.427 ms 2024-04-26T01:48:49.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 8 sec latency 18021.529 ms 2024-04-26T01:48:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:49 smithi191 ceph-mon[32745]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:49.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:49 smithi183 ceph-mon[32752]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:49 smithi052 ceph-mon[27851]: pgmap v117: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:50.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 9 sec latency 19021.597 ms 2024-04-26T01:48:51.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 10 sec latency 20021.682 ms 2024-04-26T01:48:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:51 smithi191 ceph-mon[32745]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:51.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:51 smithi183 ceph-mon[32752]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:51 smithi052 ceph-mon[27851]: pgmap v118: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-26T01:48:52.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 11 sec latency 21021.820 ms 2024-04-26T01:48:53.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 12 sec latency 22021.964 ms 2024-04-26T01:48:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:53 smithi191 ceph-mon[32745]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-26T01:48:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:53 smithi183 ceph-mon[32752]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-26T01:48:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:53 smithi052 ceph-mon[27851]: pgmap v119: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-26T01:48:54.073 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 13 sec latency 23022.130 ms 2024-04-26T01:48:55.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 14 sec latency 24022.331 ms 2024-04-26T01:48:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:55 smithi191 ceph-mon[32745]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T01:48:55.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:55 smithi183 ceph-mon[32752]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T01:48:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:55 smithi052 ceph-mon[27851]: pgmap v120: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-26T01:48:56.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 15 sec latency 25022.425 ms 2024-04-26T01:48:57.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 16 sec latency 26022.581 ms 2024-04-26T01:48:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 -- 172.21.15.52:0/2167299001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e541092d0 msgr2=0x7f0e54110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 --2- 172.21.15.52:0/2167299001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e541092d0 0x7f0e54110ea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009ca0 tx=0x7f0e4c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 -- 172.21.15.52:0/2167299001 shutdown_connections 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 --2- 172.21.15.52:0/2167299001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e541092d0 0x7f0e54110ea0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 --2- 172.21.15.52:0/2167299001 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e54104bb0 0x7f0e54108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 --2- 172.21.15.52:0/2167299001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e54104200 0x7f0e541045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 -- 172.21.15.52:0/2167299001 >> 172.21.15.52:0/2167299001 conn(0x7f0e540ffa60 msgr2=0x7f0e54101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 -- 172.21.15.52:0/2167299001 shutdown_connections 2024-04-26T01:48:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.358+0000 7f0e53e00640 1 -- 172.21.15.52:0/2167299001 wait complete. 2024-04-26T01:48:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.359+0000 7f0e53e00640 1 Processor -- start 2024-04-26T01:48:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.359+0000 7f0e53e00640 1 -- start start 2024-04-26T01:48:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 0x7f0e5419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e541092d0 0x7f0e5419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e54114270 con 0x7f0e5419efb0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541140f0 con 0x7f0e541092d0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541143f0 con 0x7f0e54104200 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e52a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 0x7f0e5419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48278/0 (socket says 172.21.15.52:48278) 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e53400640 1 -- 172.21.15.52:0/3456372376 learned_addr learned my addr 172.21.15.52:0/3456372376 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.360+0000 7f0e52000640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e541092d0 0x7f0e5419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 -- 172.21.15.52:0/3456372376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 msgr2=0x7f0e5419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 0x7f0e5419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 -- 172.21.15.52:0/3456372376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e541092d0 msgr2=0x7f0e5419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e541092d0 0x7f0e5419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 -- 172.21.15.52:0/3456372376 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5419faa0 con 0x7f0e5419efb0 2024-04-26T01:48:57.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e52a00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 0x7f0e5419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53400640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009430 tx=0x7f0e4c0317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e4b400640 1 -- 172.21.15.52:0/3456372376 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c003b70 con 0x7f0e5419efb0 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e4b400640 1 -- 172.21.15.52:0/3456372376 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c003d10 con 0x7f0e5419efb0 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5419fd30 con 0x7f0e5419efb0 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.361+0000 7f0e4b400640 1 -- 172.21.15.52:0/3456372376 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c02f990 con 0x7f0e5419efb0 2024-04-26T01:48:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.362+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e541a7550 con 0x7f0e5419efb0 2024-04-26T01:48:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.365+0000 7f0e4b400640 1 -- 172.21.15.52:0/3456372376 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0e4c02fb30 con 0x7f0e5419efb0 2024-04-26T01:48:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.365+0000 7f0e5cbed740 1 Processor -- start 2024-04-26T01:48:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 -- start start 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff320fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff321cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ff3175900 con 0x556ff32102d0 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556ff311bef0 con 0x556ff320f9b0 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.366+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556ff31718c0 con 0x556ff321ab90 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff320fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48280/0 (socket says 172.21.15.52:48280) 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 -- 172.21.15.52:0/3399103094 learned_addr learned my addr 172.21.15.52:0/3399103094 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52000640 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff321cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 -- 172.21.15.52:0/3399103094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 msgr2=0x556ff321cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff321cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 -- 172.21.15.52:0/3399103094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 msgr2=0x556ff320fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff320fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e52a00640 1 -- 172.21.15.52:0/3399103094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ff2e88210 con 0x556ff32102d0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.367+0000 7f0e4b400640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 0x7f0e380a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e52a00640 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400f4c0 tx=0x7f0e4400f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cdbdbb1790d42d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e4b400640 1 -- 172.21.15.52:0/3456372376 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0e4c0f1b40 con 0x7f0e5419efb0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e52000640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 0x7f0e380a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e3fe00640 1 -- 172.21.15.52:0/3399103094 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44005220 con 0x556ff32102d0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e3fe00640 1 -- 172.21.15.52:0/3399103094 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e440047a0 con 0x556ff32102d0 2024-04-26T01:48:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.368+0000 7f0e3fe00640 1 -- 172.21.15.52:0/3399103094 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44010a80 con 0x556ff32102d0 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3399103094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 msgr2=0x556ff321a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e52000640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 0x7f0e380a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x556ff320d2e0 tx=0x7f0e40004730 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400f4c0 tx=0x7f0e4400f980 comp rx=0 tx=0).stop 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3399103094 shutdown_connections 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff321cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff321a650 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3399103094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff320fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3399103094 >> 172.21.15.52:0/3399103094 conn(0x556ff320bec0 msgr2=0x556ff320c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3399103094 shutdown_connections 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3399103094 wait complete. 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.369+0000 7f0e5cbed740 1 -- start start 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 Processor -- start 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3217a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32184b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ff321d7c0 con 0x556ff32102d0 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556ff312f310 con 0x556ff320f9b0 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556ff321d940 con 0x556ff321ab90 2024-04-26T01:48:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e5cbed740 10 client.0 _ll_register_callbacks cb 0x556ff30fdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e52000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e52000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48282/0 (socket says 172.21.15.52:48282) 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e52000640 1 -- 172.21.15.52:0/4160813575 learned_addr learned my addr 172.21.15.52:0/4160813575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.370+0000 7f0e52a00640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3217a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e53400640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 -- 172.21.15.52:0/4160813575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 msgr2=0x556ff32184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 -- 172.21.15.52:0/4160813575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 msgr2=0x556ff3217f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 -- 172.21.15.52:0/4160813575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ff2f54620 con 0x556ff320f9b0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52000640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e52a00640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3217a30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e4401a040 tx=0x7f0e44010af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a938ab486a0a573a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.371+0000 7f0e3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:48:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44026070 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0e44005100 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e440047a0 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0e44032050 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e5cbed740 10 client.25610 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.372+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556ff3229980 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e440047a0 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f0e44018640 con 0x556ff320f9b0 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 10 client.25610 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 10 client.25610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 20 client.25610 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff322ad90 0x556ff322d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 4 client.25610 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:48:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ff31898b0 con 0x556ff322ad90 2024-04-26T01:48:57.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.373+0000 7f0e52000640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff322ad90 0x556ff322d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.374+0000 7f0e52000640 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff322ad90 0x556ff322d180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d56bbdf9d36b0f8 server_cookie=cc5fba3b0ffb682e in_seq=0 out_seq=0 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.374+0000 7f0e3cc00640 10 client.25610 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e3cc00640 1 -- 172.21.15.52:0/4160813575 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ff31898b0 con 0x556ff322ad90 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e3cc00640 10 client.25610 handle_command_reply: tid=0 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 client.25610 shutdown 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 msgr2=0x556ff3217a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3217a30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e4401a040 tx=0x7f0e44010af0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 shutdown_connections 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff322ad90 0x556ff322d180 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff3217f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/4160813575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3217a30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e3cc00640 0 client.25610 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 >> 172.21.15.52:0/4160813575 conn(0x556ff320bec0 msgr2=0x556ff3211c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 shutdown_connections 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 1 -- 172.21.15.52:0/4160813575 wait complete. 2024-04-26T01:48:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.375+0000 7f0e5cbed740 20 client.25610 trim_cache size 0 max 16384 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 Processor -- start 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 -- start start 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3226680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff320aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541b2430 con 0x556ff32102d0 2024-04-26T01:48:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541afc70 con 0x556ff320f9b0 2024-04-26T01:48:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.392+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541adae0 con 0x556ff321ab90 2024-04-26T01:48:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e52000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff320aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55926/0 (socket says 172.21.15.52:55926) 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 -- 172.21.15.52:0/3526395351 learned_addr learned my addr 172.21.15.52:0/3526395351 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e52a00640 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3226680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 -- 172.21.15.52:0/3526395351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 msgr2=0x556ff3226680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3226680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 -- 172.21.15.52:0/3526395351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 msgr2=0x556ff320aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff320aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 -- 172.21.15.52:0/3526395351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ff321d6a0 con 0x556ff321ab90 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e52000640 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff320aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.393+0000 7f0e53400640 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0ba020 tx=0x7f0e4c0b9e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa9461b25ce6685 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e3f400640 1 -- 172.21.15.52:0/3526395351 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0bdbc0 con 0x556ff321ab90 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e3f400640 1 -- 172.21.15.52:0/3526395351 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c036070 con 0x556ff321ab90 2024-04-26T01:48:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e3f400640 1 -- 172.21.15.52:0/3526395351 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0de0c0 con 0x556ff321ab90 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3526395351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 msgr2=0x556ff32328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0ba020 tx=0x7f0e4c0b9e60 comp rx=0 tx=0).stop 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3526395351 shutdown_connections 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff321ab90 0x556ff32328c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff32102d0 0x556ff320aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/3526395351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff320f9b0 0x556ff3226680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3526395351 >> 172.21.15.52:0/3526395351 conn(0x556ff320bec0 msgr2=0x556ff320a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3526395351 shutdown_connections 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.394+0000 7f0e5cbed740 1 -- 172.21.15.52:0/3526395351 wait complete. 2024-04-26T01:48:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.395+0000 7f0e5cbed740 1 -- start start 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.395+0000 7f0e5cbed740 1 Processor -- start 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 0x556ff321fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff321ab90 0x556ff328f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541ae130 con 0x556ff320f9b0 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541af5a0 con 0x556ff321ab90 2024-04-26T01:48:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541ae800 con 0x556ff32102d0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e5cbed740 10 client.0 _ll_register_callbacks cb 0x7f0e4400af10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 0x556ff321fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55942/0 (socket says 172.21.15.52:55942) 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 -- 172.21.15.52:0/2817838754 learned_addr learned my addr 172.21.15.52:0/2817838754 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 0x556ff321fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48296/0 (socket says 172.21.15.52:48296) 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e53400640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff321ab90 0x556ff328f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 -- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff321ab90 msgr2=0x556ff328f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff321ab90 0x556ff328f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 -- 172.21.15.52:0/2817838754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 msgr2=0x556ff321fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 0x556ff321fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.396+0000 7f0e52000640 1 -- 172.21.15.52:0/2817838754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ff328f670 con 0x556ff32102d0 2024-04-26T01:48:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e52000640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x556ff322d830 tx=0x7f0e40005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4259fe883b4475f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e400109c0 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e4c036070 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0e4001a070 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0e4003f050 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e5cbed740 10 client.25624 fetch_fsmap learned FSMap version 30 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.397+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556ff3290990 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4c036070 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f0e4001d070 con 0x556ff32102d0 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 10 client.25624 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 10 client.25624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 20 client.25624 populate_metadata read hostname 'smithi052' 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff3291150 0x556ff3293570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 4 client.25624 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:48:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.398+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ff3183320 con 0x556ff3291150 2024-04-26T01:48:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.399+0000 7f0e52a00640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff3291150 0x556ff3293570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:48:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.399+0000 7f0e52a00640 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff3291150 0x556ff3293570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b718862479c943f server_cookie=8c74f0c3f1044046 in_seq=0 out_seq=0 2024-04-26T01:48:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.399+0000 7f0e3cc00640 10 client.25624 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.415+0000 7f0e3cc00640 1 -- 172.21.15.52:0/2817838754 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ff3183320 con 0x556ff3291150 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.415+0000 7f0e3cc00640 10 client.25624 handle_command_reply: tid=0 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.415+0000 7f0e5cbed740 1 client.25624 shutdown 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.415+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 msgr2=0x556ff3220290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.415+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x556ff322d830 tx=0x7f0e40005f90 comp rx=0 tx=0).stop 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 shutdown_connections 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ff3291150 0x556ff3293570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ff321ab90 0x556ff328f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ff32102d0 0x556ff3220290 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 --2- 172.21.15.52:0/2817838754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ff320f9b0 0x556ff321fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 >> 172.21.15.52:0/2817838754 conn(0x556ff320bec0 msgr2=0x556ff320b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e3cc00640 0 client.25624 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 shutdown_connections 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 1 -- 172.21.15.52:0/2817838754 wait complete. 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.416+0000 7f0e5cbed740 20 client.25624 trim_cache size 0 max 16384 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:48:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:48:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 msgr2=0x7f0e380a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 0x7f0e380a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x556ff320d2e0 tx=0x7f0e40004730 comp rx=0 tx=0).stop 2024-04-26T01:48:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 msgr2=0x7f0e5419f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:48:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009430 tx=0x7f0e4c0317c0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 shutdown_connections 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e380a3d10 0x7f0e380a61d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e5419efb0 0x7f0e5419f460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e541092d0 0x7f0e5419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 --2- 172.21.15.52:0/3456372376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e54104200 0x7f0e5419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 >> 172.21.15.52:0/3456372376 conn(0x7f0e540ffa60 msgr2=0x7f0e54101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 shutdown_connections 2024-04-26T01:48:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:48:57.417+0000 7f0e53e00640 1 -- 172.21.15.52:0/3456372376 wait complete. 2024-04-26T01:48:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:57 smithi191 ceph-mon[32745]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-26T01:48:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:57 smithi183 ceph-mon[32752]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-26T01:48:57.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:57 smithi052 ceph-mon[27851]: pgmap v121: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-26T01:48:58.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 17 sec latency 27022.739 ms 2024-04-26T01:48:59.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 18 sec latency 28022.877 ms 2024-04-26T01:48:59.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:48:59 smithi052 ceph-mon[27851]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:48:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:48:59 smithi191 ceph-mon[32745]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:48:59.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:48:59 smithi183 ceph-mon[32752]: pgmap v122: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:49:00.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 19 sec latency 29023.012 ms 2024-04-26T01:49:01.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 20 sec latency 30023.199 ms 2024-04-26T01:49:01.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:01 smithi191 ceph-mon[32745]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:49:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:01 smithi183 ceph-mon[32752]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:49:01.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:01 smithi052 ceph-mon[27851]: pgmap v123: 97 pgs: 97 active+clean; 40 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T01:49:02.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:49:02.074 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 21 sec latency 31023.284 ms 2024-04-26T01:49:02.522 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:03.075 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 22 sec latency 32023.352 ms 2024-04-26T01:49:03.583 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:03 smithi052 ceph-mon[27851]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:03 smithi191 ceph-mon[32745]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:03.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:03 smithi183 ceph-mon[32752]: pgmap v124: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:04.075 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 23 sec latency 33023.469 ms 2024-04-26T01:49:04.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 -- 172.21.15.52:0/1217957194 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06a750 msgr2=0x7f4eac06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/1217957194 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06a750 0x7f4eac06abb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009990 tx=0x7f4e98009ca0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 -- 172.21.15.52:0/1217957194 shutdown_connections 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/1217957194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06b0f0 0x7f4eac10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/1217957194 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 0x7f4eac108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/1217957194 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06a750 0x7f4eac06abb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 -- 172.21.15.52:0/1217957194 >> 172.21.15.52:0/1217957194 conn(0x7f4eac0788f0 msgr2=0x7f4eac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 -- 172.21.15.52:0/1217957194 shutdown_connections 2024-04-26T01:49:04.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.851+0000 7f4eb2e00700 1 -- 172.21.15.52:0/1217957194 wait complete. 2024-04-26T01:49:04.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.852+0000 7f4eb2e00700 1 Processor -- start 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 -- start start 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06b0f0 0x7f4eac19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 0x7f4eac19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac112c60 con 0x7f4eac06b0f0 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac112ae0 con 0x7f4eac108a30 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eb2e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac112de0 con 0x7f4eac06a750 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eabe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eab400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06b0f0 0x7f4eac19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eabe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:48762/0 (socket says 172.21.15.52:48762) 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.853+0000 7f4eabe00700 1 -- 172.21.15.52:0/3132097731 learned_addr learned my addr 172.21.15.52:0/3132097731 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eb0e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 0x7f4eac19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 -- 172.21.15.52:0/3132097731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 msgr2=0x7f4eac19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:04.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 0x7f4eac19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 -- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06b0f0 msgr2=0x7f4eac19b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06b0f0 0x7f4eac19b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 -- 172.21.15.52:0/3132097731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac1a0330 con 0x7f4eac06a750 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4eabe00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9a0 tx=0x7f4e9c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.854+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00ca10 con 0x7f4eac06a750 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.855+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac1a0620 con 0x7f4eac06a750 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.855+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e9c00cbb0 con 0x7f4eac06a750 2024-04-26T01:49:04.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.855+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c018680 con 0x7f4eac06a750 2024-04-26T01:49:04.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.855+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac1a7ea0 con 0x7f4eac06a750 2024-04-26T01:49:04.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.856+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eac069730 con 0x7f4eac06a750 2024-04-26T01:49:04.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.858+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4e9c010430 con 0x7f4eac06a750 2024-04-26T01:49:04.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.859+0000 7f4ea8c00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 0x7f4e900a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:04.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.860+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f4e9c0d09f0 con 0x7f4eac06a750 2024-04-26T01:49:04.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.860+0000 7f4eab400700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 0x7f4e900a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:04.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.860+0000 7f4eab400700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 0x7f4e900a6270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4eac078ef0 tx=0x7f4e9800b260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:04.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:04.862+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e9c09d060 con 0x7f4eac06a750 2024-04-26T01:49:05.075 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 24 sec latency 34023.595 ms 2024-04-26T01:49:05.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.146+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4eac02df80 con 0x7f4eac06a750 2024-04-26T01:49:05.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.147+0000 7f4ea8c00700 1 -- 172.21.15.52:0/3132097731 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11889 (secure 0 0 0) 0x7f4e9c09c960 con 0x7f4eac06a750 2024-04-26T01:49:05.146 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:49:05.146 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:47:28.026495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1263803776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1263803776},{"type":"v1","addr":"172.21.15.183:6837","nonce":1263803776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:49:05.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.150+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 msgr2=0x7f4e900a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:05.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.150+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 0x7f4e900a6270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4eac078ef0 tx=0x7f4e9800b260 comp rx=0 tx=0).stop 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.150+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 msgr2=0x7f4eac19b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.150+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9a0 tx=0x7f4e9c00ecb0 comp rx=0 tx=0).stop 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 shutdown_connections 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e900a3db0 0x7f4e900a6270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4eac108a30 0x7f4eac19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4eac06a750 0x7f4eac19b370 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:05.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 --2- 172.21.15.52:0/3132097731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4eac06b0f0 0x7f4eac19b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:05.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 >> 172.21.15.52:0/3132097731 conn(0x7f4eac0788f0 msgr2=0x7f4eac10df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:05.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 shutdown_connections 2024-04-26T01:49:05.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:05.151+0000 7f4eb2e00700 1 -- 172.21.15.52:0/3132097731 wait complete. 2024-04-26T01:49:05.150 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-04-26T01:49:05.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:05 smithi052 ceph-mon[27851]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 23 op/s 2024-04-26T01:49:05.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:05 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3132097731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:05 smithi191 ceph-mon[32745]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 23 op/s 2024-04-26T01:49:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:05 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3132097731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:05.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:05 smithi183 ceph-mon[32752]: pgmap v125: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 31 KiB/s wr, 23 op/s 2024-04-26T01:49:05.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:05 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3132097731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:49:06.075 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 25 sec latency 35023.688 ms 2024-04-26T01:49:06.082 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 6a2a23e6 --release --await' 2024-04-26T01:49:06.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 6a2a23e6 --release --await 2024-04-26T01:49:06.571 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:49:07.075 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 26 sec latency 36024.182 ms 2024-04-26T01:49:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:07 smithi052 ceph-mon[27851]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:07 smithi191 ceph-mon[32745]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:07 smithi183 ceph-mon[32752]: pgmap v126: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 31 KiB/s wr, 26 op/s 2024-04-26T01:49:08.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53783 0.00 MB/sec execute 27 sec latency 37024.264 ms 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 -- 172.21.15.52:0/1842719504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006b120 msgr2=0x7fa11006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 --2- 172.21.15.52:0/1842719504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006b120 0x7fa11006b500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa100009990 tx=0x7fa100009ca0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 -- 172.21.15.52:0/1842719504 shutdown_connections 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 --2- 172.21.15.52:0/1842719504 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 0x7fa1101118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 --2- 172.21.15.52:0/1842719504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006b120 0x7fa11006b500 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 --2- 172.21.15.52:0/1842719504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006ba40 0x7fa11010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.873+0000 7fa115e00700 1 -- 172.21.15.52:0/1842719504 >> 172.21.15.52:0/1842719504 conn(0x7fa1100788b0 msgr2=0x7fa110078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.874+0000 7fa115e00700 1 -- 172.21.15.52:0/1842719504 shutdown_connections 2024-04-26T01:49:08.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.874+0000 7fa115e00700 1 -- 172.21.15.52:0/1842719504 wait complete. 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 Processor -- start 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 -- start start 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006b120 0x7fa11019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 0x7fa11019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa110114e50 con 0x7fa11006ba40 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa110114cd0 con 0x7fa11010f4f0 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.875+0000 7fa115e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa110114fd0 con 0x7fa11006b120 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58582/0 (socket says 172.21.15.52:58582) 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 -- 172.21.15.52:0/527753918 learned_addr learned my addr 172.21.15.52:0/527753918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa10f400700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 0x7fa11019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa10ea00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006b120 0x7fa11019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 -- 172.21.15.52:0/527753918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006b120 msgr2=0x7fa11019b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006b120 0x7fa11019b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 -- 172.21.15.52:0/527753918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 msgr2=0x7fa11019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 0x7fa11019fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.876+0000 7fa106000700 1 -- 172.21.15.52:0/527753918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1101a03f0 con 0x7fa11006ba40 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa106000700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800ef10 tx=0x7fa0f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa114a00700 1 -- 172.21.15.52:0/527753918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800c9c0 con 0x7fa11006ba40 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa114a00700 1 -- 172.21.15.52:0/527753918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f8019070 con 0x7fa11006ba40 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa114a00700 1 -- 172.21.15.52:0/527753918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8014480 con 0x7fa11006ba40 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1101a0680 con 0x7fa11006ba40 2024-04-26T01:49:08.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.877+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1101a7f60 con 0x7fa11006ba40 2024-04-26T01:49:08.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.881+0000 7fa114a00700 1 -- 172.21.15.52:0/527753918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa0f8012050 con 0x7fa11006ba40 2024-04-26T01:49:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.882+0000 7fa120715b80 1 Processor -- start 2024-04-26T01:49:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.882+0000 7fa114a00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 0x7fa0f40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.882+0000 7fa114a00700 1 -- 172.21.15.52:0/527753918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa0f80d06d0 con 0x7fa11006ba40 2024-04-26T01:49:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.882+0000 7fa120715b80 1 -- start start 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.882+0000 7fa10f400700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 0x7fa0f40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6b30f0 0x56264f6b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56264f5faf90 con 0x56264f6a8240 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56264f5a1830 con 0x56264f6b30f0 2024-04-26T01:49:08.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa120715b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56264f5f6f50 con 0x56264f6a7920 2024-04-26T01:49:08.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58598/0 (socket says 172.21.15.52:58598) 2024-04-26T01:49:08.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa106000700 1 -- 172.21.15.52:0/4188586623 learned_addr learned my addr 172.21.15.52:0/4188586623 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:08.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa10ea00700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa10f400700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6b30f0 0x56264f6b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.883+0000 7fa10f400700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 0x7fa0f40a62f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa11006acf0 tx=0x7fa108009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 -- 172.21.15.52:0/4188586623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 msgr2=0x56264f6a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6a7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 -- 172.21.15.52:0/4188586623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6b30f0 msgr2=0x56264f6b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6b30f0 0x56264f6b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 -- 172.21.15.52:0/4188586623 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264f597780 con 0x56264f6a8240 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa10ea00700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa106000700 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8098aa0 tx=0x7fa0f809c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37d267b6ce21e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa104c00700 1 -- 172.21.15.52:0/4188586623 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f809c640 con 0x56264f6a8240 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa104c00700 1 -- 172.21.15.52:0/4188586623 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f80a09f0 con 0x56264f6a8240 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.884+0000 7fa104c00700 1 -- 172.21.15.52:0/4188586623 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80a4e20 con 0x56264f6a8240 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 -- 172.21.15.52:0/4188586623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 msgr2=0x56264f6b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8098aa0 tx=0x7fa0f809c180 comp rx=0 tx=0).stop 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 -- 172.21.15.52:0/4188586623 shutdown_connections 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6b30f0 0x56264f6b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 --2- 172.21.15.52:0/4188586623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b2bb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 -- 172.21.15.52:0/4188586623 >> 172.21.15.52:0/4188586623 conn(0x56264f5d66a0 msgr2=0x56264f6a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 -- 172.21.15.52:0/4188586623 shutdown_connections 2024-04-26T01:49:08.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.885+0000 7fa120715b80 1 -- 172.21.15.52:0/4188586623 wait complete. 2024-04-26T01:49:08.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.886+0000 7fa120715b80 1 -- start start 2024-04-26T01:49:08.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.886+0000 7fa120715b80 1 Processor -- start 2024-04-26T01:49:08.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56264f6b5bb0 con 0x56264f6b30f0 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56264f5b4960 con 0x56264f6a8240 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56264f6b5d30 con 0x56264f6a7920 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa120715b80 10 client.0 _ll_register_callbacks cb 0x56264f593820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa10ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa106000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa10f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa106000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55876/0 (socket says 172.21.15.52:55876) 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.887+0000 7fa106000700 1 -- 172.21.15.52:0/3225654939 learned_addr learned my addr 172.21.15.52:0/3225654939 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 -- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 msgr2=0x56264f6afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 -- 172.21.15.52:0/3225654939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 msgr2=0x56264f6b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 -- 172.21.15.52:0/3225654939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264f5489d0 con 0x56264f6a7920 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa106000700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10f400700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa10ea00700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6b28d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1000098e0 tx=0x7fa100005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ea6fa334e6b510 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.888+0000 7fa0fd600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa100028060 con 0x56264f6a7920 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa10000bd80 con 0x56264f6a7920 2024-04-26T01:49:08.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f80a09f0 con 0x56264f6a7920 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa1000058e0 con 0x56264f6a7920 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa120715b80 10 client.25642 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56264f591bf0 con 0x56264f6a7920 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.889+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f80a09f0 con 0x56264f6a7920 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa10000f620 con 0x56264f6a7920 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 10 client.25642 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 10 client.25642 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 20 client.25642 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f6c2960 0x56264f6c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 4 client.25642 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:08.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56264f608780 con 0x56264f6c2960 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.890+0000 7fa106000700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f6c2960 0x56264f6c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.891+0000 7fa106000700 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f6c2960 0x56264f6c4d70 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f4355efde7305df server_cookie=39a3e48c075472e4 in_seq=0 out_seq=0 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.891+0000 7fa0fd600700 10 client.25642 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa0fd600700 1 -- 172.21.15.52:0/3225654939 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56264f608780 con 0x56264f6c2960 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa0fd600700 10 client.25642 handle_command_reply: tid=0 2024-04-26T01:49:08.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 client.25642 shutdown 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 msgr2=0x56264f6b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6b28d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1000098e0 tx=0x7fa100005750 comp rx=0 tx=0).stop 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 shutdown_connections 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f6c2960 0x56264f6c4d70 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6b28d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa0fd600700 0 client.25642 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 --2- 172.21.15.52:0/3225654939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.892+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 >> 172.21.15.52:0/3225654939 conn(0x56264f5d66a0 msgr2=0x56264f6a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.893+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 shutdown_connections 2024-04-26T01:49:08.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.893+0000 7fa120715b80 1 -- 172.21.15.52:0/3225654939 wait complete. 2024-04-26T01:49:08.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.893+0000 7fa120715b80 20 client.25642 trim_cache size 0 max 16384 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.914+0000 7fa120715b80 1 Processor -- start 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 -- start start 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 0x56264f6ab100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 0x56264f6beae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101b1ed0 con 0x56264f6a8240 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101af700 con 0x56264f6a7920 2024-04-26T01:49:08.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa120715b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101ad550 con 0x56264f6b30f0 2024-04-26T01:49:08.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa10f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 0x56264f6beae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa106000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58618/0 (socket says 172.21.15.52:58618) 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.915+0000 7fa106000700 1 -- 172.21.15.52:0/919665625 learned_addr learned my addr 172.21.15.52:0/919665625 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10ea00700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 0x56264f6ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 -- 172.21.15.52:0/919665625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 msgr2=0x56264f6ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 0x56264f6ab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 -- 172.21.15.52:0/919665625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 msgr2=0x56264f6b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 -- 172.21.15.52:0/919665625 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264f6a7750 con 0x56264f6b30f0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10ea00700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 0x56264f6ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa106000700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.916+0000 7fa10f400700 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 0x56264f6beae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1080117d0 tx=0x7fa108014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45028a693f910abc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa0ffe00700 1 -- 172.21.15.52:0/919665625 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108003680 con 0x56264f6b30f0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa0ffe00700 1 -- 172.21.15.52:0/919665625 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10802b070 con 0x56264f6b30f0 2024-04-26T01:49:08.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa0ffe00700 1 -- 172.21.15.52:0/919665625 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1080126e0 con 0x56264f6b30f0 2024-04-26T01:49:08.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 -- 172.21.15.52:0/919665625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 msgr2=0x56264f6beae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 0x56264f6beae0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1080117d0 tx=0x7fa108014000 comp rx=0 tx=0).stop 2024-04-26T01:49:08.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 -- 172.21.15.52:0/919665625 shutdown_connections 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6b30f0 0x56264f6beae0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a7920 0x56264f6ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 --2- 172.21.15.52:0/919665625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6a8240 0x56264f6b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 -- 172.21.15.52:0/919665625 >> 172.21.15.52:0/919665625 conn(0x56264f5d66a0 msgr2=0x56264f6aa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 -- 172.21.15.52:0/919665625 shutdown_connections 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.917+0000 7fa120715b80 1 -- 172.21.15.52:0/919665625 wait complete. 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.918+0000 7fa120715b80 1 -- start start 2024-04-26T01:49:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.918+0000 7fa120715b80 1 Processor -- start 2024-04-26T01:49:08.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101adbc0 con 0x56264f6b30f0 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101af030 con 0x56264f6a8240 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1101ae290 con 0x56264f6a7920 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa120715b80 10 client.0 _ll_register_callbacks cb 0x56264f593820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa10f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa10ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa10ea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:48810/0 (socket says 172.21.15.52:48810) 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa10f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58620/0 (socket says 172.21.15.52:58620) 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa10ea00700 1 -- 172.21.15.52:0/2291262881 learned_addr learned my addr 172.21.15.52:0/2291262881 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.919+0000 7fa106000700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 -- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 msgr2=0x56264f6b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 -- 172.21.15.52:0/2291262881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 msgr2=0x56264f6b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 -- 172.21.15.52:0/2291262881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264f6b8d40 con 0x56264f6a7920 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10f400700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa10ea00700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa100005f30 tx=0x7fa100010f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=378e593166cad918 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa0fd600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10000bc00 con 0x56264f6a7920 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa10802b070 con 0x56264f6a7920 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.920+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7fa10000bda0 con 0x56264f6a7920 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa100005bc0 con 0x56264f6a7920 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 10 client.25648 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56264f729f10 con 0x56264f6a7920 2024-04-26T01:49:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10802b070 con 0x56264f6a7920 2024-04-26T01:49:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa10002d780 con 0x56264f6a7920 2024-04-26T01:49:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 10 client.25648 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 10 client.25648 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:49:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 20 client.25648 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f72a500 0x56264f72c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 4 client.25648 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6a2a23e6", "release": true, "await": true}] 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.921+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6a2a23e6", "release": true, "await": true}) -- 0x56264f602820 con 0x56264f72a500 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.922+0000 7fa106000700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f72a500 0x56264f72c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.922+0000 7fa106000700 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f72a500 0x56264f72c920 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fdb1fb2d6912baa server_cookie=25d4191478256787 in_seq=0 out_seq=0 2024-04-26T01:49:08.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.922+0000 7fa0fd600700 10 client.25648 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.927+0000 7fa0fd600700 1 -- 172.21.15.52:0/2291262881 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56264f602820 con 0x56264f72a500 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.927+0000 7fa0fd600700 10 client.25648 handle_command_reply: tid=0 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.927+0000 7fa120715b80 1 client.25648 shutdown 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 msgr2=0x56264f6bb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa100005f30 tx=0x7fa100010f60 comp rx=0 tx=0).stop 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 shutdown_connections 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56264f72a500 0x56264f72c920 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56264f6b30f0 0x56264f6b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56264f6a7920 0x56264f6bb3b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa0fd600700 0 client.25648 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 --2- 172.21.15.52:0/2291262881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56264f6a8240 0x56264f6b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 >> 172.21.15.52:0/2291262881 conn(0x56264f5d66a0 msgr2=0x56264f6b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 shutdown_connections 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 1 -- 172.21.15.52:0/2291262881 wait complete. 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.928+0000 7fa120715b80 20 client.25648 trim_cache size 0 max 16384 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:49:08.927 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 31, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 12, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "6a2a23e6": { 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 12, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 90.0, 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:49:08.928 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:49:08.929 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 msgr2=0x7fa0f40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 0x7fa0f40a62f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa11006acf0 tx=0x7fa108009260 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 msgr2=0x7fa11019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800ef10 tx=0x7fa0f800c3a0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 shutdown_connections 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa0f40a3e30 0x7fa0f40a62f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa11010f4f0 0x7fa11019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11006b120 0x7fa11019b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 --2- 172.21.15.52:0/527753918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa11006ba40 0x7fa11019b920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.929+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 >> 172.21.15.52:0/527753918 conn(0x7fa1100788b0 msgr2=0x7fa110110a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.930+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 shutdown_connections 2024-04-26T01:49:08.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:49:08.930+0000 7fa115e00700 1 -- 172.21.15.52:0/527753918 wait complete. 2024-04-26T01:49:09.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 53901 0.00 MB/sec execute 28 sec latency 37879.386 ms 2024-04-26T01:49:09.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:09 smithi052 ceph-mon[27851]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:09.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:09 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:09 smithi191 ceph-mon[32745]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:09 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:09 smithi183 ceph-mon[32752]: pgmap v127: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:09 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:10.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 54813 0.18 MB/sec execute 29 sec latency 23.224 ms 2024-04-26T01:49:10.084 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6a2a23e6', seconds elapsed: 54.4 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 -- 172.21.15.52:0/3527154868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 msgr2=0x7f0eac10dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 --2- 172.21.15.52:0/3527154868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac10dc20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009860 tx=0x7f0ea402a340 comp rx=0 tx=0).stop 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 -- 172.21.15.52:0/3527154868 shutdown_connections 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 --2- 172.21.15.52:0/3527154868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac10dc20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 --2- 172.21.15.52:0/3527154868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 0x7f0eac101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 --2- 172.21.15.52:0/3527154868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 0x7f0eac1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 -- 172.21.15.52:0/3527154868 >> 172.21.15.52:0/3527154868 conn(0x7f0eac076b80 msgr2=0x7f0eac076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.574+0000 7f0eb1600640 1 -- 172.21.15.52:0/3527154868 shutdown_connections 2024-04-26T01:49:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.575+0000 7f0eb1600640 1 -- 172.21.15.52:0/3527154868 wait complete. 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 Processor -- start 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 -- start start 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 0x7f0eac1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 0x7f0eac19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1110d0 con 0x7f0eac1011a0 2024-04-26T01:49:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac110f50 con 0x7f0eac106fe0 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.576+0000 7f0eb1600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac111250 con 0x7f0eac101b40 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 0x7f0eac1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0eab400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 0x7f0eac19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:48830/0 (socket says 172.21.15.52:48830) 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 -- 172.21.15.52:0/1757018382 learned_addr learned my addr 172.21.15.52:0/1757018382 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 -- 172.21.15.52:0/1757018382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 msgr2=0x7f0eac19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 0x7f0eac19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 -- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 msgr2=0x7f0eac1a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 0x7f0eac1a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 -- 172.21.15.52:0/1757018382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac19aef0 con 0x7f0eac101b40 2024-04-26T01:49:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.577+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9f0 tx=0x7f0e9400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.578+0000 7f0ea3e00640 1 -- 172.21.15.52:0/1757018382 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400cd80 con 0x7f0eac101b40 2024-04-26T01:49:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.578+0000 7f0ea3e00640 1 -- 172.21.15.52:0/1757018382 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e940108e0 con 0x7f0eac101b40 2024-04-26T01:49:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.578+0000 7f0ea3e00640 1 -- 172.21.15.52:0/1757018382 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94018980 con 0x7f0eac101b40 2024-04-26T01:49:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.578+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac19b1e0 con 0x7f0eac101b40 2024-04-26T01:49:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.578+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1a74f0 con 0x7f0eac101b40 2024-04-26T01:49:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0ea3e00640 1 -- 172.21.15.52:0/1757018382 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0e94010430 con 0x7f0eac101b40 2024-04-26T01:49:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 Processor -- start 2024-04-26T01:49:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 -- start start 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da74c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da81df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55de0d9da900 con 0x55de0da74820 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55de0d980ef0 con 0x55de0da75140 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.582+0000 7f0eb439e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55de0d9d68c0 con 0x55de0da7fa00 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55912/0 (socket says 172.21.15.52:55912) 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eab400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da74c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 -- 172.21.15.52:0/1510734731 learned_addr learned my addr 172.21.15.52:0/1510734731 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da81df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 -- 172.21.15.52:0/1510734731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 msgr2=0x55de0da81df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da81df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 -- 172.21.15.52:0/1510734731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 msgr2=0x55de0da74c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da74c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0eaaa00640 1 -- 172.21.15.52:0/1510734731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0d6ed210 con 0x55de0da75140 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.583+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da81df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00e5d0 tx=0x7f0e9c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d7db08af7f2de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0ea3e00640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 0x7f0e900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0e9b400640 1 -- 172.21.15.52:0/1510734731 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00dad0 con 0x55de0da75140 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0e9b400640 1 -- 172.21.15.52:0/1510734731 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e9c01c070 con 0x55de0da75140 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0ea3e00640 1 -- 172.21.15.52:0/1757018382 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0e940d0a40 con 0x7f0eac101b40 2024-04-26T01:49:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0e9b400640 1 -- 172.21.15.52:0/1510734731 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c016380 con 0x55de0da75140 2024-04-26T01:49:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.584+0000 7f0eab400640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 0x7f0e900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 -- 172.21.15.52:0/1510734731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 msgr2=0x55de0da7f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00e5d0 tx=0x7f0e9c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 -- 172.21.15.52:0/1510734731 shutdown_connections 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da81df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7f4c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1510734731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da74c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 -- 172.21.15.52:0/1510734731 >> 172.21.15.52:0/1510734731 conn(0x55de0da70d30 msgr2=0x55de0da71110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eab400640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 0x7f0e900a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009830 tx=0x7f0ea402c040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 -- 172.21.15.52:0/1510734731 shutdown_connections 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.585+0000 7f0eb439e740 1 -- 172.21.15.52:0/1510734731 wait complete. 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 -- start start 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 Processor -- start 2024-04-26T01:49:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 0x55de0da7c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da7fa00 0x55de0da7d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55de0da82630 con 0x55de0da7fa00 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55de0d994310 con 0x55de0da75140 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.586+0000 7f0eb439e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55de0da827b0 con 0x55de0da74820 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0eb439e740 10 client.0 _ll_register_callbacks cb 0x55de0d962cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 0x55de0da7c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0eab400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da7fa00 0x55de0da7d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55922/0 (socket says 172.21.15.52:55922) 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 -- 172.21.15.52:0/4082115552 learned_addr learned my addr 172.21.15.52:0/4082115552 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 0x55de0da7c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:48850/0 (socket says 172.21.15.52:48850) 2024-04-26T01:49:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 -- 172.21.15.52:0/4082115552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 msgr2=0x55de0da7c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 0x55de0da7c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 -- 172.21.15.52:0/4082115552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da7fa00 msgr2=0x55de0da7d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da7fa00 0x55de0da7d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 -- 172.21.15.52:0/4082115552 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0d7b9620 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0ea2000640 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9c0 tx=0x7f0e940d4370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d57cd84498e50ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.587+0000 7f0e8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94010720 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0e940a51f0 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e9c01c070 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0e9409cac0 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0eb439e740 10 client.25652 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de0da8e8e0 con 0x55de0da75140 2024-04-26T01:49:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e9c01c070 con 0x55de0da75140 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0e94018b20 con 0x55de0da75140 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0eb439e740 10 client.25652 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.588+0000 7f0eb439e740 10 client.25652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eb439e740 20 client.25652 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0da8fdf0 0x55de0da921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eb439e740 4 client.25652 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de0d9ee8b0 con 0x55de0da8fdf0 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0da8fdf0 0x55de0da921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0da8fdf0 0x55de0da921e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3016eb3fa13e76e server_cookie=6ae7fa289c12bf1c in_seq=0 out_seq=0 2024-04-26T01:49:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.589+0000 7f0e8fe00640 10 client.25652 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.590+0000 7f0e8fe00640 1 -- 172.21.15.52:0/4082115552 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de0d9ee8b0 con 0x55de0da8fdf0 2024-04-26T01:49:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.590+0000 7f0e8fe00640 10 client.25652 handle_command_reply: tid=0 2024-04-26T01:49:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.590+0000 7f0eb439e740 1 client.25652 shutdown 2024-04-26T01:49:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.590+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 msgr2=0x55de0da7cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.590+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9c0 tx=0x7f0e940d4370 comp rx=0 tx=0).stop 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 shutdown_connections 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0da8fdf0 0x55de0da921e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da7fa00 0x55de0da7d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da7cf10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 --2- 172.21.15.52:0/4082115552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da74820 0x55de0da7c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0e8fe00640 0 client.25652 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 >> 172.21.15.52:0/4082115552 conn(0x55de0da70d30 msgr2=0x55de0da76ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 shutdown_connections 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 1 -- 172.21.15.52:0/4082115552 wait complete. 2024-04-26T01:49:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.591+0000 7f0eb439e740 20 client.25652 trim_cache size 0 max 16384 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 Processor -- start 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 -- start start 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da75140 0x55de0da7d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 0x55de0da97ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1b2550 con 0x55de0da74820 2024-04-26T01:49:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1afdc0 con 0x55de0da7fa00 2024-04-26T01:49:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.612+0000 7f0eb439e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1adc10 con 0x55de0da75140 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eab400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 0x55de0da97ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58652/0 (socket says 172.21.15.52:58652) 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eab400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 0x55de0da97ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55936/0 (socket says 172.21.15.52:55936) 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 -- 172.21.15.52:0/2899667053 learned_addr learned my addr 172.21.15.52:0/2899667053 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0ea2000640 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da75140 0x55de0da7d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 -- 172.21.15.52:0/2899667053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da75140 msgr2=0x55de0da7d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da75140 0x55de0da7d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 -- 172.21.15.52:0/2899667053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 msgr2=0x55de0da97ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 0x55de0da97ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.613+0000 7f0eaaa00640 1 -- 172.21.15.52:0/2899667053 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0da78590 con 0x55de0da74820 2024-04-26T01:49:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.614+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c010010 tx=0x7f0e9c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f6cb8b2181b1af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.614+0000 7f0e9aa00640 1 -- 172.21.15.52:0/2899667053 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c0049c0 con 0x55de0da74820 2024-04-26T01:49:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.614+0000 7f0e9aa00640 1 -- 172.21.15.52:0/2899667053 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e9c01c030 con 0x55de0da74820 2024-04-26T01:49:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.614+0000 7f0e9aa00640 1 -- 172.21.15.52:0/2899667053 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c02b520 con 0x55de0da74820 2024-04-26T01:49:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- 172.21.15.52:0/2899667053 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 msgr2=0x55de0da8b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c010010 tx=0x7f0e9c005bd0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- 172.21.15.52:0/2899667053 shutdown_connections 2024-04-26T01:49:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da7fa00 0x55de0da97ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da75140 0x55de0da7d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 --2- 172.21.15.52:0/2899667053 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da8b4f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- 172.21.15.52:0/2899667053 >> 172.21.15.52:0/2899667053 conn(0x55de0da70d30 msgr2=0x55de0da7c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- 172.21.15.52:0/2899667053 shutdown_connections 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- 172.21.15.52:0/2899667053 wait complete. 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 -- start start 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.615+0000 7f0eb439e740 1 Processor -- start 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da85690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da85bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1ae280 con 0x55de0da74820 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1af6f0 con 0x55de0da75140 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1ae950 con 0x55de0da7fa00 2024-04-26T01:49:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eb439e740 10 client.0 _ll_register_callbacks cb 0x7f0e94098df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da85690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eab400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da85690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58656/0 (socket says 172.21.15.52:58656) 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eab400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:48874/0 (socket says 172.21.15.52:48874) 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0eaaa00640 1 -- 172.21.15.52:0/1382108370 learned_addr learned my addr 172.21.15.52:0/1382108370 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.616+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da85bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 -- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 msgr2=0x55de0da85bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da85bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 -- 172.21.15.52:0/1382108370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 msgr2=0x55de0da85690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da85690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 -- 172.21.15.52:0/1382108370 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0daf5580 con 0x55de0da7fa00 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0ea2000640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da85bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:10.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0eab400640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ea402a860 tx=0x7f0ea402a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57548d1e82a8c67e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0e8fe00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.617+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4050070 con 0x55de0da7fa00 2024-04-26T01:49:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e9c01c030 con 0x55de0da7fa00 2024-04-26T01:49:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8477+0+0 (secure 0 0 0) 0x7f0ea4003a00 con 0x55de0da7fa00 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0ea4037040 con 0x55de0da7fa00 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0eb439e740 10 client.25666 fetch_fsmap learned FSMap version 31 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de0daf68e0 con 0x55de0da7fa00 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e9c01c030 con 0x55de0da7fa00 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.618+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0ea4049040 con 0x55de0da7fa00 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 10 client.25666 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 10 client.25666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 20 client.25666 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0daf7040 0x55de0daf9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 4 client.25666 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de0d9e8320 con 0x55de0daf7040 2024-04-26T01:49:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0daf7040 0x55de0daf9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0eaaa00640 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0daf7040 0x55de0daf9460 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=387fd69bbfc69a59 server_cookie=403c5616717f6cd8 in_seq=0 out_seq=0 2024-04-26T01:49:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.619+0000 7f0e8fe00640 10 client.25666 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.648+0000 7f0e8fe00640 1 -- 172.21.15.52:0/1382108370 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de0d9e8320 con 0x55de0daf7040 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.648+0000 7f0e8fe00640 10 client.25666 handle_command_reply: tid=0 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.648+0000 7f0eb439e740 1 client.25666 shutdown 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.648+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 msgr2=0x55de0da86110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.648+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ea402a860 tx=0x7f0ea402a340 comp rx=0 tx=0).stop 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 shutdown_connections 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55de0daf7040 0x55de0daf9460 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55de0da7fa00 0x55de0da86110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55de0da75140 0x55de0da85bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 --2- 172.21.15.52:0/1382108370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55de0da74820 0x55de0da85690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0e8fe00640 0 client.25666 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 >> 172.21.15.52:0/1382108370 conn(0x55de0da70d30 msgr2=0x55de0da7d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 shutdown_connections 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 1 -- 172.21.15.52:0/1382108370 wait complete. 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.649+0000 7f0eb439e740 20 client.25666 trim_cache size 0 max 16384 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:49:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:49:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:49:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:49:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 msgr2=0x7f0e900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 0x7f0e900a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009830 tx=0x7f0ea402c040 comp rx=0 tx=0).stop 2024-04-26T01:49:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 msgr2=0x7f0eac1a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e9f0 tx=0x7f0e9400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 shutdown_connections 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e900a3e40 0x7f0e900a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0eac106fe0 0x7f0eac19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0eac101b40 0x7f0eac1a0ab0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 --2- 172.21.15.52:0/1757018382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0eac1011a0 0x7f0eac1a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 >> 172.21.15.52:0/1757018382 conn(0x7f0eac076b80 msgr2=0x7f0eac06a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.650+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 shutdown_connections 2024-04-26T01:49:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:10.651+0000 7f0eb1600640 1 -- 172.21.15.52:0/1757018382 wait complete. 2024-04-26T01:49:11.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 56447 0.44 MB/sec execute 30 sec latency 46.553 ms 2024-04-26T01:49:11.174 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:49:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T01:49:11.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:49:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T01:49:11.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:49:10 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/1602109268 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571204fda40 con 0x5571204d0c00 2024-04-26T01:49:11.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:49:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T01:49:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:11 smithi191 ceph-mon[32745]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:11 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T01:49:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:11 smithi183 ceph-mon[32752]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:11 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T01:49:11.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:11 smithi052 ceph-mon[27851]: pgmap v128: 97 pgs: 97 active+clean; 40 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-04-26T01:49:11.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:11 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T01:49:12.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 57068 0.61 MB/sec execute 31 sec latency 27.582 ms 2024-04-26T01:49:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:12 smithi191 ceph-mon[32745]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:12 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:49:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:12 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/2373306392,v1:172.21.15.183:6837/2373306392] up:boot 2024-04-26T01:49:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:12 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:49:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:12 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:12.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:12 smithi183 ceph-mon[32752]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:12.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:12 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:49:12.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:12 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/2373306392,v1:172.21.15.183:6837/2373306392] up:boot 2024-04-26T01:49:12.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:12 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:49:12.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:12 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:12.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:12 smithi052 ceph-mon[27851]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T01:49:12.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:12 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:49:12.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:12 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/2373306392,v1:172.21.15.183:6837/2373306392] up:boot 2024-04-26T01:49:12.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:12 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:49:12.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:12 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:13.076 INFO:tasks.workunit.client.0.smithi052.stdout: 1 57383 0.60 MB/sec execute 32 sec latency 24.666 ms 2024-04-26T01:49:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:13 smithi191 ceph-mon[32745]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:13.677 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:13 smithi183 ceph-mon[32752]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:13 smithi052 ceph-mon[27851]: pgmap v130: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:14.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 58420 0.74 MB/sec execute 33 sec latency 17.360 ms 2024-04-26T01:49:15.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 60306 1.07 MB/sec execute 34 sec latency 19.127 ms 2024-04-26T01:49:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:15 smithi191 ceph-mon[32745]: pgmap v131: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:15 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:15 smithi183 ceph-mon[32752]: pgmap v131: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:15 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:15.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:15 smithi052 ceph-mon[27851]: pgmap v131: 97 pgs: 97 active+clean; 38 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 230 KiB/s wr, 48 op/s 2024-04-26T01:49:15.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:15 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:49:16.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 60759 1.09 MB/sec execute 35 sec latency 20.034 ms 2024-04-26T01:49:16.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:16 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:16.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:16 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:16.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:16 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:49:16.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:16 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:16.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:16 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:16.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:16 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:49:16.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:16 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:49:16.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:16 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:49:16.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:16 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:49:17.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 61511 1.12 MB/sec execute 36 sec latency 28.393 ms 2024-04-26T01:49:17.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:17 smithi191 ceph-mon[32745]: pgmap v132: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:17.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:17 smithi183 ceph-mon[32752]: pgmap v132: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:17 smithi052 ceph-mon[27851]: pgmap v132: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:18.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 62747 1.23 MB/sec execute 37 sec latency 20.629 ms 2024-04-26T01:49:19.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 64004 1.47 MB/sec execute 38 sec latency 33.733 ms 2024-04-26T01:49:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:19 smithi191 ceph-mon[32745]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:19.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:19 smithi183 ceph-mon[32752]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:19.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:19 smithi052 ceph-mon[27851]: pgmap v133: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:20.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 64336 1.47 MB/sec execute 39 sec latency 20.990 ms 2024-04-26T01:49:21.077 INFO:tasks.workunit.client.0.smithi052.stdout: 1 65108 1.49 MB/sec execute 40 sec latency 19.111 ms 2024-04-26T01:49:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:21 smithi191 ceph-mon[32745]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:21 smithi183 ceph-mon[32752]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:21.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:21 smithi052 ceph-mon[27851]: pgmap v134: 97 pgs: 97 active+clean; 41 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 613 KiB/s wr, 110 op/s 2024-04-26T01:49:22.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 66637 1.67 MB/sec execute 41 sec latency 20.712 ms 2024-04-26T01:49:23.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 67616 1.81 MB/sec execute 42 sec latency 38.107 ms 2024-04-26T01:49:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:23 smithi191 ceph-mon[32745]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 772 KiB/s wr, 121 op/s 2024-04-26T01:49:23.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:23 smithi183 ceph-mon[32752]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 772 KiB/s wr, 121 op/s 2024-04-26T01:49:23.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:23 smithi052 ceph-mon[27851]: pgmap v135: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 772 KiB/s wr, 121 op/s 2024-04-26T01:49:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 -- 172.21.15.52:0/479299119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c109470 msgr2=0x7f613c111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 --2- 172.21.15.52:0/479299119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c109470 0x7f613c111040 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6134009be0 tx=0x7f613402a6c0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 -- 172.21.15.52:0/479299119 shutdown_connections 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 --2- 172.21.15.52:0/479299119 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c109470 0x7f613c111040 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 --2- 172.21.15.52:0/479299119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c104d50 0x7f613c108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 --2- 172.21.15.52:0/479299119 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c1043a0 0x7f613c104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 -- 172.21.15.52:0/479299119 >> 172.21.15.52:0/479299119 conn(0x7f613c0ffc40 msgr2=0x7f613c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 -- 172.21.15.52:0/479299119 shutdown_connections 2024-04-26T01:49:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.816+0000 7f6141200640 1 -- 172.21.15.52:0/479299119 wait complete. 2024-04-26T01:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.817+0000 7f6141200640 1 Processor -- start 2024-04-26T01:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 -- start start 2024-04-26T01:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c109470 0x7f613c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c19a680 0x7f613c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1143c0 con 0x7f613c109470 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f613c114240 con 0x7f613c19a680 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6141200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f613c114540 con 0x7f613c1043a0 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6139e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.818+0000 7f6139400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c109470 0x7f613c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59430/0 (socket says 172.21.15.52:59430) 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 -- 172.21.15.52:0/421001666 learned_addr learned my addr 172.21.15.52:0/421001666 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f613a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c19a680 0x7f613c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 -- 172.21.15.52:0/421001666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c19a680 msgr2=0x7f613c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c19a680 0x7f613c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 -- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c109470 msgr2=0x7f613c1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c109470 0x7f613c1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 -- 172.21.15.52:0/421001666 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c19b280 con 0x7f613c1043a0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.819+0000 7f6139e00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9f0 tx=0x7f612c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.820+0000 7f6132a00640 1 -- 172.21.15.52:0/421001666 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c00ccc0 con 0x7f613c1043a0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.820+0000 7f6132a00640 1 -- 172.21.15.52:0/421001666 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612c0108e0 con 0x7f613c1043a0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.820+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c1a7360 con 0x7f613c1043a0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.820+0000 7f6132a00640 1 -- 172.21.15.52:0/421001666 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c018880 con 0x7f613c1043a0 2024-04-26T01:49:23.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.820+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1a77c0 con 0x7f613c1043a0 2024-04-26T01:49:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.823+0000 7f6132a00640 1 -- 172.21.15.52:0/421001666 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f612c010430 con 0x7f613c1043a0 2024-04-26T01:49:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.823+0000 7f614419a740 1 Processor -- start 2024-04-26T01:49:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.823+0000 7f614419a740 1 -- start start 2024-04-26T01:49:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea01140 0x55b04ea0b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea0ba00 0x55b04ea0ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b04e966900 con 0x55b04ea01140 2024-04-26T01:49:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b04e90cef0 con 0x55b04ea0ba00 2024-04-26T01:49:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f614419a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b04e9628c0 con 0x55b04ea00820 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f613a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f613a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59434/0 (socket says 172.21.15.52:59434) 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f613a800640 1 -- 172.21.15.52:0/493049990 learned_addr learned my addr 172.21.15.52:0/493049990 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f6139e00640 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea01140 0x55b04ea0b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.824+0000 7f6139400640 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea0ba00 0x55b04ea0ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 -- 172.21.15.52:0/493049990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea0ba00 msgr2=0x55b04ea0ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea0ba00 0x55b04ea0ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 -- 172.21.15.52:0/493049990 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea01140 msgr2=0x55b04ea0b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea01140 0x55b04ea0b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 -- 172.21.15.52:0/493049990 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04e679210 con 0x55b04ea00820 2024-04-26T01:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6132a00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 0x7f61200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f613a800640 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6134009be0 tx=0x7f6134030430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caf4a80edcbba202 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6132a00640 1 -- 172.21.15.52:0/421001666 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f612c0d0b80 con 0x7f613c1043a0 2024-04-26T01:49:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6127400640 1 -- 172.21.15.52:0/493049990 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134041030 con 0x55b04ea00820 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6139e00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 0x7f61200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6127400640 1 -- 172.21.15.52:0/493049990 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6134003e20 con 0x55b04ea00820 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.825+0000 7f6127400640 1 -- 172.21.15.52:0/493049990 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134036040 con 0x55b04ea00820 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 -- 172.21.15.52:0/493049990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 msgr2=0x55b04ea00c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6134009be0 tx=0x7f6134030430 comp rx=0 tx=0).stop 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f6139e00640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 0x7f61200a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55b04ea01dc0 tx=0x7f612c0d4630 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 -- 172.21.15.52:0/493049990 shutdown_connections 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea0ba00 0x55b04ea0ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea01140 0x55b04ea0b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 --2- 172.21.15.52:0/493049990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea00c00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.826+0000 7f614419a740 1 -- 172.21.15.52:0/493049990 >> 172.21.15.52:0/493049990 conn(0x55b04e9fcd30 msgr2=0x55b04e9fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.827+0000 7f614419a740 1 -- 172.21.15.52:0/493049990 shutdown_connections 2024-04-26T01:49:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.827+0000 7f614419a740 1 -- 172.21.15.52:0/493049990 wait complete. 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.827+0000 7f614419a740 1 -- start start 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.827+0000 7f614419a740 1 Processor -- start 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea089c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea09440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b04ea0e5d0 con 0x55b04ea0ba00 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b04e920310 con 0x55b04ea01140 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b04ea0e750 con 0x55b04ea00820 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f614419a740 10 client.0 _ll_register_callbacks cb 0x55b04e8eecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58434/0 (socket says 172.21.15.52:58434) 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f6139400640 1 -- 172.21.15.52:0/3810025331 learned_addr learned my addr 172.21.15.52:0/3810025331 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f613a800640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.828+0000 7f613a800640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea09440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 -- 172.21.15.52:0/3810025331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 msgr2=0x55b04ea089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea089c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 -- 172.21.15.52:0/3810025331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 msgr2=0x55b04ea09440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea09440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 -- 172.21.15.52:0/3810025331 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04e745620 con 0x55b04ea01140 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f6139400640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6128012700 tx=0x7f6128012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc4a1bf259d2496 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f611be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612800f920 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f612800fac0 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.829+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6134003e20 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f612800fd00 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 10 client.25676 fetch_fsmap learned FSMap version 34 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b04ea1a950 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6134003e20 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7f612801c420 con 0x55b04ea01140 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 10 client.25676 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 10 client.25676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 20 client.25676 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:23.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea1e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 4 client.25676 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:23.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.830+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b04e97a8b0 con 0x55b04ea1bed0 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.831+0000 7f613a800640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea1e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.832+0000 7f613a800640 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea1e2c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b32cbabd2e7f769c server_cookie=b8ae87e5b200c51d in_seq=0 out_seq=0 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.832+0000 7f611be00640 10 client.25676 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.832+0000 7f611be00640 1 -- 172.21.15.52:0/3810025331 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b04e97a8b0 con 0x55b04ea1bed0 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.832+0000 7f611be00640 10 client.25676 handle_command_reply: tid=0 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 client.25676 shutdown 2024-04-26T01:49:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 msgr2=0x55b04ea08f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6128012700 tx=0x7f6128012bc0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 shutdown_connections 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea1e2c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea09440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f611be00640 0 client.25676 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea08f00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 --2- 172.21.15.52:0/3810025331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 >> 172.21.15.52:0/3810025331 conn(0x55b04e9fcd30 msgr2=0x55b04ea02ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 shutdown_connections 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 1 -- 172.21.15.52:0/3810025331 wait complete. 2024-04-26T01:49:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.833+0000 7f614419a740 20 client.25676 trim_cache size 0 max 16384 2024-04-26T01:49:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.857+0000 7f614419a740 1 Processor -- start 2024-04-26T01:49:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.857+0000 7f614419a740 1 -- start start 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.857+0000 7f614419a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea17490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f614419a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f614419a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea240f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f614419a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1b2510 con 0x55b04ea0ba00 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f614419a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1afd50 con 0x55b04ea01140 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f614419a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1adba0 con 0x55b04ea00820 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f613a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea240f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58436/0 (socket says 172.21.15.52:58436) 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 -- 172.21.15.52:0/3295694097 learned_addr learned my addr 172.21.15.52:0/3295694097 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f613a800640 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea17490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 -- 172.21.15.52:0/3295694097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 msgr2=0x55b04ea17490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea17490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 -- 172.21.15.52:0/3295694097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 msgr2=0x55b04ea240f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea240f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.858+0000 7f6139400640 1 -- 172.21.15.52:0/3295694097 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04ea1a760 con 0x55b04ea01140 2024-04-26T01:49:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.859+0000 7f6139400640 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6128006010 tx=0x7f612801cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e81293cc9a85a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.859+0000 7f6126a00640 1 -- 172.21.15.52:0/3295694097 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128029020 con 0x55b04ea01140 2024-04-26T01:49:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.859+0000 7f6126a00640 1 -- 172.21.15.52:0/3295694097 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6128005560 con 0x55b04ea01140 2024-04-26T01:49:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.859+0000 7f6126a00640 1 -- 172.21.15.52:0/3295694097 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612801c680 con 0x55b04ea01140 2024-04-26T01:49:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- 172.21.15.52:0/3295694097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 msgr2=0x55b04ea23bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6128006010 tx=0x7f612801cdc0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- 172.21.15.52:0/3295694097 shutdown_connections 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea0ba00 0x55b04ea240f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea23bb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 --2- 172.21.15.52:0/3295694097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea00820 0x55b04ea17490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- 172.21.15.52:0/3295694097 >> 172.21.15.52:0/3295694097 conn(0x55b04e9fcd30 msgr2=0x55b04e9fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- 172.21.15.52:0/3295694097 shutdown_connections 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- 172.21.15.52:0/3295694097 wait complete. 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.860+0000 7f614419a740 1 -- start start 2024-04-26T01:49:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 Processor -- start 2024-04-26T01:49:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea00820 0x55b04ea10c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 0x55b04ea7a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1ae210 con 0x55b04ea00820 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1af680 con 0x55b04ea01140 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1ae8e0 con 0x55b04ea0ba00 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.861+0000 7f614419a740 10 client.0 _ll_register_callbacks cb 0x7f6128009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f613a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea00820 0x55b04ea10c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58450/0 (socket says 172.21.15.52:58450) 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 -- 172.21.15.52:0/1521053116 learned_addr learned my addr 172.21.15.52:0/1521053116 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f613a800640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 0x55b04ea7a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 -- 172.21.15.52:0/1521053116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 msgr2=0x55b04ea7a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 0x55b04ea7a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 -- 172.21.15.52:0/1521053116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea00820 msgr2=0x55b04ea10c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea00820 0x55b04ea10c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 -- 172.21.15.52:0/1521053116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04ea7a9e0 con 0x55b04ea01140 2024-04-26T01:49:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f613a800640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 0x55b04ea7a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f6139400640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55b04e9f8700 tx=0x7f6128009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=804fb919e2c5dd07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.862+0000 7f611be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61280052e0 con 0x55b04ea01140 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f61280035f0 con 0x55b04ea01140 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6128005560 con 0x55b04ea01140 2024-04-26T01:49:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6128029020 con 0x55b04ea01140 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f614419a740 10 client.25682 fetch_fsmap learned FSMap version 34 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.863+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b04ea81d70 con 0x55b04ea01140 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6128005560 con 0x55b04ea01140 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7f6128037460 con 0x55b04ea01140 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 10 client.25682 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 20 client.25682 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea84470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 4 client.25682 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b04e974320 con 0x55b04ea1bed0 2024-04-26T01:49:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.864+0000 7f613a800640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea84470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.865+0000 7f613a800640 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea84470 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73be3b5ccd5f7eda server_cookie=d9f7be9748280e5d in_seq=0 out_seq=0 2024-04-26T01:49:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.865+0000 7f611be00640 10 client.25682 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.901+0000 7f611be00640 1 -- 172.21.15.52:0/1521053116 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b04e974320 con 0x55b04ea1bed0 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.901+0000 7f611be00640 10 client.25682 handle_command_reply: tid=0 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.901+0000 7f614419a740 1 client.25682 shutdown 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 msgr2=0x55b04ea11140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55b04e9f8700 tx=0x7f6128009770 comp rx=0 tx=0).stop 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 shutdown_connections 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b04ea1bed0 0x55b04ea84470 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b04ea0ba00 0x55b04ea7a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b04ea01140 0x55b04ea11140 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 --2- 172.21.15.52:0/1521053116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b04ea00820 0x55b04ea10c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 >> 172.21.15.52:0/1521053116 conn(0x55b04e9fcd30 msgr2=0x55b04ea08980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f611be00640 0 client.25682 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 shutdown_connections 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 1 -- 172.21.15.52:0/1521053116 wait complete. 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.902+0000 7f614419a740 20 client.25682 trim_cache size 0 max 16384 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:49:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:49:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:49:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.903+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 msgr2=0x7f61200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.903+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 0x7f61200a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55b04ea01dc0 tx=0x7f612c0d4630 comp rx=0 tx=0).stop 2024-04-26T01:49:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.903+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 msgr2=0x7f613c1a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.903+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9f0 tx=0x7f612c00edf0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 shutdown_connections 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61200a3e40 0x7f61200a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f613c19a680 0x7f613c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f613c109470 0x7f613c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 --2- 172.21.15.52:0/421001666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f613c1043a0 0x7f613c1a05c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 >> 172.21.15.52:0/421001666 conn(0x7f613c0ffc40 msgr2=0x7f613c100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 shutdown_connections 2024-04-26T01:49:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:23.904+0000 7f6141200640 1 -- 172.21.15.52:0/421001666 wait complete. 2024-04-26T01:49:24.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 67945 1.78 MB/sec execute 43 sec latency 31.845 ms 2024-04-26T01:49:24.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:24 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:24.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:24 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:24.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:24 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:25.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 68824 1.85 MB/sec execute 44 sec latency 24.949 ms 2024-04-26T01:49:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:25 smithi191 ceph-mon[32745]: pgmap v136: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 685 KiB/s wr, 107 op/s 2024-04-26T01:49:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:25 smithi183 ceph-mon[32752]: pgmap v136: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 685 KiB/s wr, 107 op/s 2024-04-26T01:49:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:25 smithi052 ceph-mon[27851]: pgmap v136: 97 pgs: 97 active+clean; 44 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 685 KiB/s wr, 107 op/s 2024-04-26T01:49:26.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 70203 1.94 MB/sec execute 45 sec latency 20.685 ms 2024-04-26T01:49:27.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 71185 2.07 MB/sec execute 46 sec latency 30.330 ms 2024-04-26T01:49:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:27 smithi191 ceph-mon[32745]: pgmap v137: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-04-26T01:49:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:27 smithi183 ceph-mon[32752]: pgmap v137: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-04-26T01:49:27.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:27 smithi052 ceph-mon[27851]: pgmap v137: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 180 op/s 2024-04-26T01:49:28.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 71423 2.03 MB/sec execute 47 sec latency 22.390 ms 2024-04-26T01:49:29.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 72149 2.03 MB/sec execute 48 sec latency 24.269 ms 2024-04-26T01:49:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:29 smithi191 ceph-mon[32745]: pgmap v138: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:29.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:29 smithi183 ceph-mon[32752]: pgmap v138: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:29.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:29 smithi052 ceph-mon[27851]: pgmap v138: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:30.078 INFO:tasks.workunit.client.0.smithi052.stdout: 1 73487 2.11 MB/sec execute 49 sec latency 23.611 ms 2024-04-26T01:49:31.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 74651 2.27 MB/sec execute 50 sec latency 31.761 ms 2024-04-26T01:49:31.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:31 smithi191 ceph-mon[32745]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:31 smithi183 ceph-mon[32752]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:31 smithi052 ceph-mon[27851]: pgmap v139: 97 pgs: 97 active+clean; 44 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 125 op/s 2024-04-26T01:49:32.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 74959 2.25 MB/sec execute 51 sec latency 29.896 ms 2024-04-26T01:49:33.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 75533 2.23 MB/sec execute 52 sec latency 25.525 ms 2024-04-26T01:49:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:33 smithi191 ceph-mon[32745]: pgmap v140: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-04-26T01:49:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:33 smithi183 ceph-mon[32752]: pgmap v140: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-04-26T01:49:33.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:33 smithi052 ceph-mon[27851]: pgmap v140: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 173 op/s 2024-04-26T01:49:34.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 76557 2.27 MB/sec execute 53 sec latency 20.556 ms 2024-04-26T01:49:35.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 77913 2.39 MB/sec execute 54 sec latency 24.344 ms 2024-04-26T01:49:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:35 smithi183 ceph-mon[32752]: pgmap v141: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 718 KiB/s wr, 119 op/s 2024-04-26T01:49:35.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:35 smithi052 ceph-mon[27851]: pgmap v141: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 718 KiB/s wr, 119 op/s 2024-04-26T01:49:35.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:35 smithi191 ceph-mon[32745]: pgmap v141: 97 pgs: 97 active+clean; 47 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 718 KiB/s wr, 119 op/s 2024-04-26T01:49:36.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 78478 2.43 MB/sec execute 55 sec latency 28.526 ms 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 -- 172.21.15.52:0/194291556 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870102f30 msgr2=0x7f9870103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 --2- 172.21.15.52:0/194291556 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870102f30 0x7f9870103310 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9860009900 tx=0x7f986002a480 comp rx=0 tx=0).stop 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 -- 172.21.15.52:0/194291556 shutdown_connections 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 --2- 172.21.15.52:0/194291556 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f987010c350 0x7f987010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 --2- 172.21.15.52:0/194291556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9870103850 0x7f987010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 --2- 172.21.15.52:0/194291556 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870102f30 0x7f9870103310 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 -- 172.21.15.52:0/194291556 >> 172.21.15.52:0/194291556 conn(0x7f98700fc730 msgr2=0x7f98700feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 -- 172.21.15.52:0/194291556 shutdown_connections 2024-04-26T01:49:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.058+0000 7f9876000640 1 -- 172.21.15.52:0/194291556 wait complete. 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.059+0000 7f9876000640 1 Processor -- start 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.059+0000 7f9876000640 1 -- start start 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 0x7f98701010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f987010c350 0x7f9870101600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9870111fe0 con 0x7f987010c350 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9870111e60 con 0x7f9870103850 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9876000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9870112160 con 0x7f9870102f30 2024-04-26T01:49:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9874c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9874c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45884/0 (socket says 172.21.15.52:45884) 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9874c00640 1 -- 172.21.15.52:0/2132725566 learned_addr learned my addr 172.21.15.52:0/2132725566 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9867e00640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 0x7f98701010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.060+0000 7f9875600640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f987010c350 0x7f9870101600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 -- 172.21.15.52:0/2132725566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 msgr2=0x7f98701010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 0x7f98701010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 -- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f987010c350 msgr2=0x7f9870101600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f987010c350 0x7f9870101600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 -- 172.21.15.52:0/2132725566 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98701a6a50 con 0x7f9870102f30 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9867e00640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 0x7f98701010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f9874c00640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9860005940 tx=0x7f9860004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f986e000640 1 -- 172.21.15.52:0/2132725566 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986003f030 con 0x7f9870102f30 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.061+0000 7f986e000640 1 -- 172.21.15.52:0/2132725566 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9860003aa0 con 0x7f9870102f30 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.062+0000 7f986e000640 1 -- 172.21.15.52:0/2132725566 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860035070 con 0x7f9870102f30 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.062+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98701a6ce0 con 0x7f9870102f30 2024-04-26T01:49:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.062+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98701a7180 con 0x7f9870102f30 2024-04-26T01:49:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.065+0000 7f987a3cb740 1 Processor -- start 2024-04-26T01:49:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.065+0000 7f987a3cb740 1 -- start start 2024-04-26T01:49:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f986e000640 1 -- 172.21.15.52:0/2132725566 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9860042030 con 0x7f9870102f30 2024-04-26T01:49:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa45d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa50650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555f8f9ab900 con 0x555f8fa459b0 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555f8f951ef0 con 0x555f8fa50b90 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f987a3cb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555f8f9a78c0 con 0x555f8fa462d0 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f9867e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f9875600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa45d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f9867e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54986/0 (socket says 172.21.15.52:54986) 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f9867e00640 1 -- 172.21.15.52:0/259693746 learned_addr learned my addr 172.21.15.52:0/259693746 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.066+0000 7f9874c00640 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa50650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 -- 172.21.15.52:0/259693746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 msgr2=0x555f8fa50650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa50650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 -- 172.21.15.52:0/259693746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 msgr2=0x555f8fa45d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa45d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 -- 172.21.15.52:0/259693746 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8f6be210 con 0x555f8fa50b90 2024-04-26T01:49:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9867e00640 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f985c010480 tx=0x7f985c010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=857246c80d43223d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9865c00640 1 -- 172.21.15.52:0/259693746 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c004220 con 0x555f8fa50b90 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9865c00640 1 -- 172.21.15.52:0/259693746 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f985c0043c0 con 0x555f8fa50b90 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.067+0000 7f9865c00640 1 -- 172.21.15.52:0/259693746 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c011a10 con 0x555f8fa50b90 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.068+0000 7f986e000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 0x7f98540a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.068+0000 7f986e000640 1 -- 172.21.15.52:0/2132725566 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f98600f0d70 con 0x7f9870102f30 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.068+0000 7f9875600640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 0x7f98540a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.068+0000 7f987a3cb740 1 -- 172.21.15.52:0/259693746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 msgr2=0x555f8fa52f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.068+0000 7f987a3cb740 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f985c010480 tx=0x7f985c010940 comp rx=0 tx=0).stop 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 -- 172.21.15.52:0/259693746 shutdown_connections 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa52f80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa50650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 --2- 172.21.15.52:0/259693746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa45d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 -- 172.21.15.52:0/259693746 >> 172.21.15.52:0/259693746 conn(0x555f8fa41ec0 msgr2=0x555f8fa422a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 -- 172.21.15.52:0/259693746 shutdown_connections 2024-04-26T01:49:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f9875600640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 0x7f98540a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9868005fd0 tx=0x7f9868009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 -- 172.21.15.52:0/259693746 wait complete. 2024-04-26T01:49:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.069+0000 7f987a3cb740 1 -- start start 2024-04-26T01:49:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 Processor -- start 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa4e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555f8fa537c0 con 0x555f8fa459b0 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555f8f965310 con 0x555f8fa50b90 2024-04-26T01:49:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555f8fa53940 con 0x555f8fa462d0 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.070+0000 7f987a3cb740 10 client.0 _ll_register_callbacks cb 0x555f8f933cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9874c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9867e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa4e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54990/0 (socket says 172.21.15.52:54990) 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 -- 172.21.15.52:0/3903947512 learned_addr learned my addr 172.21.15.52:0/3903947512 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9874c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54368/0 (socket says 172.21.15.52:54368) 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 -- 172.21.15.52:0/3903947512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 msgr2=0x555f8fa4e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa4e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 -- 172.21.15.52:0/3903947512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 msgr2=0x555f8fa4db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 -- 172.21.15.52:0/3903947512 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8f78a620 con 0x555f8fa50b90 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9874c00640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:37.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f9875600640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f98680099c0 tx=0x7f9868004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc9416ffca977b38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.071+0000 7f985a000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868004b20 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f9868004cc0 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f985c0043c0 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f986800f750 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f987a3cb740 10 client.25694 fetch_fsmap learned FSMap version 34 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555f8fa5faa0 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.072+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f985c0043c0 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7f9868026070 con 0x555f8fa50b90 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 10 client.25694 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 10 client.25694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 20 client.25694 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fa61020 0x555f8fa63470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 4 client.25694 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f8f9bf8b0 con 0x555f8fa61020 2024-04-26T01:49:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.073+0000 7f9874c00640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fa61020 0x555f8fa63470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.074+0000 7f9874c00640 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fa61020 0x555f8fa63470 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cf14192b24c415e server_cookie=4da3ce25c3d81d93 in_seq=0 out_seq=0 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.074+0000 7f985a000640 10 client.25694 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f985a000640 1 -- 172.21.15.52:0/3903947512 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f8f9bf8b0 con 0x555f8fa61020 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f985a000640 10 client.25694 handle_command_reply: tid=0 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 client.25694 shutdown 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 msgr2=0x555f8fa4e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f98680099c0 tx=0x7f9868004ab0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 shutdown_connections 2024-04-26T01:49:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fa61020 0x555f8fa63470 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa50b90 0x555f8fa4e5b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa462d0 0x555f8fa4e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 --2- 172.21.15.52:0/3903947512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f985a000640 0 client.25694 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 >> 172.21.15.52:0/3903947512 conn(0x555f8fa41ec0 msgr2=0x555f8fa47c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 shutdown_connections 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 1 -- 172.21.15.52:0/3903947512 wait complete. 2024-04-26T01:49:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.075+0000 7f987a3cb740 20 client.25694 trim_cache size 0 max 16384 2024-04-26T01:49:37.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 79158 2.43 MB/sec execute 56 sec latency 23.308 ms 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 Processor -- start 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 -- start start 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa5c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa483c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f98701b24d0 con 0x555f8fa459b0 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98701afd10 con 0x555f8fa462d0 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.096+0000 7f987a3cb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f98701adb60 con 0x555f8fa50b90 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45912/0 (socket says 172.21.15.52:45912) 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 -- 172.21.15.52:0/747369612 learned_addr learned my addr 172.21.15.52:0/747369612 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9867e00640 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 -- 172.21.15.52:0/747369612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 msgr2=0x555f8fa483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9874c00640 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa5c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 -- 172.21.15.52:0/747369612 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 msgr2=0x555f8fa5c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa5c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 -- 172.21.15.52:0/747369612 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8fa48900 con 0x555f8fa50b90 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.097+0000 7f9875600640 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9868005730 tx=0x7f9868005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abb96fd187d68cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.098+0000 7f9865200640 1 -- 172.21.15.52:0/747369612 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868034070 con 0x555f8fa50b90 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.098+0000 7f9865200640 1 -- 172.21.15.52:0/747369612 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9868004b20 con 0x555f8fa50b90 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.098+0000 7f9865200640 1 -- 172.21.15.52:0/747369612 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986800fb50 con 0x555f8fa50b90 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- 172.21.15.52:0/747369612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 msgr2=0x555f8fa40ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9868005730 tx=0x7f9868005bd0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- 172.21.15.52:0/747369612 shutdown_connections 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa40ea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 --2- 172.21.15.52:0/747369612 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa5c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- 172.21.15.52:0/747369612 >> 172.21.15.52:0/747369612 conn(0x555f8fa41ec0 msgr2=0x555f8fa40790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- 172.21.15.52:0/747369612 shutdown_connections 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- 172.21.15.52:0/747369612 wait complete. 2024-04-26T01:49:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 -- start start 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.099+0000 7f987a3cb740 1 Processor -- start 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa56540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa56a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa56fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f98701ae1d0 con 0x555f8fa459b0 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f98701af640 con 0x555f8fa462d0 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f98701ae8a0 con 0x555f8fa50b90 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f987a3cb740 10 client.0 _ll_register_callbacks cb 0x7f9868003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f9874c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa56540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f9874c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa56540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54386/0 (socket says 172.21.15.52:54386) 2024-04-26T01:49:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f9874c00640 1 -- 172.21.15.52:0/348705031 learned_addr learned my addr 172.21.15.52:0/348705031 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f9875600640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa56fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.100+0000 7f9867e00640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa56a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 -- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 msgr2=0x555f8fa56a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa56a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 -- 172.21.15.52:0/348705031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 msgr2=0x555f8fa56540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa56540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 -- 172.21.15.52:0/348705031 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8fac6600 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f9875600640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa56fc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9868018040 tx=0x7f9868011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676c71336347c5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f985a000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868034070 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f986800f8f0 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.101+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9868004b20 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f986800faf0 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 10 client.25705 fetch_fsmap learned FSMap version 34 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555f8fac7870 con 0x555f8fa50b90 2024-04-26T01:49:37.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9868004b20 con 0x555f8fa50b90 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 34) ==== 8350+0+0 (secure 0 0 0) 0x7f9868026040 con 0x555f8fa50b90 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 10 client.25705 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 10 client.25705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 20 client.25705 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fac7ea0 0x555f8faca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 4 client.25705 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.102+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f8f9b9320 con 0x555f8fac7ea0 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.103+0000 7f9874c00640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fac7ea0 0x555f8faca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.103+0000 7f9874c00640 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fac7ea0 0x555f8faca2c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bf8f873bd22994 server_cookie=7430d21691a3949 in_seq=0 out_seq=0 2024-04-26T01:49:37.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.103+0000 7f985a000640 10 client.25705 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f985a000640 1 -- 172.21.15.52:0/348705031 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f8f9b9320 con 0x555f8fac7ea0 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f985a000640 10 client.25705 handle_command_reply: tid=0 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 client.25705 shutdown 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 msgr2=0x555f8fa56fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa56fc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9868018040 tx=0x7f9868011440 comp rx=0 tx=0).stop 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 shutdown_connections 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f8fac7ea0 0x555f8faca2c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f8fa50b90 0x555f8fa56fc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f8fa462d0 0x555f8fa56a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f985a000640 0 client.25705 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 --2- 172.21.15.52:0/348705031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f8fa459b0 0x555f8fa56540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.137+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 >> 172.21.15.52:0/348705031 conn(0x555f8fa41ec0 msgr2=0x555f8fa4e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.138+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 shutdown_connections 2024-04-26T01:49:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.138+0000 7f987a3cb740 1 -- 172.21.15.52:0/348705031 wait complete. 2024-04-26T01:49:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.138+0000 7f987a3cb740 20 client.25705 trim_cache size 0 max 16384 2024-04-26T01:49:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:49:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:49:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:49:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 msgr2=0x7f98540a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 0x7f98540a6280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9868005fd0 tx=0x7f9868009500 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 msgr2=0x7f9870100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9860005940 tx=0x7f9860004270 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 shutdown_connections 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f98540a3dc0 0x7f98540a6280 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f987010c350 0x7f9870101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9870103850 0x7f98701010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 --2- 172.21.15.52:0/2132725566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9870102f30 0x7f9870100b80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.139+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 >> 172.21.15.52:0/2132725566 conn(0x7f98700fc730 msgr2=0x7f98700fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.140+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 shutdown_connections 2024-04-26T01:49:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:37.140+0000 7f9876000640 1 -- 172.21.15.52:0/2132725566 wait complete. 2024-04-26T01:49:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:37 smithi183 ceph-mon[32752]: pgmap v142: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 999 KiB/s wr, 165 op/s 2024-04-26T01:49:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:37 smithi052 ceph-mon[27851]: pgmap v142: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 999 KiB/s wr, 165 op/s 2024-04-26T01:49:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:37 smithi191 ceph-mon[32745]: pgmap v142: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 999 KiB/s wr, 165 op/s 2024-04-26T01:49:38.079 INFO:tasks.workunit.client.0.smithi052.stdout: 1 80577 2.48 MB/sec execute 57 sec latency 36.151 ms 2024-04-26T01:49:39.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 81850 2.63 MB/sec execute 58 sec latency 21.019 ms 2024-04-26T01:49:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:39 smithi183 ceph-mon[32752]: pgmap v143: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:39.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:39 smithi052 ceph-mon[27851]: pgmap v143: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:39.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:39 smithi191 ceph-mon[32745]: pgmap v143: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:40.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 82158 2.59 MB/sec execute 59 sec latency 21.408 ms 2024-04-26T01:49:41.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 83131 2.63 MB/sec execute 60 sec latency 26.233 ms 2024-04-26T01:49:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:41 smithi183 ceph-mon[32752]: pgmap v144: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:41.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:41 smithi052 ceph-mon[27851]: pgmap v144: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:41.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:41 smithi191 ceph-mon[32745]: pgmap v144: 97 pgs: 97 active+clean; 50 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 93 op/s 2024-04-26T01:49:42.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 84809 2.72 MB/sec execute 61 sec latency 17.475 ms 2024-04-26T01:49:43.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 85541 2.78 MB/sec execute 62 sec latency 21.330 ms 2024-04-26T01:49:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:43 smithi052 ceph-mon[27851]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T01:49:43.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:43 smithi191 ceph-mon[32745]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T01:49:43.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:43 smithi183 ceph-mon[32752]: pgmap v145: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 169 op/s 2024-04-26T01:49:44.080 INFO:tasks.workunit.client.0.smithi052.stdout: 1 86120 2.74 MB/sec execute 63 sec latency 19.216 ms 2024-04-26T01:49:45.082 INFO:tasks.workunit.client.0.smithi052.stdout: 1 87376 2.78 MB/sec execute 64 sec latency 22.749 ms 2024-04-26T01:49:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:45 smithi052 ceph-mon[27851]: pgmap v146: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 121 op/s 2024-04-26T01:49:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:45 smithi191 ceph-mon[32745]: pgmap v146: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 121 op/s 2024-04-26T01:49:45.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:45 smithi183 ceph-mon[32752]: pgmap v146: 97 pgs: 97 active+clean; 49 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 121 op/s 2024-04-26T01:49:46.082 INFO:tasks.workunit.client.0.smithi052.stdout: 1 88830 2.92 MB/sec execute 65 sec latency 44.986 ms 2024-04-26T01:49:47.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 89176 2.90 MB/sec execute 66 sec latency 20.816 ms 2024-04-26T01:49:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:47 smithi052 ceph-mon[27851]: pgmap v147: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-04-26T01:49:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:47 smithi191 ceph-mon[32745]: pgmap v147: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-04-26T01:49:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:47 smithi183 ceph-mon[32752]: pgmap v147: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-04-26T01:49:48.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 89977 2.89 MB/sec execute 67 sec latency 18.700 ms 2024-04-26T01:49:49.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 91625 2.99 MB/sec execute 68 sec latency 15.810 ms 2024-04-26T01:49:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:49 smithi052 ceph-mon[27851]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:49 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:49 smithi191 ceph-mon[32745]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:49 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:49.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:49 smithi183 ceph-mon[32752]: pgmap v148: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:49.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:49 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:49:50.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 92523 3.04 MB/sec execute 69 sec latency 22.058 ms 2024-04-26T01:49:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 -- 172.21.15.52:0/2290678237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f903810c330 msgr2=0x7f903810e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 --2- 172.21.15.52:0/2290678237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f903810c330 0x7f903810e7d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9030009be0 tx=0x7f903002a750 comp rx=0 tx=0).stop 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 -- 172.21.15.52:0/2290678237 shutdown_connections 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 --2- 172.21.15.52:0/2290678237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f903810c330 0x7f903810e7d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 --2- 172.21.15.52:0/2290678237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9038103830 0x7f903810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 --2- 172.21.15.52:0/2290678237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f90381032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 -- 172.21.15.52:0/2290678237 >> 172.21.15.52:0/2290678237 conn(0x7f90380fc770 msgr2=0x7f90380feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 -- 172.21.15.52:0/2290678237 shutdown_connections 2024-04-26T01:49:50.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.315+0000 7f903ea00640 1 -- 172.21.15.52:0/2290678237 wait complete. 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.316+0000 7f903ea00640 1 Processor -- start 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.316+0000 7f903ea00640 1 -- start start 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f9038100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f903810c330 0x7f90381015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9038111fc0 con 0x7f9038102f10 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9038111e40 con 0x7f9038103830 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903ea00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9038112140 con 0x7f903810c330 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f9038100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35546/0 (socket says 172.21.15.52:35546) 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903cc00640 1 -- 172.21.15.52:0/4031136 learned_addr learned my addr 172.21.15.52:0/4031136 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.317+0000 7f903e000640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f903810c330 0x7f90381015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 -- 172.21.15.52:0/4031136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f903810c330 msgr2=0x7f90381015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f903810c330 0x7f90381015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 -- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 msgr2=0x7f9038100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f9038100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 -- 172.21.15.52:0/4031136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90381a6a60 con 0x7f9038103830 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903cc00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f902400e9f0 tx=0x7f902400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f9036000640 1 -- 172.21.15.52:0/4031136 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902400cac0 con 0x7f9038103830 2024-04-26T01:49:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903d600640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f9038100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f9036000640 1 -- 172.21.15.52:0/4031136 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90240054d0 con 0x7f9038103830 2024-04-26T01:49:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f9036000640 1 -- 172.21.15.52:0/4031136 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024004e50 con 0x7f9038103830 2024-04-26T01:49:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90381a6d50 con 0x7f9038103830 2024-04-26T01:49:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.318+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90381a71f0 con 0x7f9038103830 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.322+0000 7f9036000640 1 -- 172.21.15.52:0/4031136 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9024005020 con 0x7f9038103830 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.322+0000 7f9042e5d740 1 Processor -- start 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 -- start start 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b57f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fd38ab0900 con 0x55fd38b4a9b0 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fd38a56ef0 con 0x55fd38b4b2d0 2024-04-26T01:49:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9042e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fd38aac8c0 con 0x55fd38b55b90 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f9036000640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 0x7f90200a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f903d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f903d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35554/0 (socket says 172.21.15.52:35554) 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f903d600640 1 -- 172.21.15.52:0/1772427026 learned_addr learned my addr 172.21.15.52:0/1772427026 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.323+0000 7f903cc00640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b57f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f9036000640 1 -- 172.21.15.52:0/4031136 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f90240d0a00 con 0x7f9038103830 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903e000640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 -- 172.21.15.52:0/1772427026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 msgr2=0x55fd38b57f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b57f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 -- 172.21.15.52:0/1772427026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 msgr2=0x55fd38b4ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 -- 172.21.15.52:0/1772427026 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd387c3210 con 0x55fd38b4b2d0 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903e000640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903cc00640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b57f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903e000640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 0x7f90200a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.324+0000 7f903d600640 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f902c011510 tx=0x7f902c0119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae5f71e63ee25e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f902a600640 1 -- 172.21.15.52:0/1772427026 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c004020 con 0x55fd38b4b2d0 2024-04-26T01:49:50.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f902a600640 1 -- 172.21.15.52:0/1772427026 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902c0041c0 con 0x55fd38b4b2d0 2024-04-26T01:49:50.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f902a600640 1 -- 172.21.15.52:0/1772427026 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c00f870 con 0x55fd38b4b2d0 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f903e000640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 0x7f90200a63f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9030006010 tx=0x7f9030009300 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f9042e5d740 1 -- 172.21.15.52:0/1772427026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 msgr2=0x55fd38b55650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.325+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f902c011510 tx=0x7f902c0119d0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 -- 172.21.15.52:0/1772427026 shutdown_connections 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b57f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b55650 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1772427026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 -- 172.21.15.52:0/1772427026 >> 172.21.15.52:0/1772427026 conn(0x55fd38b46ec0 msgr2=0x55fd38b472a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 -- 172.21.15.52:0/1772427026 shutdown_connections 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 -- 172.21.15.52:0/1772427026 wait complete. 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 -- start start 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.326+0000 7f9042e5d740 1 Processor -- start 2024-04-26T01:49:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 0x55fd38b53550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fd38b587c0 con 0x55fd38b55b90 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fd38a6a310 con 0x55fd38b4b2d0 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fd38b58940 con 0x55fd38b4a9b0 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f9042e5d740 10 client.0 _ll_register_callbacks cb 0x55fd38a38cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f903d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f903d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45022/0 (socket says 172.21.15.52:45022) 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.327+0000 7f903d600640 1 -- 172.21.15.52:0/1532948352 learned_addr learned my addr 172.21.15.52:0/1532948352 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903e000640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 0x55fd38b53550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 -- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 msgr2=0x55fd38b53010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 -- 172.21.15.52:0/1532948352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 msgr2=0x55fd38b53550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 0x55fd38b53550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 -- 172.21.15.52:0/1532948352 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3888f620 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903e000640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 0x55fd38b53550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f903d600640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f902c005df0 tx=0x7f902c005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9361176bc90f6bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f901ea00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.328+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c027070 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f902c0041c0 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f902c003f30 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f902c033050 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 10 client.25717 fetch_fsmap learned FSMap version 35 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fd38b64940 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902c0041c0 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f902c004ad0 con 0x55fd38b4a9b0 2024-04-26T01:49:50.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 10 client.25717 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:49:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 10 client.25717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.329+0000 7f9042e5d740 20 client.25717 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.330+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38b681a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.330+0000 7f9042e5d740 4 client.25717 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:49:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.330+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd38ac48b0 con 0x55fd38b65db0 2024-04-26T01:49:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.330+0000 7f903cc00640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38b681a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.331+0000 7f903cc00640 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38b681a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f510feb68967b20 server_cookie=9d73af51b62a10c1 in_seq=0 out_seq=0 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.331+0000 7f901ea00640 10 client.25717 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.331+0000 7f901ea00640 1 -- 172.21.15.52:0/1532948352 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd38ac48b0 con 0x55fd38b65db0 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.331+0000 7f901ea00640 10 client.25717 handle_command_reply: tid=0 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.331+0000 7f9042e5d740 1 client.25717 shutdown 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 msgr2=0x55fd38b52ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f902c005df0 tx=0x7f902c005e20 comp rx=0 tx=0).stop 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 shutdown_connections 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38b681a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b55b90 0x55fd38b53550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f901ea00640 0 client.25717 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 --2- 172.21.15.52:0/1532948352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b4a9b0 0x55fd38b52ad0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 >> 172.21.15.52:0/1532948352 conn(0x55fd38b46ec0 msgr2=0x55fd38b4cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 shutdown_connections 2024-04-26T01:49:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 1 -- 172.21.15.52:0/1532948352 wait complete. 2024-04-26T01:49:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.332+0000 7f9042e5d740 20 client.25717 trim_cache size 0 max 16384 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 Processor -- start 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 -- start start 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b45ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f90381b24e0 con 0x55fd38b4b2d0 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90381afd50 con 0x55fd38b4a9b0 2024-04-26T01:49:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.352+0000 7f9042e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90381adba0 con 0x55fd38b55b90 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35572/0 (socket says 172.21.15.52:35572) 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 -- 172.21.15.52:0/331763122 learned_addr learned my addr 172.21.15.52:0/331763122 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903cc00640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903e000640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b45ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 -- 172.21.15.52:0/331763122 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 msgr2=0x55fd38b45ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b45ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 -- 172.21.15.52:0/331763122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 msgr2=0x55fd38b53370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903d600640 1 -- 172.21.15.52:0/331763122 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd38b4b100 con 0x55fd38b4a9b0 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903cc00640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:49:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.353+0000 7f903e000640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b45ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:49:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.354+0000 7f903d600640 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b163a9d94a962b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.354+0000 7f9029c00640 1 -- 172.21.15.52:0/331763122 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c027070 con 0x55fd38b4a9b0 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.354+0000 7f9029c00640 1 -- 172.21.15.52:0/331763122 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902c030070 con 0x55fd38b4a9b0 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.354+0000 7f9029c00640 1 -- 172.21.15.52:0/331763122 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c00fde0 con 0x55fd38b4a9b0 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- 172.21.15.52:0/331763122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 msgr2=0x55fd38b4d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c009b70 comp rx=0 tx=0).stop 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- 172.21.15.52:0/331763122 shutdown_connections 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b45ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4b2d0 0x55fd38b53370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 --2- 172.21.15.52:0/331763122 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4a9b0 0x55fd38b4d370 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- 172.21.15.52:0/331763122 >> 172.21.15.52:0/331763122 conn(0x55fd38b46ec0 msgr2=0x55fd38b45790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- 172.21.15.52:0/331763122 shutdown_connections 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- 172.21.15.52:0/331763122 wait complete. 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.355+0000 7f9042e5d740 1 -- start start 2024-04-26T01:49:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 Processor -- start 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b5ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b5b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f90381ae210 con 0x55fd38b4a9b0 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90381af680 con 0x55fd38b4b2d0 2024-04-26T01:49:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90381ae8e0 con 0x55fd38b55b90 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.356+0000 7f9042e5d740 10 client.0 _ll_register_callbacks cb 0x7f902c003920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b5ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b5b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45038/0 (socket says 172.21.15.52:45038) 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 -- 172.21.15.52:0/2534094008 learned_addr learned my addr 172.21.15.52:0/2534094008 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b5ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34354/0 (socket says 172.21.15.52:34354) 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 -- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 msgr2=0x55fd38b5b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b5b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 -- 172.21.15.52:0/2534094008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 msgr2=0x55fd38b5ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b5ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.357+0000 7f903e000640 1 -- 172.21.15.52:0/2534094008 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd38bca350 con 0x55fd38b55b90 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f903e000640 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9030009be0 tx=0x7f903002a750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3cc361598b7fc0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:49:50.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f901ea00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9030042040 con 0x55fd38b55b90 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f9030004040 con 0x55fd38b55b90 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f902c030070 con 0x55fd38b55b90 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9030036040 con 0x55fd38b55b90 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f9042e5d740 10 client.25723 fetch_fsmap learned FSMap version 35 2024-04-26T01:49:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.358+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fd38bcb640 con 0x55fd38b55b90 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902c030070 con 0x55fd38b55b90 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f9030048040 con 0x55fd38b55b90 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 10 client.25723 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 10 client.25723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 20 client.25723 populate_metadata read hostname 'smithi052' 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38bcde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 4 client.25723 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.359+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd38abe320 con 0x55fd38b65db0 2024-04-26T01:49:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.360+0000 7f903d600640 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38bcde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:49:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.360+0000 7f903d600640 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38bcde70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cd10fd48f895441 server_cookie=cdf444f707e9b825 in_seq=0 out_seq=0 2024-04-26T01:49:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.360+0000 7f901ea00640 10 client.25723 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f901ea00640 1 -- 172.21.15.52:0/2534094008 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd38abe320 con 0x55fd38b65db0 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f901ea00640 10 client.25723 handle_command_reply: tid=0 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 client.25723 shutdown 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 msgr2=0x55fd38b5b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9030009be0 tx=0x7f903002a750 comp rx=0 tx=0).stop 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 shutdown_connections 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fd38b65db0 0x55fd38bcde70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd38b55b90 0x55fd38b5b5b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fd38b4b2d0 0x55fd38b5b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 --2- 172.21.15.52:0/2534094008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fd38b4a9b0 0x55fd38b5ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f901ea00640 0 client.25723 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 >> 172.21.15.52:0/2534094008 conn(0x55fd38b46ec0 msgr2=0x55fd38b46020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 shutdown_connections 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.394+0000 7f9042e5d740 1 -- 172.21.15.52:0/2534094008 wait complete. 2024-04-26T01:49:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.395+0000 7f9042e5d740 20 client.25723 trim_cache size 0 max 16384 2024-04-26T01:49:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:49:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:49:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:49:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:49:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.395+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 msgr2=0x7f90200a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.395+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 0x7f90200a63f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9030006010 tx=0x7f9030009300 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.395+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 msgr2=0x7f90381010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.395+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f902400e9f0 tx=0x7f902400edf0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 shutdown_connections 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90200a3f30 0x7f90200a63f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f903810c330 0x7f90381015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9038103830 0x7f90381010a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 --2- 172.21.15.52:0/4031136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9038102f10 0x7f9038100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 >> 172.21.15.52:0/4031136 conn(0x7f90380fc770 msgr2=0x7f90380fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 shutdown_connections 2024-04-26T01:49:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:49:50.396+0000 7f903ea00640 1 -- 172.21.15.52:0/4031136 wait complete. 2024-04-26T01:49:51.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 93070 3.01 MB/sec execute 70 sec latency 26.775 ms 2024-04-26T01:49:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:51 smithi052 ceph-mon[27851]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:51 smithi191 ceph-mon[32745]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:51.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:51 smithi183 ceph-mon[32752]: pgmap v149: 97 pgs: 97 active+clean; 52 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-26T01:49:52.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 94185 3.04 MB/sec execute 71 sec latency 26.611 ms 2024-04-26T01:49:53.083 INFO:tasks.workunit.client.0.smithi052.stdout: 1 95822 3.16 MB/sec execute 72 sec latency 27.983 ms 2024-04-26T01:49:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:53 smithi191 ceph-mon[32745]: pgmap v150: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T01:49:53.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:53 smithi183 ceph-mon[32752]: pgmap v150: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T01:49:53.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:53 smithi052 ceph-mon[27851]: pgmap v150: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T01:49:54.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 96229 3.14 MB/sec execute 73 sec latency 21.866 ms 2024-04-26T01:49:55.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 96952 3.13 MB/sec execute 74 sec latency 22.144 ms 2024-04-26T01:49:55.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:55 smithi191 ceph-mon[32745]: pgmap v151: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T01:49:55.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:55 smithi183 ceph-mon[32752]: pgmap v151: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T01:49:55.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:55 smithi052 ceph-mon[27851]: pgmap v151: 97 pgs: 97 active+clean; 53 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T01:49:56.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 98319 3.16 MB/sec execute 75 sec latency 56.187 ms 2024-04-26T01:49:57.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 99581 3.27 MB/sec execute 76 sec latency 34.703 ms 2024-04-26T01:49:57.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:57 smithi191 ceph-mon[32745]: pgmap v152: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-04-26T01:49:57.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:57 smithi183 ceph-mon[32752]: pgmap v152: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-04-26T01:49:57.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:57 smithi052 ceph-mon[27851]: pgmap v152: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-04-26T01:49:58.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 99833 3.23 MB/sec execute 77 sec latency 23.537 ms 2024-04-26T01:49:59.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 100746 3.25 MB/sec execute 78 sec latency 23.280 ms 2024-04-26T01:49:59.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:49:59 smithi191 ceph-mon[32745]: pgmap v153: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:49:59.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:49:59 smithi183 ceph-mon[32752]: pgmap v153: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:49:59.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:49:59 smithi052 ceph-mon[27851]: pgmap v153: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:50:00.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 102289 3.31 MB/sec execute 79 sec latency 35.632 ms 2024-04-26T01:50:00.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:00 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T01:50:00.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:00 smithi183 ceph-mon[32752]: overall HEALTH_OK 2024-04-26T01:50:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:00 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T01:50:01.084 INFO:tasks.workunit.client.0.smithi052.stdout: 1 102988 3.33 MB/sec execute 80 sec latency 95.136 ms 2024-04-26T01:50:01.803 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:01 smithi191 ceph-mon[32745]: pgmap v154: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:50:01.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:01 smithi183 ceph-mon[32752]: pgmap v154: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:50:01.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:01 smithi052 ceph-mon[27851]: pgmap v154: 97 pgs: 97 active+clean; 54 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-26T01:50:02.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 103331 3.31 MB/sec execute 81 sec latency 33.287 ms 2024-04-26T01:50:03.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 104117 3.29 MB/sec execute 82 sec latency 24.790 ms 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.555+0000 7fbd20c00640 1 -- 172.21.15.52:0/1818312766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c102b80 msgr2=0x7fbd1c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.555+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1818312766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c102b80 0x7fbd1c106bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009860 tx=0x7fbd0802a340 comp rx=0 tx=0).stop 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 -- 172.21.15.52:0/1818312766 shutdown_connections 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1818312766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1818312766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c102b80 0x7fbd1c106bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1818312766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 -- 172.21.15.52:0/1818312766 >> 172.21.15.52:0/1818312766 conn(0x7fbd1c0793e0 msgr2=0x7fbd1c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 -- 172.21.15.52:0/1818312766 shutdown_connections 2024-04-26T01:50:03.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.556+0000 7fbd20c00640 1 -- 172.21.15.52:0/1818312766 wait complete. 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.557+0000 7fbd20c00640 1 Processor -- start 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.557+0000 7fbd20c00640 1 -- start start 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.557+0000 7fbd20c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd20c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd20c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c19a720 0x7fbd1c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd20c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1121e0 con 0x7fbd1c19a720 2024-04-26T01:50:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd20c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c112060 con 0x7fbd1c1021d0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd20c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c112360 con 0x7fbd1c1072a0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36500/0 (socket says 172.21.15.52:36500) 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19e00640 1 -- 172.21.15.52:0/1774460478 learned_addr learned my addr 172.21.15.52:0/1774460478 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd1a800640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c19a720 0x7fbd1c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 -- 172.21.15.52:0/1774460478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 msgr2=0x7fbd1c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 -- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c19a720 msgr2=0x7fbd1c19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c19a720 0x7fbd1c19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19400640 1 -- 172.21.15.52:0/1774460478 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c19b320 con 0x7fbd1c1072a0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.558+0000 7fbd19e00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd19400640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c1a0ba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd0802a860 tx=0x7fbd08032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd12a00640 1 -- 172.21.15.52:0/1774460478 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08037040 con 0x7fbd1c1072a0 2024-04-26T01:50:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1c19b5b0 con 0x7fbd1c1072a0 2024-04-26T01:50:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd12a00640 1 -- 172.21.15.52:0/1774460478 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd08030c10 con 0x7fbd1c1072a0 2024-04-26T01:50:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1c1a76e0 con 0x7fbd1c1072a0 2024-04-26T01:50:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.559+0000 7fbd12a00640 1 -- 172.21.15.52:0/1774460478 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd080035f0 con 0x7fbd1c1072a0 2024-04-26T01:50:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd12a00640 1 -- 172.21.15.52:0/1774460478 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbd08043020 con 0x7fbd1c1072a0 2024-04-26T01:50:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 Processor -- start 2024-04-26T01:50:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 -- start start 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558af1868900 con 0x558af1903140 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558af180eef0 con 0x558af1902820 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.563+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558af18648c0 con 0x558af190da00 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36506/0 (socket says 172.21.15.52:36506) 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 -- 172.21.15.52:0/3308537299 learned_addr learned my addr 172.21.15.52:0/3308537299 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd19400640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd19e00640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 -- 172.21.15.52:0/3308537299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 msgr2=0x558af190fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 -- 172.21.15.52:0/3308537299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 msgr2=0x558af190d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 -- 172.21.15.52:0/3308537299 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af157b210 con 0x558af1902820 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd19e00640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.564+0000 7fbd1a800640 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009f40 tx=0x7fbd140118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=391655706e2e01e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd12a00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 0x7fbd000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd07000640 1 -- 172.21.15.52:0/3308537299 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1401e070 con 0x558af1902820 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd07000640 1 -- 172.21.15.52:0/3308537299 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd14003c30 con 0x558af1902820 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd07000640 1 -- 172.21.15.52:0/3308537299 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400f920 con 0x558af1902820 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd12a00640 1 -- 172.21.15.52:0/1774460478 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbd080efa30 con 0x7fbd1c1072a0 2024-04-26T01:50:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd19e00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 0x7fbd000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3308537299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 msgr2=0x558af1902c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.565+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009f40 tx=0x7fbd140118d0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3308537299 shutdown_connections 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3308537299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af1902c00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3308537299 >> 172.21.15.52:0/3308537299 conn(0x558af18fed30 msgr2=0x558af18ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3308537299 shutdown_connections 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd19e00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 0x7fbd000a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x558af1903dc0 tx=0x7fbd0c009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3308537299 wait complete. 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.566+0000 7fbd23a1a740 1 -- start start 2024-04-26T01:50:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 Processor -- start 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af190aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558af1910630 con 0x558af1903140 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x558af1822310 con 0x558af1902820 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558af19107b0 con 0x558af190da00 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.567+0000 7fbd23a1a740 10 client.0 _ll_register_callbacks cb 0x558af17f0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd1a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60374/0 (socket says 172.21.15.52:60374) 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 -- 172.21.15.52:0/3255462224 learned_addr learned my addr 172.21.15.52:0/3255462224 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19e00640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af190aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 -- 172.21.15.52:0/3255462224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 msgr2=0x558af190b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 -- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 msgr2=0x558af190aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af190aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 -- 172.21.15.52:0/3255462224 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af1647620 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd1a800640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbd19400640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbd08030a80 tx=0x7fbd080b7d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df50cbfae5bfdd1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.568+0000 7fbcfb400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd080bb410 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbd080dc0d0 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd14003c30 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbd08030450 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbd23a1a740 10 client.15630 fetch_fsmap learned FSMap version 35 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558af191c950 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd14003c30 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fbd080ed460 con 0x558af1903140 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbd23a1a740 10 client.15630 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:50:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.569+0000 7fbd23a1a740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd23a1a740 20 client.15630 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af191de80 0x558af19202c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd23a1a740 4 client.15630 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558af187c8b0 con 0x558af191de80 2024-04-26T01:50:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd19e00640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af191de80 0x558af19202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbd19e00640 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af191de80 0x558af19202c0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28252a709f76827f server_cookie=1e2a86b17b382467 in_seq=0 out_seq=0 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.570+0000 7fbcfb400640 10 client.15630 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.571+0000 7fbcfb400640 1 -- 172.21.15.52:0/3255462224 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558af187c8b0 con 0x558af191de80 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.571+0000 7fbcfb400640 10 client.15630 handle_command_reply: tid=0 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.571+0000 7fbd23a1a740 1 client.15630 shutdown 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.571+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 msgr2=0x558af190af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.571+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbd08030a80 tx=0x7fbd080b7d50 comp rx=0 tx=0).stop 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 shutdown_connections 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af191de80 0x558af19202c0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af190b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1903140 0x558af190af80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/3255462224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af190aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbcfb400640 0 client.15630 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 >> 172.21.15.52:0/3255462224 conn(0x558af18fed30 msgr2=0x558af1904ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 shutdown_connections 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 1 -- 172.21.15.52:0/3255462224 wait complete. 2024-04-26T01:50:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.572+0000 7fbd23a1a740 20 client.15630 trim_cache size 0 max 16384 2024-04-26T01:50:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.589+0000 7fbd23a1a740 1 Processor -- start 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.589+0000 7fbd23a1a740 1 -- start start 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af19202a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 0x558af19194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 0x558af18fde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1b24f0 con 0x558af190da00 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1afd60 con 0x558af1902820 2024-04-26T01:50:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1adbb0 con 0x558af1903140 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 0x558af18fde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd19e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af19202a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd19400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 0x558af19194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 0x558af18fde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60378/0 (socket says 172.21.15.52:60378) 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.590+0000 7fbd1a800640 1 -- 172.21.15.52:0/2990292527 learned_addr learned my addr 172.21.15.52:0/2990292527 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 -- 172.21.15.52:0/2990292527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 msgr2=0x558af19194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 0x558af19194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 -- 172.21.15.52:0/2990292527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 msgr2=0x558af18fde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 0x558af18fde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 -- 172.21.15.52:0/2990292527 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af19099f0 con 0x558af1902820 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19400640 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 0x558af19194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.591+0000 7fbd19e00640 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af19202a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b9d60 tx=0x7fbd0c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74a1174da57f24c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.592+0000 7fbd06600640 1 -- 172.21.15.52:0/2990292527 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c02a430 con 0x558af1902820 2024-04-26T01:50:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.592+0000 7fbd06600640 1 -- 172.21.15.52:0/2990292527 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd0c005020 con 0x558af1902820 2024-04-26T01:50:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.592+0000 7fbd06600640 1 -- 172.21.15.52:0/2990292527 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c004a30 con 0x558af1902820 2024-04-26T01:50:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 -- 172.21.15.52:0/2990292527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 msgr2=0x558af19202a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af19202a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b9d60 tx=0x7fbd0c01a000 comp rx=0 tx=0).stop 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 -- 172.21.15.52:0/2990292527 shutdown_connections 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af190da00 0x558af18fde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af1903140 0x558af19194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/2990292527 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1902820 0x558af19202a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 -- 172.21.15.52:0/2990292527 >> 172.21.15.52:0/2990292527 conn(0x558af18fed30 msgr2=0x558af18ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 -- 172.21.15.52:0/2990292527 shutdown_connections 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.593+0000 7fbd23a1a740 1 -- 172.21.15.52:0/2990292527 wait complete. 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.594+0000 7fbd23a1a740 1 -- start start 2024-04-26T01:50:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.594+0000 7fbd23a1a740 1 Processor -- start 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.594+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1902820 0x558af1913370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.594+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.594+0000 7fbd23a1a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af1913df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1ae220 con 0x558af1902820 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1af690 con 0x558af1903140 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd23a1a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c1ae8f0 con 0x558af190da00 2024-04-26T01:50:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd23a1a740 10 client.0 _ll_register_callbacks cb 0x7fbd08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1902820 0x558af1913370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36548/0 (socket says 172.21.15.52:36548) 2024-04-26T01:50:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 -- 172.21.15.52:0/68714849 learned_addr learned my addr 172.21.15.52:0/68714849 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd1a800640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af1913df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 -- 172.21.15.52:0/68714849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 msgr2=0x558af1913df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af1913df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 -- 172.21.15.52:0/68714849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1902820 msgr2=0x558af1913370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1902820 0x558af1913370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd19400640 1 -- 172.21.15.52:0/68714849 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af198bc60 con 0x558af1903140 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.595+0000 7fbd1a800640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af1913df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.596+0000 7fbd19400640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b8480 tx=0x7fbd080b7e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eebd7a2d784e60ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.596+0000 7fbcfb400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.596+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd080bb410 con 0x558af1903140 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.596+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbd080328a0 con 0x558af1903140 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.596+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd0c005020 con 0x558af1903140 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbd080bb880 con 0x558af1903140 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 10 client.25733 fetch_fsmap learned FSMap version 35 2024-04-26T01:50:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558af198ce90 con 0x558af1903140 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd0c005020 con 0x558af1903140 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fbd080c0ff0 con 0x558af1903140 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 10 client.25733 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 20 client.25733 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af198d4d0 0x558af198f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 4 client.25733 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.597+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558af1876320 con 0x558af198d4d0 2024-04-26T01:50:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.598+0000 7fbd19e00640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af198d4d0 0x558af198f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.598+0000 7fbd19e00640 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af198d4d0 0x558af198f8f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a86b017170c50c0 server_cookie=494f3700ba0f4ba in_seq=0 out_seq=0 2024-04-26T01:50:03.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.598+0000 7fbcfb400640 10 client.25733 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbcfb400640 1 -- 172.21.15.52:0/68714849 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558af1876320 con 0x558af198d4d0 2024-04-26T01:50:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbcfb400640 10 client.25733 handle_command_reply: tid=0 2024-04-26T01:50:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 client.25733 shutdown 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 msgr2=0x558af19138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd080b8480 tx=0x7fbd080b7e80 comp rx=0 tx=0).stop 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 shutdown_connections 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x558af198d4d0 0x558af198f8f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558af190da00 0x558af1913df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x558af1903140 0x558af19138b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 --2- 172.21.15.52:0/68714849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558af1902820 0x558af1913370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 >> 172.21.15.52:0/68714849 conn(0x558af18fed30 msgr2=0x558af190b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbcfb400640 0 client.25733 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 shutdown_connections 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 1 -- 172.21.15.52:0/68714849 wait complete. 2024-04-26T01:50:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.633+0000 7fbd23a1a740 20 client.25733 trim_cache size 0 max 16384 2024-04-26T01:50:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:50:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:50:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:50:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:50:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 msgr2=0x7fbd000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 0x7fbd000a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x558af1903dc0 tx=0x7fbd0c009260 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 msgr2=0x7fbd1c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c1a0ba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd0802a860 tx=0x7fbd08032d50 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 shutdown_connections 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbd000a3e40 0x7fbd000a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd1c19a720 0x7fbd1c19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd1c1072a0 0x7fbd1c1a0ba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 --2- 172.21.15.52:0/1774460478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbd1c1021d0 0x7fbd1c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.634+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 >> 172.21.15.52:0/1774460478 conn(0x7fbd1c0793e0 msgr2=0x7fbd1c0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.635+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 shutdown_connections 2024-04-26T01:50:03.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:03.635+0000 7fbd20c00640 1 -- 172.21.15.52:0/1774460478 wait complete. 2024-04-26T01:50:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:03 smithi191 ceph-mon[32745]: pgmap v155: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 242 op/s 2024-04-26T01:50:03.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:03 smithi183 ceph-mon[32752]: pgmap v155: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 242 op/s 2024-04-26T01:50:03.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:03 smithi052 ceph-mon[27851]: pgmap v155: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 242 op/s 2024-04-26T01:50:04.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 105672 3.36 MB/sec execute 83 sec latency 64.225 ms 2024-04-26T01:50:05.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 106680 3.41 MB/sec execute 84 sec latency 25.281 ms 2024-04-26T01:50:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:05 smithi191 ceph-mon[32745]: pgmap v156: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T01:50:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:05 smithi183 ceph-mon[32752]: pgmap v156: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T01:50:05.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:05 smithi052 ceph-mon[27851]: pgmap v156: 97 pgs: 97 active+clean; 61 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 160 op/s 2024-04-26T01:50:06.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 107129 3.38 MB/sec execute 85 sec latency 26.156 ms 2024-04-26T01:50:06.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:06 smithi191 ceph-mon[32745]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:06.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:06 smithi183 ceph-mon[32752]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:06.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:06 smithi052 ceph-mon[27851]: pgmap v157: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2024-04-26T01:50:07.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 108118 3.40 MB/sec execute 86 sec latency 29.242 ms 2024-04-26T01:50:08.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 109793 3.46 MB/sec execute 87 sec latency 29.022 ms 2024-04-26T01:50:09.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110343 3.48 MB/sec execute 88 sec latency 23.946 ms 2024-04-26T01:50:09.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:09 smithi191 ceph-mon[32745]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:09.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:09 smithi183 ceph-mon[32752]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:09 smithi052 ceph-mon[27851]: pgmap v158: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:10.085 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.44 MB/sec execute 89 sec latency 186.701 ms 2024-04-26T01:50:11.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.40 MB/sec execute 90 sec latency 1186.892 ms 2024-04-26T01:50:11.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:11 smithi191 ceph-mon[32745]: pgmap v159: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:11.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:11 smithi183 ceph-mon[32752]: pgmap v159: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:11.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:11 smithi052 ceph-mon[27851]: pgmap v159: 97 pgs: 97 active+clean; 57 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 177 op/s 2024-04-26T01:50:12.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.36 MB/sec execute 91 sec latency 2186.990 ms 2024-04-26T01:50:13.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.33 MB/sec execute 92 sec latency 3187.123 ms 2024-04-26T01:50:13.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:13 smithi191 ceph-mon[32745]: pgmap v160: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T01:50:13.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:13 smithi183 ceph-mon[32752]: pgmap v160: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T01:50:13.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:13 smithi052 ceph-mon[27851]: pgmap v160: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 227 op/s 2024-04-26T01:50:14.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.29 MB/sec execute 93 sec latency 4187.257 ms 2024-04-26T01:50:15.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.26 MB/sec execute 94 sec latency 5187.401 ms 2024-04-26T01:50:15.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:15 smithi191 ceph-mon[32745]: pgmap v161: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-26T01:50:15.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:15 smithi183 ceph-mon[32752]: pgmap v161: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-26T01:50:15.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:15 smithi052 ceph-mon[27851]: pgmap v161: 97 pgs: 97 active+clean; 62 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 134 op/s 2024-04-26T01:50:16.086 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.22 MB/sec execute 95 sec latency 6187.557 ms 2024-04-26T01:50:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 -- 172.21.15.52:0/3370933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11481021d0 msgr2=0x7f11481025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 --2- 172.21.15.52:0/3370933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11481021d0 0x7f11481025b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1138009900 tx=0x7f113802a3f0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 -- 172.21.15.52:0/3370933503 shutdown_connections 2024-04-26T01:50:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 --2- 172.21.15.52:0/3370933503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f114810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 --2- 172.21.15.52:0/3370933503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1148102b80 0x7f1148106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 --2- 172.21.15.52:0/3370933503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11481021d0 0x7f11481025b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 -- 172.21.15.52:0/3370933503 >> 172.21.15.52:0/3370933503 conn(0x7f11480793e0 msgr2=0x7f11480797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 -- 172.21.15.52:0/3370933503 shutdown_connections 2024-04-26T01:50:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.773+0000 7f114ce00640 1 -- 172.21.15.52:0/3370933503 wait complete. 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.774+0000 7f114ce00640 1 Processor -- start 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.774+0000 7f114ce00640 1 -- start start 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f11481a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f114819a680 0x7f114819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1148112250 con 0x7f11481072a0 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f11481120d0 con 0x7f114819a680 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f114ce00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f11481123d0 con 0x7f11481021d0 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f11481a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45316/0 (socket says 172.21.15.52:45316) 2024-04-26T01:50:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 -- 172.21.15.52:0/2463262289 learned_addr learned my addr 172.21.15.52:0/2463262289 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1146800640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f114819a680 0x7f114819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 -- 172.21.15.52:0/2463262289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f114819a680 msgr2=0x7f114819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f114819a680 0x7f114819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 -- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 msgr2=0x7f11481a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.775+0000 7f1145e00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f11481a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f1145e00640 1 -- 172.21.15.52:0/2463262289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114819b280 con 0x7f11481021d0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f1145400640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f11481a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f1145e00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1138005ab0 tx=0x7f1138003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f113ea00640 1 -- 172.21.15.52:0/2463262289 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113803f070 con 0x7f11481021d0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114819b510 con 0x7f11481021d0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f113ea00640 1 -- 172.21.15.52:0/2463262289 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1138031d60 con 0x7f11481021d0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f113ea00640 1 -- 172.21.15.52:0/2463262289 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1138035030 con 0x7f11481021d0 2024-04-26T01:50:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.776+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11481a76a0 con 0x7f11481021d0 2024-04-26T01:50:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f113ea00640 1 -- 172.21.15.52:0/2463262289 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1138042070 con 0x7f11481021d0 2024-04-26T01:50:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 Processor -- start 2024-04-26T01:50:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 -- start start 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb17df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb7ba70900 con 0x55bb7bb15a00 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bb7ba16ef0 con 0x55bb7bb0a820 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.780+0000 7f114fc7e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bb7ba6c8c0 con 0x55bb7bb0b140 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48306/0 (socket says 172.21.15.52:48306) 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 -- 172.21.15.52:0/1445482074 learned_addr learned my addr 172.21.15.52:0/1445482074 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1145e00640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1145400640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb17df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 -- 172.21.15.52:0/1445482074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 msgr2=0x55bb7bb154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb154c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 -- 172.21.15.52:0/1445482074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 msgr2=0x55bb7bb17df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb17df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 -- 172.21.15.52:0/1445482074 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb7b783210 con 0x55bb7bb0a820 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1145400640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f1146800640 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1140007eb0 tx=0x7f114000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7485fb0e3fe519 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.781+0000 7f113ea00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 0x7f112c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.782+0000 7f113ea00640 1 -- 172.21.15.52:0/2463262289 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f11380f0fc0 con 0x7f11481021d0 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.782+0000 7f1133400640 1 -- 172.21.15.52:0/1445482074 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140010840 con 0x55bb7bb0a820 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.782+0000 7f1133400640 1 -- 172.21.15.52:0/1445482074 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11400183f0 con 0x55bb7bb0a820 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.782+0000 7f1133400640 1 -- 172.21.15.52:0/1445482074 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140017600 con 0x55bb7bb0a820 2024-04-26T01:50:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.782+0000 7f1145e00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 0x7f112c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- 172.21.15.52:0/1445482074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 msgr2=0x55bb7bb0ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1140007eb0 tx=0x7f114000f790 comp rx=0 tx=0).stop 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- 172.21.15.52:0/1445482074 shutdown_connections 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb17df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1445482074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0a820 0x55bb7bb0ac00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- 172.21.15.52:0/1445482074 >> 172.21.15.52:0/1445482074 conn(0x55bb7bb06d30 msgr2=0x55bb7bb07110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- 172.21.15.52:0/1445482074 shutdown_connections 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f1145e00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 0x7f112c0a6280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55bb7bb0bdc0 tx=0x7f1138009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- 172.21.15.52:0/1445482074 wait complete. 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.783+0000 7f114fc7e740 1 -- start start 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 Processor -- start 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb13470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb7bb18630 con 0x55bb7bb15a00 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bb7ba2a310 con 0x55bb7bb0b140 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bb7bb187b0 con 0x55bb7bb0a820 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.784+0000 7f114fc7e740 10 client.0 _ll_register_callbacks cb 0x55bb7b9f8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48310/0 (socket says 172.21.15.52:48310) 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 -- 172.21.15.52:0/3593685798 learned_addr learned my addr 172.21.15.52:0/3593685798 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1146800640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb13470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1146800640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 -- 172.21.15.52:0/3593685798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 msgr2=0x55bb7bb129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 -- 172.21.15.52:0/3593685798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 msgr2=0x55bb7bb13470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb13470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.785+0000 7f1145400640 1 -- 172.21.15.52:0/3593685798 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb7b84f620 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1145400640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11340139d0 tx=0x7f1134013e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec136c242e0c384 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1127e00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134020060 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f1134004050 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11400183f0 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1134004250 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f114fc7e740 10 client.25745 fetch_fsmap learned FSMap version 35 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bb7bb248c0 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.786+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11400183f0 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f113400f430 con 0x55bb7bb0b140 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 10 client.25745 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 20 client.25745 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb25df0 0x55bb7bb28200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 4 client.25745 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb7ba848b0 con 0x55bb7bb25df0 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.787+0000 7f1146800640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb25df0 0x55bb7bb28200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.788+0000 7f1146800640 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb25df0 0x55bb7bb28200 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b42f4996487078d server_cookie=6eb3ba11782c775b in_seq=0 out_seq=0 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.788+0000 7f1127e00640 10 client.25745 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f1127e00640 1 -- 172.21.15.52:0/3593685798 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb7ba848b0 con 0x55bb7bb25df0 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f1127e00640 10 client.25745 handle_command_reply: tid=0 2024-04-26T01:50:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 client.25745 shutdown 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 msgr2=0x55bb7bb12f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11340139d0 tx=0x7f1134013e90 comp rx=0 tx=0).stop 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 shutdown_connections 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb25df0 0x55bb7bb28200 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb15a00 0x55bb7bb13470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb0b140 0x55bb7bb12f30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 --2- 172.21.15.52:0/3593685798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f1127e00640 0 client.25745 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 >> 172.21.15.52:0/3593685798 conn(0x55bb7bb06d30 msgr2=0x55bb7bb0ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 shutdown_connections 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 1 -- 172.21.15.52:0/3593685798 wait complete. 2024-04-26T01:50:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.789+0000 7f114fc7e740 20 client.25745 trim_cache size 0 max 16384 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 Processor -- start 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 -- start start 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0a820 0x55bb7bb214f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb2db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b2510 con 0x55bb7bb0a820 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f11481afd80 con 0x55bb7bb15a00 2024-04-26T01:50:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.806+0000 7f114fc7e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f11481adbd0 con 0x55bb7bb0b140 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1146800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0a820 0x55bb7bb214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45352/0 (socket says 172.21.15.52:45352) 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 -- 172.21.15.52:0/1523536301 learned_addr learned my addr 172.21.15.52:0/1523536301 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 -- 172.21.15.52:0/1523536301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 msgr2=0x55bb7bb2db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb2db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 -- 172.21.15.52:0/1523536301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0a820 msgr2=0x55bb7bb214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0a820 0x55bb7bb214f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 -- 172.21.15.52:0/1523536301 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb7bb119a0 con 0x55bb7bb0b140 2024-04-26T01:50:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.807+0000 7f1145400640 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11340139a0 tx=0x7f113400fe10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf0757d75a3eb4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.808+0000 7f1132a00640 1 -- 172.21.15.52:0/1523536301 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113400f3c0 con 0x55bb7bb0b140 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.808+0000 7f1132a00640 1 -- 172.21.15.52:0/1523536301 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f113400f560 con 0x55bb7bb0b140 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.808+0000 7f1132a00640 1 -- 172.21.15.52:0/1523536301 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11340258b0 con 0x55bb7bb0b140 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.808+0000 7f114fc7e740 1 -- 172.21.15.52:0/1523536301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 msgr2=0x55bb7bb13290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.808+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11340139a0 tx=0x7f113400fe10 comp rx=0 tx=0).stop 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 -- 172.21.15.52:0/1523536301 shutdown_connections 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb2db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0b140 0x55bb7bb13290 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 --2- 172.21.15.52:0/1523536301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0a820 0x55bb7bb214f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 -- 172.21.15.52:0/1523536301 >> 172.21.15.52:0/1523536301 conn(0x55bb7bb06d30 msgr2=0x55bb7bb0ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 -- 172.21.15.52:0/1523536301 shutdown_connections 2024-04-26T01:50:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 -- 172.21.15.52:0/1523536301 wait complete. 2024-04-26T01:50:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 -- start start 2024-04-26T01:50:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.809+0000 7f114fc7e740 1 Processor -- start 2024-04-26T01:50:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb1de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb1b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11481ae240 con 0x55bb7bb0b140 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f11481af6b0 con 0x55bb7bb15a00 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f11481ae910 con 0x55bb7bb0a820 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f114fc7e740 10 client.0 _ll_register_callbacks cb 0x7f1134013630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1146800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb1de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1145400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1145400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60926/0 (socket says 172.21.15.52:60926) 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1146800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb1de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45358/0 (socket says 172.21.15.52:45358) 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1145400640 1 -- 172.21.15.52:0/127350498 learned_addr learned my addr 172.21.15.52:0/127350498 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.810+0000 7f1146800640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb1b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 -- 172.21.15.52:0/127350498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 msgr2=0x55bb7bb1de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb1de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 -- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 msgr2=0x55bb7bb1b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb1b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 -- 172.21.15.52:0/127350498 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb7bb1c040 con 0x55bb7bb0b140 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1145400640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f113400f750 tx=0x7f113400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e062d54befbb029f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1127e00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134020040 con 0x55bb7bb0b140 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f1134005450 con 0x55bb7bb0b140 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.811+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f113400f560 con 0x55bb7bb0b140 2024-04-26T01:50:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.812+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1134005c50 con 0x55bb7bb0b140 2024-04-26T01:50:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.812+0000 7f114fc7e740 10 client.15648 fetch_fsmap learned FSMap version 35 2024-04-26T01:50:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.812+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bb7bb8c490 con 0x55bb7bb0b140 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.812+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f113400f560 con 0x55bb7bb0b140 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.812+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f1134035030 con 0x55bb7bb0b140 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 10 client.15648 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 20 client.15648 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb8cb80 0x55bb7bb8efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 4 client.15648 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb7ba7e320 con 0x55bb7bb8cb80 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.813+0000 7f1146800640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb8cb80 0x55bb7bb8efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.814+0000 7f1146800640 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb8cb80 0x55bb7bb8efa0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a10c4d45d9261c9 server_cookie=2f4fd0f384ffb4b7 in_seq=0 out_seq=0 2024-04-26T01:50:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.814+0000 7f1127e00640 10 client.15648 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.845+0000 7f1127e00640 1 -- 172.21.15.52:0/127350498 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb7ba7e320 con 0x55bb7bb8cb80 2024-04-26T01:50:16.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.845+0000 7f1127e00640 10 client.15648 handle_command_reply: tid=0 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.845+0000 7f114fc7e740 1 client.15648 shutdown 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 msgr2=0x55bb7bb1b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f113400f750 tx=0x7f113400fc20 comp rx=0 tx=0).stop 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 shutdown_connections 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb7bb8cb80 0x55bb7bb8efa0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb7bb15a00 0x55bb7bb1b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb7bb0b140 0x55bb7bb1b290 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 --2- 172.21.15.52:0/127350498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb7bb0a820 0x55bb7bb1de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f1127e00640 0 client.15648 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:16.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 >> 172.21.15.52:0/127350498 conn(0x55bb7bb06d30 msgr2=0x55bb7bb14220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 shutdown_connections 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 1 -- 172.21.15.52:0/127350498 wait complete. 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.846+0000 7f114fc7e740 20 client.15648 trim_cache size 0 max 16384 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:50:16.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 msgr2=0x7f112c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 0x7f112c0a6280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55bb7bb0bdc0 tx=0x7f1138009260 comp rx=0 tx=0).stop 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 msgr2=0x7f11481a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1138005ab0 tx=0x7f1138003c80 comp rx=0 tx=0).stop 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 shutdown_connections 2024-04-26T01:50:16.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f112c0a3dc0 0x7f112c0a6280 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f114819a680 0x7f114819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11481072a0 0x7f11481a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 --2- 172.21.15.52:0/2463262289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11481021d0 0x7f11481a05c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.847+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 >> 172.21.15.52:0/2463262289 conn(0x7f11480793e0 msgr2=0x7f11480feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.848+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 shutdown_connections 2024-04-26T01:50:16.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:16.848+0000 7f114ce00640 1 -- 172.21.15.52:0/2463262289 wait complete. 2024-04-26T01:50:17.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110596 3.19 MB/sec execute 96 sec latency 7187.684 ms 2024-04-26T01:50:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:17 smithi191 ceph-mon[32745]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-26T01:50:17.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:17 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:17.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:17 smithi183 ceph-mon[32752]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-26T01:50:17.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:17 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:17 smithi052 ceph-mon[27851]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-26T01:50:17.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:17 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:50:18.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 110806 3.16 MB/sec execute 97 sec latency 7952.358 ms 2024-04-26T01:50:18.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:18 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:18.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:18 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:18.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:18 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:50:18.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:18 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:18.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:18 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:18.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:18 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:50:18.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:18 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:50:18.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:18 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:50:18.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:18 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:50:19.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 111659 3.18 MB/sec execute 98 sec latency 30.811 ms 2024-04-26T01:50:19.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:19 smithi191 ceph-mon[32745]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:19.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:19 smithi183 ceph-mon[32752]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:19.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:19 smithi052 ceph-mon[27851]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:20.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 113193 3.23 MB/sec execute 99 sec latency 22.724 ms 2024-04-26T01:50:21.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 113868 3.25 MB/sec execute 100 sec latency 47.150 ms 2024-04-26T01:50:21.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:21 smithi191 ceph-mon[32745]: pgmap v164: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:21.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:21 smithi183 ceph-mon[32752]: pgmap v164: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:21.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:21 smithi052 ceph-mon[27851]: pgmap v164: 97 pgs: 97 active+clean; 64 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.2 MiB/s wr, 71 op/s 2024-04-26T01:50:22.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 114333 3.22 MB/sec execute 101 sec latency 31.224 ms 2024-04-26T01:50:22.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:22 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:22.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:22 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:22.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:22 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:23.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 115192 3.24 MB/sec execute 102 sec latency 37.049 ms 2024-04-26T01:50:23.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:23 smithi183 ceph-mon[32752]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 133 op/s 2024-04-26T01:50:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:23 smithi052 ceph-mon[27851]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 133 op/s 2024-04-26T01:50:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:23 smithi191 ceph-mon[32745]: pgmap v165: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 133 op/s 2024-04-26T01:50:24.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 116564 3.28 MB/sec execute 103 sec latency 22.888 ms 2024-04-26T01:50:25.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 117351 3.31 MB/sec execute 104 sec latency 32.655 ms 2024-04-26T01:50:25.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:25 smithi183 ceph-mon[32752]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-04-26T01:50:25.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:25 smithi052 ceph-mon[27851]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-04-26T01:50:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:25 smithi191 ceph-mon[32745]: pgmap v166: 97 pgs: 97 active+clean; 60 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.1 MiB/s wr, 83 op/s 2024-04-26T01:50:26.087 INFO:tasks.workunit.client.0.smithi052.stdout: 1 117648 3.28 MB/sec execute 105 sec latency 50.764 ms 2024-04-26T01:50:27.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 118543 3.29 MB/sec execute 106 sec latency 32.840 ms 2024-04-26T01:50:27.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:27 smithi183 ceph-mon[32752]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-26T01:50:27.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:27 smithi052 ceph-mon[27851]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-26T01:50:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:27 smithi191 ceph-mon[32745]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-26T01:50:28.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 119985 3.33 MB/sec execute 107 sec latency 24.584 ms 2024-04-26T01:50:29.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 120878 3.36 MB/sec execute 108 sec latency 29.327 ms 2024-04-26T01:50:29.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:29 smithi183 ceph-mon[32752]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:29 smithi052 ceph-mon[27851]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:29 smithi191 ceph-mon[32745]: pgmap v168: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 -- 172.21.15.52:0/1940812790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8288104200 msgr2=0x7f82881045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 --2- 172.21.15.52:0/1940812790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8288104200 0x7f82881045e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8270009900 tx=0x7f827002a410 comp rx=0 tx=0).stop 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 -- 172.21.15.52:0/1940812790 shutdown_connections 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 --2- 172.21.15.52:0/1940812790 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82881092d0 0x7f8288110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 --2- 172.21.15.52:0/1940812790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8288104bb0 0x7f8288108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 --2- 172.21.15.52:0/1940812790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8288104200 0x7f82881045e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 -- 172.21.15.52:0/1940812790 >> 172.21.15.52:0/1940812790 conn(0x7f82880ffac0 msgr2=0x7f8288101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 -- 172.21.15.52:0/1940812790 shutdown_connections 2024-04-26T01:50:30.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.002+0000 7f8287200640 1 -- 172.21.15.52:0/1940812790 wait complete. 2024-04-26T01:50:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.003+0000 7f8287200640 1 Processor -- start 2024-04-26T01:50:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.003+0000 7f8287200640 1 -- start start 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82881092d0 0x7f82881a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 0x7f828819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8288114280 con 0x7f8288104200 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8288114100 con 0x7f828819a620 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8287200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8288114400 con 0x7f82881092d0 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8286800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 0x7f828819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42158/0 (socket says 172.21.15.52:42158) 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8285e00640 1 -- 172.21.15.52:0/2665833363 learned_addr learned my addr 172.21.15.52:0/2665833363 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.004+0000 7f8285400640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82881092d0 0x7f82881a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 -- 172.21.15.52:0/2665833363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82881092d0 msgr2=0x7f82881a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82881092d0 0x7f82881a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 -- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 msgr2=0x7f82881a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 -- 172.21.15.52:0/2665833363 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828819b140 con 0x7f828819a620 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8285e00640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8286800640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 0x7f828819aad0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f828000da10 tx=0x7f828000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f827ea00640 1 -- 172.21.15.52:0/2665833363 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828000bd70 con 0x7f828819a620 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f827ea00640 1 -- 172.21.15.52:0/2665833363 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8280003e20 con 0x7f828819a620 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828819b430 con 0x7f828819a620 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f827ea00640 1 -- 172.21.15.52:0/2665833363 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82800106c0 con 0x7f828819a620 2024-04-26T01:50:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.005+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82881a7680 con 0x7f828819a620 2024-04-26T01:50:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.009+0000 7f828f820740 1 Processor -- start 2024-04-26T01:50:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.009+0000 7f827ea00640 1 -- 172.21.15.52:0/2665833363 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8280004270 con 0x7f828819a620 2024-04-26T01:50:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.009+0000 7f828f820740 1 -- start start 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adabd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adb6b90 0x55db3adb8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55db3ad11900 con 0x55db3adac2d0 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55db3acb7ef0 con 0x55db3adab9b0 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f828f820740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55db3ad0d8c0 con 0x55db3adb6b90 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f8285400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adabd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42172/0 (socket says 172.21.15.52:42172) 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f8285e00640 1 -- 172.21.15.52:0/3502401872 learned_addr learned my addr 172.21.15.52:0/3502401872 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.010+0000 7f8285400640 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adb6b90 0x55db3adb8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 -- 172.21.15.52:0/3502401872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adb6b90 msgr2=0x55db3adb8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adb6b90 0x55db3adb8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 -- 172.21.15.52:0/3502401872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 msgr2=0x55db3adabd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adabd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 -- 172.21.15.52:0/3502401872 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db3aa24210 con 0x55db3adac2d0 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285400640 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adabd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f827ea00640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 0x7f826c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f827ea00640 1 -- 172.21.15.52:0/2665833363 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f82800d0ad0 con 0x7f828819a620 2024-04-26T01:50:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285400640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 0x7f826c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8285e00640 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8270006010 tx=0x7f8270004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371b1f554ee08b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.011+0000 7f8277000640 1 -- 172.21.15.52:0/3502401872 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270040070 con 0x55db3adac2d0 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f8277000640 1 -- 172.21.15.52:0/3502401872 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82700047a0 con 0x55db3adac2d0 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f8277000640 1 -- 172.21.15.52:0/3502401872 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270035040 con 0x55db3adac2d0 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 -- 172.21.15.52:0/3502401872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 msgr2=0x55db3adb6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f8285400640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 0x7f826c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55db3ada92e0 tx=0x7f82780046c0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8270006010 tx=0x7f8270004560 comp rx=0 tx=0).stop 2024-04-26T01:50:30.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 -- 172.21.15.52:0/3502401872 shutdown_connections 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adb6b90 0x55db3adb8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adb6650 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 --2- 172.21.15.52:0/3502401872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adabd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.012+0000 7f828f820740 1 -- 172.21.15.52:0/3502401872 >> 172.21.15.52:0/3502401872 conn(0x55db3ada7ec0 msgr2=0x55db3ada82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.013+0000 7f828f820740 1 -- 172.21.15.52:0/3502401872 shutdown_connections 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.013+0000 7f828f820740 1 -- 172.21.15.52:0/3502401872 wait complete. 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.013+0000 7f828f820740 1 -- start start 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.013+0000 7f828f820740 1 Processor -- start 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adb3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adb6b90 0x55db3adb4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55db3adb9760 con 0x55db3adb6b90 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55db3accb310 con 0x55db3adac2d0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55db3adb98e0 con 0x55db3adab9b0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f828f820740 10 client.0 _ll_register_callbacks cb 0x55db3ac99cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adb3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57718/0 (socket says 172.21.15.52:57718) 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adb3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:56616/0 (socket says 172.21.15.52:56616) 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 -- 172.21.15.52:0/4041099647 learned_addr learned my addr 172.21.15.52:0/4041099647 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8286800640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adb6b90 0x55db3adb4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 -- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 msgr2=0x55db3adb3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adb3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 -- 172.21.15.52:0/4041099647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adb6b90 msgr2=0x55db3adb4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adb6b90 0x55db3adb4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.014+0000 7f8285e00640 1 -- 172.21.15.52:0/4041099647 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db3aaf0620 con 0x55db3adab9b0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f8285e00640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f827002aa00 tx=0x7f8270004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddcc762dbd97274c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f826b400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270040070 con 0x55db3adab9b0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f82700053f0 con 0x55db3adab9b0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82700047a0 con 0x55db3adab9b0 2024-04-26T01:50:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8270056050 con 0x55db3adab9b0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f828f820740 10 client.25768 fetch_fsmap learned FSMap version 36 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.015+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55db3adb9760 con 0x55db3adab9b0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82700047a0 con 0x55db3adab9b0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 36) ==== 8342+0+0 (secure 0 0 0) 0x7f8270031630 con 0x55db3adab9b0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 10 client.25768 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 10 client.25768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 20 client.25768 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adc6ac0 0x55db3adc8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 4 client.25768 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db3ad258b0 con 0x55db3adc6ac0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.016+0000 7f8285400640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adc6ac0 0x55db3adc8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.017+0000 7f8285400640 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adc6ac0 0x55db3adc8eb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb37dd2bb4877c8 server_cookie=ad93a5f33a9c107a in_seq=0 out_seq=0 2024-04-26T01:50:30.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.017+0000 7f826b400640 10 client.25768 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f826b400640 1 -- 172.21.15.52:0/4041099647 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db3ad258b0 con 0x55db3adc6ac0 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f826b400640 10 client.25768 handle_command_reply: tid=0 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 client.25768 shutdown 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 msgr2=0x55db3adb3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f827002aa00 tx=0x7f8270004b80 comp rx=0 tx=0).stop 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 shutdown_connections 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adc6ac0 0x55db3adc8eb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adb6b90 0x55db3adb4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adb3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f826b400640 0 client.25768 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 --2- 172.21.15.52:0/4041099647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adb3aa0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 >> 172.21.15.52:0/4041099647 conn(0x55db3ada7ec0 msgr2=0x55db3adadc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 shutdown_connections 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 1 -- 172.21.15.52:0/4041099647 wait complete. 2024-04-26T01:50:30.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.018+0000 7f828f820740 20 client.25768 trim_cache size 0 max 16384 2024-04-26T01:50:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.036+0000 7f828f820740 1 Processor -- start 2024-04-26T01:50:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.036+0000 7f828f820740 1 -- start start 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f82881b24e0 con 0x55db3ada7ec0 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82881afd80 con 0x55db3adac2d0 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f828f820740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f82881adbd0 con 0x55db3adab9b0 2024-04-26T01:50:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8285400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8285400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adce660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57732/0 (socket says 172.21.15.52:57732) 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8285e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42186/0 (socket says 172.21.15.52:42186) 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8286800640 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8285400640 1 -- 172.21.15.52:0/986962463 learned_addr learned my addr 172.21.15.52:0/986962463 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8286800640 1 -- 172.21.15.52:0/986962463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 msgr2=0x55db3adce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8286800640 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.037+0000 7f8286800640 1 -- 172.21.15.52:0/986962463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 msgr2=0x55db3adc2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8286800640 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8286800640 1 -- 172.21.15.52:0/986962463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db3adb1ff0 con 0x55db3adac2d0 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8285e00640 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8286800640 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adceba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f82800ce1f0 tx=0x7f82800bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae6a9476f4e3b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8276600640 1 -- 172.21.15.52:0/986962463 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828009d140 con 0x55db3adac2d0 2024-04-26T01:50:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8276600640 1 -- 172.21.15.52:0/986962463 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f828009ca40 con 0x55db3adac2d0 2024-04-26T01:50:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.038+0000 7f8276600640 1 -- 172.21.15.52:0/986962463 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82800ce5c0 con 0x55db3adac2d0 2024-04-26T01:50:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 -- 172.21.15.52:0/986962463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 msgr2=0x55db3adceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adceba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f82800ce1f0 tx=0x7f82800bd040 comp rx=0 tx=0).stop 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 -- 172.21.15.52:0/986962463 shutdown_connections 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adac2d0 0x55db3adceba0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3adab9b0 0x55db3adce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 --2- 172.21.15.52:0/986962463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3ada7ec0 0x55db3adc2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 -- 172.21.15.52:0/986962463 >> 172.21.15.52:0/986962463 conn(0x55db3adc9220 msgr2=0x55db3adaedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 -- 172.21.15.52:0/986962463 shutdown_connections 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.039+0000 7f828f820740 1 -- 172.21.15.52:0/986962463 wait complete. 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 -- start start 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 Processor -- start 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adbc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adbc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f82881ae240 con 0x55db3adac2d0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82881af6b0 con 0x55db3adab9b0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f82881ae910 con 0x55db3ada7ec0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.040+0000 7f828f820740 10 client.0 _ll_register_callbacks cb 0x7f8270004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adbc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57736/0 (socket says 172.21.15.52:57736) 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adbc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:56642/0 (socket says 172.21.15.52:56642) 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8286800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adbc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 -- 172.21.15.52:0/2777695672 learned_addr learned my addr 172.21.15.52:0/2777695672 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 -- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 msgr2=0x55db3adbc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adbc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 -- 172.21.15.52:0/2777695672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 msgr2=0x55db3adbc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adbc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 -- 172.21.15.52:0/2777695672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db3adbcff0 con 0x55db3ada7ec0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f8285e00640 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8270005c40 tx=0x7f827002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f5d27185c2b4a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.041+0000 7f826b400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827005f060 con 0x55db3ada7ec0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f82700051f0 con 0x55db3ada7ec0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f828009ca40 con 0x55db3ada7ec0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8270040040 con 0x55db3ada7ec0 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f828f820740 10 client.25774 fetch_fsmap learned FSMap version 36 2024-04-26T01:50:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55db3ae2cf20 con 0x55db3ada7ec0 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f828009ca40 con 0x55db3ada7ec0 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.042+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 36) ==== 8342+0+0 (secure 0 0 0) 0x7f8270005430 con 0x55db3ada7ec0 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 10 client.25774 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 20 client.25774 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adb6b90 0x55db3ae2f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 4 client.25774 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db3ad1f320 con 0x55db3adb6b90 2024-04-26T01:50:30.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f8285400640 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adb6b90 0x55db3ae2f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:30.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f8285400640 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adb6b90 0x55db3ae2f5c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0903afa37e1ac27 server_cookie=9c62725014f6a8ba in_seq=0 out_seq=0 2024-04-26T01:50:30.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.043+0000 7f826b400640 10 client.25774 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.077+0000 7f826b400640 1 -- 172.21.15.52:0/2777695672 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db3ad1f320 con 0x55db3adb6b90 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.077+0000 7f826b400640 10 client.25774 handle_command_reply: tid=0 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.077+0000 7f828f820740 1 client.25774 shutdown 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.077+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 msgr2=0x55db3adbef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.077+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8270005c40 tx=0x7f827002a410 comp rx=0 tx=0).stop 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 shutdown_connections 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55db3adb6b90 0x55db3ae2f5c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55db3adac2d0 0x55db3adbc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55db3adab9b0 0x55db3adbc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f826b400640 0 client.25774 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 --2- 172.21.15.52:0/2777695672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55db3ada7ec0 0x55db3adbef20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 >> 172.21.15.52:0/2777695672 conn(0x55db3adc9220 msgr2=0x55db3adad360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 shutdown_connections 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 1 -- 172.21.15.52:0/2777695672 wait complete. 2024-04-26T01:50:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.078+0000 7f828f820740 20 client.25774 trim_cache size 0 max 16384 2024-04-26T01:50:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:50:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:50:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:50:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 msgr2=0x7f826c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 0x7f826c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55db3ada92e0 tx=0x7f82780046c0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 msgr2=0x7f828819aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 0x7f828819aad0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f828000da10 tx=0x7f828000ded0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 shutdown_connections 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f826c0a3d90 0x7f826c0a6250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f828819a620 0x7f828819aad0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82881092d0 0x7f82881a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 --2- 172.21.15.52:0/2665833363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8288104200 0x7f82881a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 >> 172.21.15.52:0/2665833363 conn(0x7f82880ffac0 msgr2=0x7f8288100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 shutdown_connections 2024-04-26T01:50:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:30.079+0000 7f8287200640 1 -- 172.21.15.52:0/2665833363 wait complete. 2024-04-26T01:50:30.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 121235 3.33 MB/sec execute 109 sec latency 26.003 ms 2024-04-26T01:50:31.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 122163 3.35 MB/sec execute 110 sec latency 28.014 ms 2024-04-26T01:50:31.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:31 smithi183 ceph-mon[32752]: pgmap v169: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:31.453 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:31 smithi191 ceph-mon[32745]: pgmap v169: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:31.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:31 smithi052 ceph-mon[27851]: pgmap v169: 97 pgs: 97 active+clean; 61 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 965 KiB/s wr, 116 op/s 2024-04-26T01:50:32.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 123562 3.39 MB/sec execute 111 sec latency 23.497 ms 2024-04-26T01:50:33.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 124422 3.41 MB/sec execute 112 sec latency 33.238 ms 2024-04-26T01:50:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:33 smithi052 ceph-mon[27851]: pgmap v170: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T01:50:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:33 smithi191 ceph-mon[32745]: pgmap v170: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T01:50:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:33 smithi183 ceph-mon[32752]: pgmap v170: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-26T01:50:34.088 INFO:tasks.workunit.client.0.smithi052.stdout: 1 124705 3.38 MB/sec execute 113 sec latency 23.332 ms 2024-04-26T01:50:35.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 125592 3.40 MB/sec execute 114 sec latency 31.670 ms 2024-04-26T01:50:35.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:35 smithi052 ceph-mon[27851]: pgmap v171: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-04-26T01:50:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:35 smithi191 ceph-mon[32745]: pgmap v171: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-04-26T01:50:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:35 smithi183 ceph-mon[32752]: pgmap v171: 97 pgs: 97 active+clean; 60 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-04-26T01:50:36.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 126954 3.42 MB/sec execute 115 sec latency 24.444 ms 2024-04-26T01:50:37.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 127976 3.46 MB/sec execute 116 sec latency 38.866 ms 2024-04-26T01:50:37.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:37 smithi052 ceph-mon[27851]: pgmap v172: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:50:37.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:37 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:37 smithi191 ceph-mon[32745]: pgmap v172: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:50:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:37 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:37 smithi183 ceph-mon[32752]: pgmap v172: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:50:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:37 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:50:38.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 128301 3.43 MB/sec execute 117 sec latency 22.674 ms 2024-04-26T01:50:39.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 129183 3.45 MB/sec execute 118 sec latency 27.892 ms 2024-04-26T01:50:39.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:39 smithi052 ceph-mon[27851]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:39 smithi191 ceph-mon[32745]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:39 smithi183 ceph-mon[32752]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:40.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 130497 3.47 MB/sec execute 119 sec latency 21.711 ms 2024-04-26T01:50:41.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 131514 3.50 MB/sec execute 120 sec latency 31.065 ms 2024-04-26T01:50:41.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:41 smithi052 ceph-mon[27851]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:41 smithi191 ceph-mon[32745]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:41 smithi183 ceph-mon[32752]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-26T01:50:42.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 131888 3.48 MB/sec execute 121 sec latency 35.385 ms 2024-04-26T01:50:43.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 132855 3.49 MB/sec execute 122 sec latency 22.082 ms 2024-04-26T01:50:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 -- 172.21.15.52:0/2733837417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 msgr2=0x7f27d810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 --2- 172.21.15.52:0/2733837417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d810e9e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009ca0 tx=0x7f27d002a940 comp rx=0 tx=0).stop 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 -- 172.21.15.52:0/2733837417 shutdown_connections 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 --2- 172.21.15.52:0/2733837417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d810e9e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 --2- 172.21.15.52:0/2733837417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 --2- 172.21.15.52:0/2733837417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 0x7f27d81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 -- 172.21.15.52:0/2733837417 >> 172.21.15.52:0/2733837417 conn(0x7f27d8077f60 msgr2=0x7f27d8078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 -- 172.21.15.52:0/2733837417 shutdown_connections 2024-04-26T01:50:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.241+0000 7f27df600640 1 -- 172.21.15.52:0/2733837417 wait complete. 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.242+0000 7f27df600640 1 Processor -- start 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.242+0000 7f27df600640 1 -- start start 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 0x7f27d81a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d81a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81121d0 con 0x7f27d81031e0 2024-04-26T01:50:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8112050 con 0x7f27d810c5b0 2024-04-26T01:50:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27df600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8112350 con 0x7f27d8103b00 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d81a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27dd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d81a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50396/0 (socket says 172.21.15.52:50396) 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27d7e00640 1 -- 172.21.15.52:0/3416235789 learned_addr learned my addr 172.21.15.52:0/3416235789 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.243+0000 7f27dcc00640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 0x7f27d81a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 -- 172.21.15.52:0/3416235789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 msgr2=0x7f27d81a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d81a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 -- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 msgr2=0x7f27d81a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 0x7f27d81a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 -- 172.21.15.52:0/3416235789 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d819ada0 con 0x7f27d810c5b0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.244+0000 7f27dd600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d819a620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f27d0006010 tx=0x7f27d00308f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.245+0000 7f27d5600640 1 -- 172.21.15.52:0/3416235789 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0034040 con 0x7f27d810c5b0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.245+0000 7f27d5600640 1 -- 172.21.15.52:0/3416235789 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d0003e20 con 0x7f27d810c5b0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.245+0000 7f27d5600640 1 -- 172.21.15.52:0/3416235789 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0004610 con 0x7f27d810c5b0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.245+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d819b030 con 0x7f27d810c5b0 2024-04-26T01:50:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.245+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d81a7330 con 0x7f27d810c5b0 2024-04-26T01:50:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27d5600640 1 -- 172.21.15.52:0/3416235789 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f27d0037020 con 0x7f27d810c5b0 2024-04-26T01:50:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 Processor -- start 2024-04-26T01:50:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 -- start start 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a8c691a900 con 0x55a8c69b5140 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a8c68c0ef0 con 0x55a8c69bfa00 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.249+0000 7f27e23f1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a8c69168c0 con 0x55a8c69b4820 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50398/0 (socket says 172.21.15.52:50398) 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 -- 172.21.15.52:0/3390707199 learned_addr learned my addr 172.21.15.52:0/3390707199 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27dcc00640 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 -- 172.21.15.52:0/3390707199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 msgr2=0x55a8c69c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 -- 172.21.15.52:0/3390707199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 msgr2=0x55a8c69bf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 -- 172.21.15.52:0/3390707199 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8c662d210 con 0x55a8c69b4820 2024-04-26T01:50:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.250+0000 7f27d7e00640 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009640 tx=0x7f27c4009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8b33fd39d7cb9ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27c9c00640 1 -- 172.21.15.52:0/3390707199 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4013d10 con 0x55a8c69b4820 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27c9c00640 1 -- 172.21.15.52:0/3390707199 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c400fd70 con 0x55a8c69b4820 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27c9c00640 1 -- 172.21.15.52:0/3390707199 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c40047a0 con 0x55a8c69b4820 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27d5600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 0x7f27c00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27d5600640 1 -- 172.21.15.52:0/3416235789 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f27d00f1610 con 0x7f27d810c5b0 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.251+0000 7f27dcc00640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 0x7f27c00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 -- 172.21.15.52:0/3390707199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 msgr2=0x55a8c69b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009640 tx=0x7f27c4009530 comp rx=0 tx=0).stop 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 -- 172.21.15.52:0/3390707199 shutdown_connections 2024-04-26T01:50:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 --2- 172.21.15.52:0/3390707199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69b4c00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 -- 172.21.15.52:0/3390707199 >> 172.21.15.52:0/3390707199 conn(0x55a8c69b0d30 msgr2=0x55a8c69b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 -- 172.21.15.52:0/3390707199 shutdown_connections 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.252+0000 7f27e23f1740 1 -- 172.21.15.52:0/3390707199 wait complete. 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.253+0000 7f27dcc00640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 0x7f27c00a63e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55a8c69b5dc0 tx=0x7f27cc00a2d0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.253+0000 7f27e23f1740 1 -- start start 2024-04-26T01:50:43.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.253+0000 7f27e23f1740 1 Processor -- start 2024-04-26T01:50:43.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69bc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a8c69c2630 con 0x55a8c69b5140 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a8c68d4310 con 0x55a8c69bfa00 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a8c69c27b0 con 0x55a8c69b4820 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27e23f1740 10 client.0 _ll_register_callbacks cb 0x55a8c68a2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27dcc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69bc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27dd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27dd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48268/0 (socket says 172.21.15.52:48268) 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.254+0000 7f27dd600640 1 -- 172.21.15.52:0/2272473642 learned_addr learned my addr 172.21.15.52:0/2272473642 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:43.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 -- 172.21.15.52:0/2272473642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 msgr2=0x55a8c69bc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69bc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 -- 172.21.15.52:0/2272473642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 msgr2=0x55a8c69bceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 -- 172.21.15.52:0/2272473642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8c66f9620 con 0x55a8c69bfa00 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dcc00640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69bc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27dd600640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27d00b9b80 tx=0x7f27d00b9ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af4722881146891 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27be000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00bde20 con 0x55a8c69bfa00 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f27d002fb80 con 0x55a8c69bfa00 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.255+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c400fd70 con 0x55a8c69bfa00 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f27d00098b0 con 0x55a8c69bfa00 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 10 client.25778 fetch_fsmap learned FSMap version 37 2024-04-26T01:50:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a8c69ce8b0 con 0x55a8c69bfa00 2024-04-26T01:50:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c400fd70 con 0x55a8c69bfa00 2024-04-26T01:50:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f27d002f070 con 0x55a8c69bfa00 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 10 client.25778 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 10 client.25778 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 20 client.25778 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c69cfde0 0x55a8c69d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.256+0000 7f27e23f1740 4 client.25778 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.257+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8c692e8b0 con 0x55a8c69cfde0 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.257+0000 7f27dcc00640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c69cfde0 0x55a8c69d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.257+0000 7f27dcc00640 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c69cfde0 0x55a8c69d2240 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2effb64b703407d5 server_cookie=e47996d9c35aa3dd in_seq=0 out_seq=0 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.257+0000 7f27be000640 10 client.25778 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.258+0000 7f27be000640 1 -- 172.21.15.52:0/2272473642 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8c692e8b0 con 0x55a8c69cfde0 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.258+0000 7f27be000640 10 client.25778 handle_command_reply: tid=0 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 client.25778 shutdown 2024-04-26T01:50:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 msgr2=0x55a8c69bd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27d00b9b80 tx=0x7f27d00b9ab0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 shutdown_connections 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c69cfde0 0x55a8c69d2240 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c69bd3f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b5140 0x55a8c69bceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2272473642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b4820 0x55a8c69bc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 >> 172.21.15.52:0/2272473642 conn(0x55a8c69b0d30 msgr2=0x55a8c69b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27be000640 0 client.25778 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 shutdown_connections 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 1 -- 172.21.15.52:0/2272473642 wait complete. 2024-04-26T01:50:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.259+0000 7f27e23f1740 20 client.25778 trim_cache size 0 max 16384 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.276+0000 7f27e23f1740 1 Processor -- start 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.276+0000 7f27e23f1740 1 -- start start 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.276+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.276+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 0x55a8c69b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 0x55a8c69d7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27e23f1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81b2400 con 0x55a8c69b4820 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27e23f1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81afcd0 con 0x55a8c69b5140 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27e23f1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81adb20 con 0x55a8c69bfa00 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 0x55a8c69b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 0x55a8c69d7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 0x55a8c69b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48272/0 (socket says 172.21.15.52:48272) 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27d7e00640 1 -- 172.21.15.52:0/4148766506 learned_addr learned my addr 172.21.15.52:0/4148766506 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dcc00640 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 -- 172.21.15.52:0/4148766506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 msgr2=0x55a8c69b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 0x55a8c69b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 -- 172.21.15.52:0/4148766506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 msgr2=0x55a8c69cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.277+0000 7f27dd600640 1 -- 172.21.15.52:0/4148766506 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8c69ac690 con 0x55a8c69bfa00 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.278+0000 7f27dd600640 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 0x55a8c69d7b10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27d00c5dd0 tx=0x7f27d002a940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38914d44348b20a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.278+0000 7f27c9200640 1 -- 172.21.15.52:0/4148766506 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00bd260 con 0x55a8c69bfa00 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.278+0000 7f27c9200640 1 -- 172.21.15.52:0/4148766506 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d00d4060 con 0x55a8c69bfa00 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.278+0000 7f27c9200640 1 -- 172.21.15.52:0/4148766506 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00bcbc0 con 0x55a8c69bfa00 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- 172.21.15.52:0/4148766506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 msgr2=0x55a8c69d7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 0x55a8c69d7b10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27d00c5dd0 tx=0x7f27d002a940 comp rx=0 tx=0).stop 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- 172.21.15.52:0/4148766506 shutdown_connections 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69bfa00 0x55a8c69d7b10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69b5140 0x55a8c69b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 --2- 172.21.15.52:0/4148766506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- 172.21.15.52:0/4148766506 >> 172.21.15.52:0/4148766506 conn(0x55a8c69b0d30 msgr2=0x55a8c69b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- 172.21.15.52:0/4148766506 shutdown_connections 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- 172.21.15.52:0/4148766506 wait complete. 2024-04-26T01:50:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 -- start start 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.279+0000 7f27e23f1740 1 Processor -- start 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 0x55a8c69c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81ae190 con 0x55a8c69b4820 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81af600 con 0x55a8c69bfa00 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81ae860 con 0x55a8c69b5140 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27e23f1740 10 client.0 _ll_register_callbacks cb 0x7f27d00b9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27dd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 0x55a8c69c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27dcc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27dd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48282/0 (socket says 172.21.15.52:48282) 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27dd600640 1 -- 172.21.15.52:0/2504897321 learned_addr learned my addr 172.21.15.52:0/2504897321 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.280+0000 7f27d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 0x55a8c69c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50434/0 (socket says 172.21.15.52:50434) 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 -- 172.21.15.52:0/2504897321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 msgr2=0x55a8c69c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 0x55a8c69c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 -- 172.21.15.52:0/2504897321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 msgr2=0x55a8c69c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 -- 172.21.15.52:0/2504897321 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8c6a2e940 con 0x55a8c69bfa00 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27dd600640 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27d00bd420 tx=0x7f27d0004140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ccee5cd9367eb49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27be000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00ca020 con 0x55a8c69bfa00 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f27d00bcb70 con 0x55a8c69bfa00 2024-04-26T01:50:43.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d00d4060 con 0x55a8c69bfa00 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f27d00bd260 con 0x55a8c69bfa00 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27e23f1740 10 client.25784 fetch_fsmap learned FSMap version 37 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.281+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a8c6a2e940 con 0x55a8c69bfa00 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d00d4060 con 0x55a8c69bfa00 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f27d002f040 con 0x55a8c69bfa00 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 10 client.25784 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 10 client.25784 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 20 client.25784 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c6a35fd0 0x55a8c6a383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 4 client.25784 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8c6928320 con 0x55a8c6a35fd0 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.282+0000 7f27dcc00640 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c6a35fd0 0x55a8c6a383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:43.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.283+0000 7f27dcc00640 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c6a35fd0 0x55a8c6a383c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d2ad92b5b8675d5 server_cookie=427ac2898909e309 in_seq=0 out_seq=0 2024-04-26T01:50:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.283+0000 7f27be000640 10 client.25784 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.315+0000 7f27be000640 1 -- 172.21.15.52:0/2504897321 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a8c6928320 con 0x55a8c6a35fd0 2024-04-26T01:50:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.315+0000 7f27be000640 10 client.25784 handle_command_reply: tid=0 2024-04-26T01:50:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.315+0000 7f27e23f1740 1 client.25784 shutdown 2024-04-26T01:50:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 msgr2=0x55a8c6a2e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27d00bd420 tx=0x7f27d0004140 comp rx=0 tx=0).stop 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 shutdown_connections 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a8c6a35fd0 0x55a8c6a383c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a8c69bfa00 0x55a8c6a2e350 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a8c69b5140 0x55a8c69c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27be000640 0 client.25784 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 --2- 172.21.15.52:0/2504897321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a8c69b4820 0x55a8c69c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 >> 172.21.15.52:0/2504897321 conn(0x55a8c69b0d30 msgr2=0x55a8c69bc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 shutdown_connections 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 1 -- 172.21.15.52:0/2504897321 wait complete. 2024-04-26T01:50:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.316+0000 7f27e23f1740 20 client.25784 trim_cache size 0 max 16384 2024-04-26T01:50:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:50:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:50:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:50:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:50:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 msgr2=0x7f27c00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 0x7f27c00a63e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55a8c69b5dc0 tx=0x7f27cc00a2d0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 msgr2=0x7f27d819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d819a620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f27d0006010 tx=0x7f27d00308f0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 shutdown_connections 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27c00a3f20 0x7f27c00a63e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27d810c5b0 0x7f27d819a620 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27d8103b00 0x7f27d81a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 --2- 172.21.15.52:0/3416235789 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27d81031e0 0x7f27d81a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.317+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 >> 172.21.15.52:0/3416235789 conn(0x7f27d8077f60 msgr2=0x7f27d80fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.318+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 shutdown_connections 2024-04-26T01:50:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:43.318+0000 7f27df600640 1 -- 172.21.15.52:0/3416235789 wait complete. 2024-04-26T01:50:43.476 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:43 smithi183 ceph-mon[32752]: pgmap v175: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-26T01:50:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:43 smithi191 ceph-mon[32745]: pgmap v175: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-26T01:50:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:43 smithi052 ceph-mon[27851]: pgmap v175: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-04-26T01:50:44.089 INFO:tasks.workunit.client.0.smithi052.stdout: 1 134458 3.53 MB/sec execute 123 sec latency 55.764 ms 2024-04-26T01:50:45.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 135126 3.54 MB/sec execute 124 sec latency 23.803 ms 2024-04-26T01:50:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:45 smithi191 ceph-mon[32745]: pgmap v176: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-26T01:50:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:45 smithi183 ceph-mon[32752]: pgmap v176: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-26T01:50:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:45 smithi052 ceph-mon[27851]: pgmap v176: 97 pgs: 97 active+clean; 66 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-26T01:50:46.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 135611 3.52 MB/sec execute 125 sec latency 35.574 ms 2024-04-26T01:50:47.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 136479 3.53 MB/sec execute 126 sec latency 28.953 ms 2024-04-26T01:50:47.514 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:47 smithi052 ceph-mon[27851]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-04-26T01:50:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:47 smithi191 ceph-mon[32745]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-04-26T01:50:47.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:47 smithi183 ceph-mon[32752]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 196 op/s 2024-04-26T01:50:48.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 137842 3.56 MB/sec execute 127 sec latency 21.927 ms 2024-04-26T01:50:49.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 138647 3.58 MB/sec execute 128 sec latency 30.487 ms 2024-04-26T01:50:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:49 smithi191 ceph-mon[32745]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:49.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:49 smithi183 ceph-mon[32752]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:49 smithi052 ceph-mon[27851]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:50.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 138938 3.56 MB/sec execute 129 sec latency 23.481 ms 2024-04-26T01:50:51.090 INFO:tasks.workunit.client.0.smithi052.stdout: 1 139768 3.57 MB/sec execute 130 sec latency 28.855 ms 2024-04-26T01:50:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:51 smithi191 ceph-mon[32745]: pgmap v179: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:51.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:51 smithi183 ceph-mon[32752]: pgmap v179: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:51 smithi052 ceph-mon[27851]: pgmap v179: 97 pgs: 97 active+clean; 65 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-26T01:50:52.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 140765 3.56 MB/sec execute 131 sec latency 35.877 ms 2024-04-26T01:50:53.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 141984 3.61 MB/sec execute 132 sec latency 39.521 ms 2024-04-26T01:50:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:53 smithi191 ceph-mon[32745]: pgmap v180: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T01:50:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:53 smithi183 ceph-mon[32752]: pgmap v180: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T01:50:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:53 smithi052 ceph-mon[27851]: pgmap v180: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T01:50:54.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 142302 3.59 MB/sec execute 133 sec latency 27.365 ms 2024-04-26T01:50:55.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 142835 3.57 MB/sec execute 134 sec latency 28.517 ms 2024-04-26T01:50:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:55 smithi191 ceph-mon[32745]: pgmap v181: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 KiB/s wr, 123 op/s 2024-04-26T01:50:55.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:55 smithi183 ceph-mon[32752]: pgmap v181: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 KiB/s wr, 123 op/s 2024-04-26T01:50:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:55 smithi052 ceph-mon[27851]: pgmap v181: 97 pgs: 97 active+clean; 68 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 KiB/s wr, 123 op/s 2024-04-26T01:50:56.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 143570 3.58 MB/sec execute 135 sec latency 35.970 ms 2024-04-26T01:50:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 -- 172.21.15.52:0/408316453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca41092d0 msgr2=0x7feca4110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 --2- 172.21.15.52:0/408316453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca41092d0 0x7feca4110ea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009be0 tx=0x7fec9c02a6c0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 -- 172.21.15.52:0/408316453 shutdown_connections 2024-04-26T01:50:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 --2- 172.21.15.52:0/408316453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca41092d0 0x7feca4110ea0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 --2- 172.21.15.52:0/408316453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca4104bb0 0x7feca4108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 --2- 172.21.15.52:0/408316453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca4104200 0x7feca41045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 -- 172.21.15.52:0/408316453 >> 172.21.15.52:0/408316453 conn(0x7feca40ffa60 msgr2=0x7feca4101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 -- 172.21.15.52:0/408316453 shutdown_connections 2024-04-26T01:50:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.469+0000 7feca9000640 1 -- 172.21.15.52:0/408316453 wait complete. 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.470+0000 7feca9000640 1 Processor -- start 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.470+0000 7feca9000640 1 -- start start 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca41092d0 0x7feca419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca419f020 0x7feca419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feca4114220 con 0x7feca41092d0 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feca41140a0 con 0x7feca4104200 2024-04-26T01:50:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca9000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feca41143a0 con 0x7feca419f020 2024-04-26T01:50:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca2a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca41092d0 0x7feca419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53006/0 (socket says 172.21.15.52:53006) 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca3400640 1 -- 172.21.15.52:0/1010487531 learned_addr learned my addr 172.21.15.52:0/1010487531 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.471+0000 7feca3e00640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca419f020 0x7feca419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 -- 172.21.15.52:0/1010487531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca419f020 msgr2=0x7feca419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca419f020 0x7feca419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 -- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca41092d0 msgr2=0x7feca419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca41092d0 0x7feca419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 -- 172.21.15.52:0/1010487531 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca419fb10 con 0x7feca4104200 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7feca3400640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e9c0 tx=0x7fec9400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7fec9be00640 1 -- 172.21.15.52:0/1010487531 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400cde0 con 0x7feca4104200 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7fec9be00640 1 -- 172.21.15.52:0/1010487531 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec940035f0 con 0x7feca4104200 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.472+0000 7fec9be00640 1 -- 172.21.15.52:0/1010487531 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94010690 con 0x7feca4104200 2024-04-26T01:50:56.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.473+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca419fe00 con 0x7feca4104200 2024-04-26T01:50:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.473+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41a75a0 con 0x7feca4104200 2024-04-26T01:50:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.476+0000 7fec9be00640 1 -- 172.21.15.52:0/1010487531 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fec94004000 con 0x7feca4104200 2024-04-26T01:50:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.476+0000 7fecad5e9740 1 Processor -- start 2024-04-26T01:50:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 -- start start 2024-04-26T01:50:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ecfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6eda650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6e35900 con 0x55e2c6ed02d0 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6ddbef0 con 0x55e2c6edab90 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7fecad5e9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6e318c0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7feca2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7feca2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53016/0 (socket says 172.21.15.52:53016) 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7feca2a00640 1 -- 172.21.15.52:0/4031791598 learned_addr learned my addr 172.21.15.52:0/4031791598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7feca3e00640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ecfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.477+0000 7feca3400640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6eda650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 -- 172.21.15.52:0/4031791598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ecf9b0 msgr2=0x55e2c6ecfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ecfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 -- 172.21.15.52:0/4031791598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 msgr2=0x55e2c6eda650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6eda650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 -- 172.21.15.52:0/4031791598 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c6b48210 con 0x55e2c6edab90 2024-04-26T01:50:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca3400640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6eda650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7fec9be00640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 0x7fec880a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca2a00640 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec90009c20 tx=0x7fec90011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f03e8f5576dd9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7fec9be00640 1 -- 172.21.15.52:0/1010487531 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fec940d0ad0 con 0x7feca4104200 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7feca3e00640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 0x7fec880a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.478+0000 7fec98c00640 1 -- 172.21.15.52:0/4031791598 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004050 con 0x55e2c6edab90 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fec98c00640 1 -- 172.21.15.52:0/4031791598 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec900054d0 con 0x55e2c6edab90 2024-04-26T01:50:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fec98c00640 1 -- 172.21.15.52:0/4031791598 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004d00 con 0x55e2c6edab90 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7feca3e00640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 0x7fec880a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fec9c004af0 tx=0x7fec9c005fd0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 -- 172.21.15.52:0/4031791598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 msgr2=0x55e2c6edcf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec90009c20 tx=0x7fec90011810 comp rx=0 tx=0).stop 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 -- 172.21.15.52:0/4031791598 shutdown_connections 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6edab90 0x55e2c6edcf80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6eda650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 --2- 172.21.15.52:0/4031791598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ecfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.479+0000 7fecad5e9740 1 -- 172.21.15.52:0/4031791598 >> 172.21.15.52:0/4031791598 conn(0x55e2c6ecbec0 msgr2=0x55e2c6ecc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.480+0000 7fecad5e9740 1 -- 172.21.15.52:0/4031791598 shutdown_connections 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.480+0000 7fecad5e9740 1 -- 172.21.15.52:0/4031791598 wait complete. 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.480+0000 7fecad5e9740 1 -- start start 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.480+0000 7fecad5e9740 1 Processor -- start 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ed85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6edd7c0 con 0x55e2c6ed02d0 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6def310 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e2c6edd940 con 0x55e2c6edab90 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7fecad5e9740 10 client.0 _ll_register_callbacks cb 0x55e2c6dbdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ed85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53022/0 (socket says 172.21.15.52:53022) 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 -- 172.21.15.52:0/302324096 learned_addr learned my addr 172.21.15.52:0/302324096 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca2a00640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 -- 172.21.15.52:0/302324096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 msgr2=0x55e2c6ed85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ed85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 -- 172.21.15.52:0/302324096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 msgr2=0x55e2c6ed8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.481+0000 7feca3400640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7feca3400640 1 -- 172.21.15.52:0/302324096 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c6c14620 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7feca3400640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c6a0 tx=0x7fec9409c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea324bdb6f7f521 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fec8d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9409d140 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fec940037a0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec900054d0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fec9409ca40 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fecad5e9740 10 client.25796 fetch_fsmap learned FSMap version 37 2024-04-26T01:50:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.482+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e2c6ee9aa0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec900054d0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fec94098790 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 10 client.25796 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 10 client.25796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 20 client.25796 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6eeaf10 0x55e2c6eed320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 4 client.25796 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2c6e498b0 con 0x55e2c6eeaf10 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.483+0000 7feca2a00640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6eeaf10 0x55e2c6eed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.484+0000 7feca2a00640 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6eeaf10 0x55e2c6eed320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd99f0613495f29 server_cookie=28b8996f3be0d427 in_seq=0 out_seq=0 2024-04-26T01:50:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.484+0000 7fec8d600640 10 client.25796 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.486+0000 7fec8d600640 1 -- 172.21.15.52:0/302324096 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2c6e498b0 con 0x55e2c6eeaf10 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.486+0000 7fec8d600640 10 client.25796 handle_command_reply: tid=0 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.486+0000 7fecad5e9740 1 client.25796 shutdown 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.486+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 msgr2=0x55e2c6ed7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.486+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c6a0 tx=0x7fec9409c940 comp rx=0 tx=0).stop 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 shutdown_connections 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6eeaf10 0x55e2c6eed320 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ed85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 --2- 172.21.15.52:0/302324096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ed7b30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fec8d600640 0 client.25796 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 >> 172.21.15.52:0/302324096 conn(0x55e2c6ecbec0 msgr2=0x55e2c6ed1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 shutdown_connections 2024-04-26T01:50:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 1 -- 172.21.15.52:0/302324096 wait complete. 2024-04-26T01:50:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.487+0000 7fecad5e9740 20 client.25796 trim_cache size 0 max 16384 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 Processor -- start 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 -- start start 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ef2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feca41b24c0 con 0x55e2c6ed02d0 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feca41afd90 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.505+0000 7fecad5e9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feca41adbe0 con 0x55e2c6edab90 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53024/0 (socket says 172.21.15.52:53024) 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ef2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 -- 172.21.15.52:0/450535280 learned_addr learned my addr 172.21.15.52:0/450535280 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca2a00640 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 -- 172.21.15.52:0/450535280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 msgr2=0x55e2c6ef2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ef2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 -- 172.21.15.52:0/450535280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 msgr2=0x55e2c6ed2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca3400640 1 -- 172.21.15.52:0/450535280 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c6ecb730 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.506+0000 7feca2a00640 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.507+0000 7feca3400640 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec940a5180 tx=0x7fec9409c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f482643d67003f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.507+0000 7fec8fe00640 1 -- 172.21.15.52:0/450535280 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9409d140 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.507+0000 7fec8fe00640 1 -- 172.21.15.52:0/450535280 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec940a0df0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.507+0000 7fec8fe00640 1 -- 172.21.15.52:0/450535280 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec940ce470 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- 172.21.15.52:0/450535280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 msgr2=0x55e2c6ee6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec940a5180 tx=0x7fec9409c810 comp rx=0 tx=0).stop 2024-04-26T01:50:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- 172.21.15.52:0/450535280 shutdown_connections 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6edab90 0x55e2c6ef2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ed2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 --2- 172.21.15.52:0/450535280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee6680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- 172.21.15.52:0/450535280 >> 172.21.15.52:0/450535280 conn(0x55e2c6ecbec0 msgr2=0x55e2c6ecb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- 172.21.15.52:0/450535280 shutdown_connections 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- 172.21.15.52:0/450535280 wait complete. 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.508+0000 7fecad5e9740 1 -- start start 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 Processor -- start 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ee04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 0x55e2c6ee0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feca41ae250 con 0x55e2c6edab90 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feca41af6c0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feca41ae920 con 0x55e2c6ed02d0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7fecad5e9740 10 client.0 _ll_register_callbacks cb 0x7fec94098ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7feca3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53028/0 (socket says 172.21.15.52:53028) 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7feca3400640 1 -- 172.21.15.52:0/3969787351 learned_addr learned my addr 172.21.15.52:0/3969787351 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7feca3e00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 0x55e2c6ee0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.509+0000 7feca2a00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ee04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 -- 172.21.15.52:0/3969787351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 msgr2=0x55e2c6ee04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ee04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 -- 172.21.15.52:0/3969787351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 msgr2=0x55e2c6ee0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 0x55e2c6ee0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 -- 172.21.15.52:0/3969787351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c6f504b0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3e00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 0x55e2c6ee0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:56.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca2a00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ee04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:50:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7feca3400640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec9409acf0 tx=0x7fec9409c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25b004a9f76f92c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:50:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7fec8d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:50:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9409d140 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fec9409be90 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec940a0df0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.510+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fec9407d080 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 10 client.25802 fetch_fsmap learned FSMap version 37 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e2c6f515d0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec940a0df0 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fec9409bb00 con 0x55e2c6ecf9b0 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 10 client.25802 fetch_fsmap finished waiting for FSMap version 37 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:50:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 20 client.25802 populate_metadata read hostname 'smithi052' 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6f51c30 0x55e2c6f54020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 4 client.25802 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e2c6e43320 con 0x55e2c6f51c30 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.511+0000 7feca2a00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6f51c30 0x55e2c6f54020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.512+0000 7feca2a00640 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6f51c30 0x55e2c6f54020 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841aabf255b4dd00 server_cookie=d96631e5a3168df9 in_seq=0 out_seq=0 2024-04-26T01:50:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.512+0000 7fec8d600640 10 client.25802 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fec8d600640 1 -- 172.21.15.52:0/3969787351 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e2c6e43320 con 0x55e2c6f51c30 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fec8d600640 10 client.25802 handle_command_reply: tid=0 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 client.25802 shutdown 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 msgr2=0x55e2c6ee30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec9409acf0 tx=0x7fec9409c940 comp rx=0 tx=0).stop 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 shutdown_connections 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e2c6f51c30 0x55e2c6f54020 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e2c6edab90 0x55e2c6ee0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e2c6ed02d0 0x55e2c6ee04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 --2- 172.21.15.52:0/3969787351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e2c6ecf9b0 0x55e2c6ee30e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fec8d600640 0 client.25802 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.553+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 >> 172.21.15.52:0/3969787351 conn(0x55e2c6ecbec0 msgr2=0x55e2c6ed7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 shutdown_connections 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7fecad5e9740 1 -- 172.21.15.52:0/3969787351 wait complete. 2024-04-26T01:50:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7fecad5e9740 20 client.25802 trim_cache size 0 max 16384 2024-04-26T01:50:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:50:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:50:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:50:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 msgr2=0x7fec880a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 0x7fec880a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fec9c004af0 tx=0x7fec9c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 msgr2=0x7feca419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.554+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e9c0 tx=0x7fec9400ee80 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 shutdown_connections 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec880a3d90 0x7fec880a6250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feca419f020 0x7feca419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feca41092d0 0x7feca419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 --2- 172.21.15.52:0/1010487531 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feca4104200 0x7feca419a760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 >> 172.21.15.52:0/1010487531 conn(0x7feca40ffa60 msgr2=0x7feca4100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 shutdown_connections 2024-04-26T01:50:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:50:56.555+0000 7feca9000640 1 -- 172.21.15.52:0/1010487531 wait complete. 2024-04-26T01:50:57.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 145142 3.61 MB/sec execute 136 sec latency 30.636 ms 2024-04-26T01:50:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:57 smithi191 ceph-mon[32745]: pgmap v182: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-04-26T01:50:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:57 smithi183 ceph-mon[32752]: pgmap v182: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-04-26T01:50:57.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:57 smithi052 ceph-mon[27851]: pgmap v182: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 176 op/s 2024-04-26T01:50:58.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 145797 3.63 MB/sec execute 137 sec latency 36.687 ms 2024-04-26T01:50:59.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 146254 3.61 MB/sec execute 138 sec latency 25.306 ms 2024-04-26T01:50:59.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:50:59 smithi191 ceph-mon[32745]: pgmap v183: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:50:59.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:50:59 smithi183 ceph-mon[32752]: pgmap v183: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:50:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:50:59 smithi052 ceph-mon[27851]: pgmap v183: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:51:00.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 147161 3.61 MB/sec execute 139 sec latency 28.274 ms 2024-04-26T01:51:01.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 148693 3.65 MB/sec execute 140 sec latency 53.939 ms 2024-04-26T01:51:01.513 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-26T01:51:01.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:51:01.596 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:01 smithi052 ceph-mon[27851]: pgmap v184: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:51:01.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:01 smithi191 ceph-mon[32745]: pgmap v184: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:51:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:01 smithi183 ceph-mon[32752]: pgmap v184: 97 pgs: 97 active+clean; 71 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 988 KiB/s wr, 101 op/s 2024-04-26T01:51:02.029 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:02.091 INFO:tasks.workunit.client.0.smithi052.stdout: 1 149246 3.66 MB/sec execute 141 sec latency 56.263 ms 2024-04-26T01:51:03.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 149868 3.64 MB/sec execute 142 sec latency 30.515 ms 2024-04-26T01:51:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:03 smithi052 ceph-mon[27851]: pgmap v185: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T01:51:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:03 smithi191 ceph-mon[32745]: pgmap v185: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T01:51:03.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:03 smithi183 ceph-mon[32752]: pgmap v185: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 199 op/s 2024-04-26T01:51:04.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.65 MB/sec execute 143 sec latency 365.253 ms 2024-04-26T01:51:04.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.381+0000 7fef71c00700 1 -- 172.21.15.52:0/2879213430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c103490 msgr2=0x7fef6c10f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.381+0000 7fef71c00700 1 --2- 172.21.15.52:0/2879213430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c103490 0x7fef6c10f970 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef640098f0 tx=0x7fef64009c00 comp rx=0 tx=0).stop 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 -- 172.21.15.52:0/2879213430 shutdown_connections 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 --2- 172.21.15.52:0/2879213430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c103490 0x7fef6c10f970 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 --2- 172.21.15.52:0/2879213430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c108aa0 0x7fef6c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 --2- 172.21.15.52:0/2879213430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c102ad0 0x7fef6c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 -- 172.21.15.52:0/2879213430 >> 172.21.15.52:0/2879213430 conn(0x7fef6c079c30 msgr2=0x7fef6c07a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 -- 172.21.15.52:0/2879213430 shutdown_connections 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.382+0000 7fef71c00700 1 -- 172.21.15.52:0/2879213430 wait complete. 2024-04-26T01:51:04.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.384+0000 7fef71c00700 1 Processor -- start 2024-04-26T01:51:04.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.384+0000 7fef71c00700 1 -- start start 2024-04-26T01:51:04.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:04.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 0x7fef6c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:04.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 0x7fef6c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:04.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c112cd0 con 0x7fef6c102ad0 2024-04-26T01:51:04.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c112b50 con 0x7fef6c103490 2024-04-26T01:51:04.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef71c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c112e50 con 0x7fef6c108aa0 2024-04-26T01:51:04.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef6aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef6aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53418/0 (socket says 172.21.15.52:53418) 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef6b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 0x7fef6c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef6a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 0x7fef6c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.385+0000 7fef6aa00700 1 -- 172.21.15.52:0/4058845289 learned_addr learned my addr 172.21.15.52:0/4058845289 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 -- 172.21.15.52:0/4058845289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 msgr2=0x7fef6c199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 0x7fef6c199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 -- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 msgr2=0x7fef6c1991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 -- 172.21.15.52:0/4058845289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef6c19e1f0 con 0x7fef6c108aa0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6aa00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6a000700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 0x7fef6c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef6b400700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 0x7fef6c19dab0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef64009810 tx=0x7fef6400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64027070 con 0x7fef6c108aa0 2024-04-26T01:51:04.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef640056b0 con 0x7fef6c108aa0 2024-04-26T01:51:04.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64004950 con 0x7fef6c108aa0 2024-04-26T01:51:04.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.386+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef6c19e480 con 0x7fef6c108aa0 2024-04-26T01:51:04.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.387+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef6c1a5d60 con 0x7fef6c108aa0 2024-04-26T01:51:04.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.388+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef6c069730 con 0x7fef6c108aa0 2024-04-26T01:51:04.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.390+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fef6402a070 con 0x7fef6c108aa0 2024-04-26T01:51:04.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.392+0000 7fef63400700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 0x7fef500a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:04.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.392+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fef640d1270 con 0x7fef6c108aa0 2024-04-26T01:51:04.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.392+0000 7fef6aa00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 0x7fef500a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:04.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.393+0000 7fef6aa00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 0x7fef500a62f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009fd0 tx=0x7fef5c00b040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:04.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.394+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef64013030 con 0x7fef6c108aa0 2024-04-26T01:51:04.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:04 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:04.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:04 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:04.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:04 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:51:04.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.684+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef6c02df80 con 0x7fef6c108aa0 2024-04-26T01:51:04.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.685+0000 7fef63400700 1 -- 172.21.15.52:0/4058845289 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7fef6409d970 con 0x7fef6c108aa0 2024-04-26T01:51:04.684 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:51:04.684 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:51:04.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 msgr2=0x7fef500a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:04.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 0x7fef500a62f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009fd0 tx=0x7fef5c00b040 comp rx=0 tx=0).stop 2024-04-26T01:51:04.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 msgr2=0x7fef6c19dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:04.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 0x7fef6c19dab0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef64009810 tx=0x7fef6400fa00 comp rx=0 tx=0).stop 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 shutdown_connections 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef500a3e30 0x7fef500a62f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef6c108aa0 0x7fef6c19dab0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef6c102ad0 0x7fef6c1991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 --2- 172.21.15.52:0/4058845289 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef6c103490 0x7fef6c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 >> 172.21.15.52:0/4058845289 conn(0x7fef6c079c30 msgr2=0x7fef6c100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 shutdown_connections 2024-04-26T01:51:04.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:04.690+0000 7fef71c00700 1 -- 172.21.15.52:0/4058845289 wait complete. 2024-04-26T01:51:04.690 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:51:05.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.62 MB/sec execute 144 sec latency 1365.327 ms 2024-04-26T01:51:05.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:05 smithi052 ceph-mon[27851]: pgmap v186: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-26T01:51:05.547 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:05 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4058845289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:05.598 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-26T01:51:05.598 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-26T01:51:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:05 smithi191 ceph-mon[32745]: pgmap v186: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-26T01:51:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:05 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4058845289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:05.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:05 smithi183 ceph-mon[32752]: pgmap v186: 97 pgs: 97 active+clean; 70 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-26T01:51:05.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:05 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4058845289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:06.040 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:06.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.60 MB/sec execute 145 sec latency 2365.410 ms 2024-04-26T01:51:07.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.57 MB/sec execute 146 sec latency 3365.513 ms 2024-04-26T01:51:07.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:07 smithi052 ceph-mon[27851]: pgmap v187: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T01:51:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:07 smithi191 ceph-mon[32745]: pgmap v187: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T01:51:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:07 smithi183 ceph-mon[32752]: pgmap v187: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T01:51:08.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.55 MB/sec execute 147 sec latency 4365.607 ms 2024-04-26T01:51:08.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.373+0000 7fb67e600700 1 -- 172.21.15.52:0/1615246625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678101780 msgr2=0x7fb678101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.373+0000 7fb67e600700 1 --2- 172.21.15.52:0/1615246625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678101780 0x7fb678101c00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb664009990 tx=0x7fb664009ca0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 -- 172.21.15.52:0/1615246625 shutdown_connections 2024-04-26T01:51:08.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 --2- 172.21.15.52:0/1615246625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678102140 0x7fb67810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 --2- 172.21.15.52:0/1615246625 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678107780 0x7fb678107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 --2- 172.21.15.52:0/1615246625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678101780 0x7fb678101c00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 -- 172.21.15.52:0/1615246625 >> 172.21.15.52:0/1615246625 conn(0x7fb6780fd290 msgr2=0x7fb6780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:08.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 -- 172.21.15.52:0/1615246625 shutdown_connections 2024-04-26T01:51:08.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.374+0000 7fb67e600700 1 -- 172.21.15.52:0/1615246625 wait complete. 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.375+0000 7fb67e600700 1 Processor -- start 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.375+0000 7fb67e600700 1 -- start start 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 0x7fb6780fdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 0x7fb6780fe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb678111960 con 0x7fb678107780 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781117e0 con 0x7fb678102140 2024-04-26T01:51:08.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67e600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb678111ae0 con 0x7fb678101780 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67d200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 0x7fb6780fe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 0x7fb6780fdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 0x7fb6780fe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57626/0 (socket says 172.21.15.52:57626) 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67dc00700 1 -- 172.21.15.52:0/468855301 learned_addr learned my addr 172.21.15.52:0/468855301 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 0x7fb6780fdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43740/0 (socket says 172.21.15.52:43740) 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.376+0000 7fb67d200700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56986/0 (socket says 172.21.15.52:56986) 2024-04-26T01:51:08.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 -- 172.21.15.52:0/468855301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 msgr2=0x7fb6780fdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 0x7fb6780fdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 -- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 msgr2=0x7fb6780fe1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 0x7fb6780fe1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 -- 172.21.15.52:0/468855301 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6781a73f0 con 0x7fb678101780 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67d200700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00e9f0 tx=0x7fb66c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb675600700 1 -- 172.21.15.52:0/468855301 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00cb70 con 0x7fb678101780 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb675600700 1 -- 172.21.15.52:0/468855301 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb66c00ee90 con 0x7fb678101780 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6781a76e0 con 0x7fb678101780 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb675600700 1 -- 172.21.15.52:0/468855301 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c003d40 con 0x7fb678101780 2024-04-26T01:51:08.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.377+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781a7c40 con 0x7fb678101780 2024-04-26T01:51:08.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.381+0000 7fb675600700 1 -- 172.21.15.52:0/468855301 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb66c003ee0 con 0x7fb678101780 2024-04-26T01:51:08.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.382+0000 7fb675600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 0x7fb6600a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.382+0000 7fb68a666b80 1 Processor -- start 2024-04-26T01:51:08.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.382+0000 7fb675600700 1 -- 172.21.15.52:0/468855301 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb66c0d07e0 con 0x7fb678101780 2024-04-26T01:51:08.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb677e00700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 0x7fb6600a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 -- start start 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df9b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 0x55e10df9e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e10dedb020 con 0x55e10df88850 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e10de818c0 con 0x55e10df9bf10 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.383+0000 7fb68a666b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e10ded6fe0 con 0x55e10df87f30 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb677e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df9b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56990/0 (socket says 172.21.15.52:56990) 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 -- 172.21.15.52:0/264784804 learned_addr learned my addr 172.21.15.52:0/264784804 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb677e00700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 0x7fb6600a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b2e0 tx=0x7fb664005960 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67dc00700 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 0x55e10df9e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 -- 172.21.15.52:0/264784804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 msgr2=0x55e10df9e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 0x55e10df9e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 -- 172.21.15.52:0/264784804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 msgr2=0x55e10df9b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df9b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 -- 172.21.15.52:0/264784804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10de77810 con 0x55e10df87f30 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67dc00700 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 0x55e10df9e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.384+0000 7fb67d200700 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0a4e90 tx=0x7fb66c09cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761c556c23e8b69d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.385+0000 7fb66a000700 1 -- 172.21.15.52:0/264784804 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0a53c0 con 0x55e10df87f30 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.385+0000 7fb66a000700 1 -- 172.21.15.52:0/264784804 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb66c09aa50 con 0x55e10df87f30 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.385+0000 7fb66a000700 1 -- 172.21.15.52:0/264784804 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0a4f30 con 0x55e10df87f30 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.385+0000 7fb68a666b80 1 -- 172.21.15.52:0/264784804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 msgr2=0x55e10df88310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.385+0000 7fb68a666b80 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0a4e90 tx=0x7fb66c09cf20 comp rx=0 tx=0).stop 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 -- 172.21.15.52:0/264784804 shutdown_connections 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df9bf10 0x55e10df9e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df88310 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 --2- 172.21.15.52:0/264784804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df9b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 -- 172.21.15.52:0/264784804 >> 172.21.15.52:0/264784804 conn(0x55e10deb70e0 msgr2=0x55e10df84480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 -- 172.21.15.52:0/264784804 shutdown_connections 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 -- 172.21.15.52:0/264784804 wait complete. 2024-04-26T01:51:08.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.386+0000 7fb68a666b80 1 -- start start 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 Processor -- start 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 0x55e10df98a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df9bf10 0x55e10df98fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e10df9e9a0 con 0x55e10df9bf10 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e10de949f0 con 0x55e10df88850 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e10df9eb20 con 0x55e10df87f30 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb68a666b80 10 client.0 _ll_register_callbacks cb 0x55e10de738b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.387+0000 7fb67dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df9bf10 0x55e10df98fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57006/0 (socket says 172.21.15.52:57006) 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 -- 172.21.15.52:0/678389089 learned_addr learned my addr 172.21.15.52:0/678389089 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb677e00700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 0x55e10df98a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 -- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 msgr2=0x55e10df98a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 0x55e10df98a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 -- 172.21.15.52:0/678389089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df9bf10 msgr2=0x55e10df98fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df9bf10 0x55e10df98fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 -- 172.21.15.52:0/678389089 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10de28a60 con 0x55e10df87f30 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb677e00700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 0x55e10df98a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb67d200700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0ce480 tx=0x7fb66c009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca24876c9fdf0ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.388+0000 7fb65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c080060 con 0x55e10df87f30 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb66c0a5130 con 0x55e10df87f30 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb66c09aa50 con 0x55e10df87f30 2024-04-26T01:51:08.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb66c0bb030 con 0x55e10df87f30 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb68a666b80 10 client.25828 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e10de71c80 con 0x55e10df87f30 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.389+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb66c09aa50 con 0x55e10df87f30 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fb66c0984a0 con 0x55e10df87f30 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 10 client.25828 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:08.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 10 client.25828 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 20 client.25828 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10dfab610 0x55e10dfada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 4 client.25828 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e10dee8810 con 0x55e10dfab610 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.390+0000 7fb677e00700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10dfab610 0x55e10dfada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.391+0000 7fb677e00700 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10dfab610 0x55e10dfada00 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb41fbc444ebd61b server_cookie=646bcf4eb240c771 in_seq=0 out_seq=0 2024-04-26T01:51:08.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.391+0000 7fb65ea00700 10 client.25828 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb65ea00700 1 -- 172.21.15.52:0/678389089 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e10dee8810 con 0x55e10dfab610 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb65ea00700 10 client.25828 handle_command_reply: tid=0 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 client.25828 shutdown 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 msgr2=0x55e10df9b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0ce480 tx=0x7fb66c009630 comp rx=0 tx=0).stop 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 shutdown_connections 2024-04-26T01:51:08.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10dfab610 0x55e10dfada00 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df9bf10 0x55e10df98fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df87f30 0x55e10df9b770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 --2- 172.21.15.52:0/678389089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df88850 0x55e10df98a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb65ea00700 0 client.25828 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 >> 172.21.15.52:0/678389089 conn(0x55e10deb70e0 msgr2=0x55e10df84070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 shutdown_connections 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.392+0000 7fb68a666b80 1 -- 172.21.15.52:0/678389089 wait complete. 2024-04-26T01:51:08.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.393+0000 7fb68a666b80 20 client.25828 trim_cache size 0 max 16384 2024-04-26T01:51:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.414+0000 7fb68a666b80 1 Processor -- start 2024-04-26T01:51:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 -- start start 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10deb70e0 0x55e10df97c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10df8a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781ad730 con 0x55e10df88850 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781aafc0 con 0x55e10df87f30 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb68a666b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb67819cd40 con 0x55e10deb70e0 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.415+0000 7fb67dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb67dc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57662/0 (socket says 172.21.15.52:57662) 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb67dc00700 1 -- 172.21.15.52:0/836259041 learned_addr learned my addr 172.21.15.52:0/836259041 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10df8a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb67d200700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10deb70e0 0x55e10df97c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 -- 172.21.15.52:0/836259041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10deb70e0 msgr2=0x55e10df97c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10deb70e0 0x55e10df97c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 -- 172.21.15.52:0/836259041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 msgr2=0x55e10df8abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 -- 172.21.15.52:0/836259041 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10df89d80 con 0x55e10df87f30 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb67dc00700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.416+0000 7fb677e00700 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10df8a6b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb6640059d0 tx=0x7fb664021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=603254b10fe4b915 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.417+0000 7fb669600700 1 -- 172.21.15.52:0/836259041 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664004970 con 0x55e10df87f30 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.417+0000 7fb669600700 1 -- 172.21.15.52:0/836259041 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb664032340 con 0x55e10df87f30 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.417+0000 7fb669600700 1 -- 172.21.15.52:0/836259041 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66400f6a0 con 0x55e10df87f30 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.417+0000 7fb68a666b80 1 -- 172.21.15.52:0/836259041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 msgr2=0x55e10df8a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.417+0000 7fb68a666b80 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10df8a6b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb6640059d0 tx=0x7fb664021000 comp rx=0 tx=0).stop 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 -- 172.21.15.52:0/836259041 shutdown_connections 2024-04-26T01:51:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10df88850 0x55e10df8abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10deb70e0 0x55e10df97c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 --2- 172.21.15.52:0/836259041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10df8a6b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 -- 172.21.15.52:0/836259041 >> 172.21.15.52:0/836259041 conn(0x55e10df94500 msgr2=0x55e10df89fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:08.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 -- 172.21.15.52:0/836259041 shutdown_connections 2024-04-26T01:51:08.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 -- 172.21.15.52:0/836259041 wait complete. 2024-04-26T01:51:08.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.418+0000 7fb68a666b80 1 -- start start 2024-04-26T01:51:08.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 Processor -- start 2024-04-26T01:51:08.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df88850 0x55e10dfaff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781a9480 con 0x55e10deb70e0 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781aa8f0 con 0x55e10df87f30 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781a9b50 con 0x55e10df88850 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb68a666b80 10 client.0 _ll_register_callbacks cb 0x55e10de738b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb67d200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.419+0000 7fb677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb67d200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57664/0 (socket says 172.21.15.52:57664) 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb67d200700 1 -- 172.21.15.52:0/3965325423 learned_addr learned my addr 172.21.15.52:0/3965325423 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43786/0 (socket says 172.21.15.52:43786) 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb67dc00700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df88850 0x55e10dfaff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 -- 172.21.15.52:0/3965325423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df88850 msgr2=0x55e10dfaff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df88850 0x55e10dfaff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 -- 172.21.15.52:0/3965325423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 msgr2=0x55e10dfaf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 -- 172.21.15.52:0/3965325423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10df84f50 con 0x55e10df87f30 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb67d200700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb677e00700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e10df85b80 tx=0x7fb664009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b830841215615d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:08.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.420+0000 7fb65ea00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:51:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664041060 con 0x55e10df87f30 2024-04-26T01:51:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb664032340 con 0x55e10df87f30 2024-04-26T01:51:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb6640326f0 con 0x55e10df87f30 2024-04-26T01:51:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb6640328f0 con 0x55e10df87f30 2024-04-26T01:51:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb68a666b80 10 client.25826 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.421+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e10e012990 con 0x55e10df87f30 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb664032340 con 0x55e10df87f30 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fb664046ae0 con 0x55e10df87f30 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 10 client.25826 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 10 client.25826 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 20 client.25826 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10df9bf10 0x55e10e015010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 4 client.25826 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x55e10dee28b0 con 0x55e10df9bf10 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.422+0000 7fb67d200700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10df9bf10 0x55e10e015010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.423+0000 7fb67d200700 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10df9bf10 0x55e10e015010 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bd5c1f44e0c4782 server_cookie=ad751cde91a2f2ee in_seq=0 out_seq=0 2024-04-26T01:51:08.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:08.423+0000 7fb65ea00700 10 client.25826 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:09.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.52 MB/sec execute 148 sec latency 5365.752 ms 2024-04-26T01:51:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:09 smithi191 ceph-mon[32745]: pgmap v188: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:09 smithi183 ceph-mon[32752]: pgmap v188: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.701+0000 7fdaf5400640 1 -- 172.21.15.52:0/3686706345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf010c5d0 msgr2=0x7fdaf010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.701+0000 7fdaf5400640 1 --2- 172.21.15.52:0/3686706345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf010c5d0 0x7fdaf010ea00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009900 tx=0x7fdae802a480 comp rx=0 tx=0).stop 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 -- 172.21.15.52:0/3686706345 shutdown_connections 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 --2- 172.21.15.52:0/3686706345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf010c5d0 0x7fdaf010ea00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 --2- 172.21.15.52:0/3686706345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 0x7fdaf010c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 --2- 172.21.15.52:0/3686706345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf0103200 0x7fdaf01035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 -- 172.21.15.52:0/3686706345 >> 172.21.15.52:0/3686706345 conn(0x7fdaf00771d0 msgr2=0x7fdaf00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 -- 172.21.15.52:0/3686706345 shutdown_connections 2024-04-26T01:51:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.702+0000 7fdaf5400640 1 -- 172.21.15.52:0/3686706345 wait complete. 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 Processor -- start 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 -- start start 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 0x7fdaf0101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 0x7fdaf01018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf0112220 con 0x7fdaf0103200 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01120a0 con 0x7fdaf010c5d0 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.703+0000 7fdaf5400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01123a0 con 0x7fdaf0103b20 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57682/0 (socket says 172.21.15.52:57682) 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 -- 172.21.15.52:0/731303614 learned_addr learned my addr 172.21.15.52:0/731303614 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaef200640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 0x7fdaf01018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdae5e00640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 0x7fdaf0101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 -- 172.21.15.52:0/731303614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 msgr2=0x7fdaf0101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 0x7fdaf0101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 -- 172.21.15.52:0/731303614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 msgr2=0x7fdaf01018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 0x7fdaf01018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 -- 172.21.15.52:0/731303614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf01a6b20 con 0x7fdaf0103200 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaef200640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 0x7fdaf01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.704+0000 7fdaee800640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdae000ebc0 tx=0x7fdae000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.705+0000 7fdae7e00640 1 -- 172.21.15.52:0/731303614 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0010630 con 0x7fdaf0103200 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.705+0000 7fdae7e00640 1 -- 172.21.15.52:0/731303614 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae0010c80 con 0x7fdaf0103200 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.705+0000 7fdae7e00640 1 -- 172.21.15.52:0/731303614 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0018b60 con 0x7fdaf0103200 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.705+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf01a6e10 con 0x7fdaf0103200 2024-04-26T01:51:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.705+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf01a72d0 con 0x7fdaf0103200 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.708+0000 7fdaf8389740 1 Processor -- start 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.708+0000 7fdae7e00640 1 -- 172.21.15.52:0/731303614 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdae0018d00 con 0x7fdaf0103200 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.708+0000 7fdaf8389740 1 -- start start 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 0x562efa4d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562efa42d900 con 0x562efa4d2a00 2024-04-26T01:51:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562efa3d3ef0 con 0x562efa4c7820 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaf8389740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562efa4298c0 con 0x562efa4c8140 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaef200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdaee800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 0x562efa4d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57694/0 (socket says 172.21.15.52:57694) 2024-04-26T01:51:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.709+0000 7fdae5e00640 1 -- 172.21.15.52:0/1843885825 learned_addr learned my addr 172.21.15.52:0/1843885825 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 -- 172.21.15.52:0/1843885825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 msgr2=0x562efa4c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 -- 172.21.15.52:0/1843885825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 msgr2=0x562efa4d4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 -- 172.21.15.52:0/1843885825 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562efa140210 con 0x562efa4c8140 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaef200640 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdaee800640 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 0x562efa4d24c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdae00ce510 tx=0x7fdae00d7830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db9bfb09cc9020a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdae7e00640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 0x7fdad40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.710+0000 7fdadf400640 1 -- 172.21.15.52:0/1843885825 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae009c260 con 0x562efa4c8140 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdadf400640 1 -- 172.21.15.52:0/1843885825 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae0018420 con 0x562efa4c8140 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdadf400640 1 -- 172.21.15.52:0/1843885825 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae009c590 con 0x562efa4c8140 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaef200640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 0x7fdad40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdae7e00640 1 -- 172.21.15.52:0/731303614 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fdae00d0df0 con 0x7fdaf0103200 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 -- 172.21.15.52:0/1843885825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 msgr2=0x562efa4d24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 0x562efa4d24c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdae00ce510 tx=0x7fdae00d7830 comp rx=0 tx=0).stop 2024-04-26T01:51:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaef200640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 0x7fdad40a6280 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdae8005ab0 tx=0x7fdae8005a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 -- 172.21.15.52:0/1843885825 shutdown_connections 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c8140 0x562efa4d24c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 --2- 172.21.15.52:0/1843885825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.711+0000 7fdaf8389740 1 -- 172.21.15.52:0/1843885825 >> 172.21.15.52:0/1843885825 conn(0x562efa4c3d30 msgr2=0x562efa4c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.712+0000 7fdaf8389740 1 -- 172.21.15.52:0/1843885825 shutdown_connections 2024-04-26T01:51:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.712+0000 7fdaf8389740 1 -- 172.21.15.52:0/1843885825 wait complete. 2024-04-26T01:51:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.712+0000 7fdaf8389740 1 -- start start 2024-04-26T01:51:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.712+0000 7fdaf8389740 1 Processor -- start 2024-04-26T01:51:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c8140 0x562efa4cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562efa4d5630 con 0x562efa4d2a00 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562efa3e7310 con 0x562efa4c8140 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562efa4d57b0 con 0x562efa4c7820 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaf8389740 10 client.0 _ll_register_callbacks cb 0x562efa3b5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaee800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaee800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57058/0 (socket says 172.21.15.52:57058) 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaee800640 1 -- 172.21.15.52:0/771672155 learned_addr learned my addr 172.21.15.52:0/771672155 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdae5e00640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c8140 0x562efa4cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:09 smithi052 ceph-mon[27851]: pgmap v188: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaef200640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.713+0000 7fdaee800640 1 -- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c8140 msgr2=0x562efa4cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaee800640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c8140 0x562efa4cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaee800640 1 -- 172.21.15.52:0/771672155 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 msgr2=0x562efa4d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaee800640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaee800640 1 -- 172.21.15.52:0/771672155 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562efa20c620 con 0x562efa4c7820 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaee800640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdae000eb90 tx=0x7fdae0018600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c16cb4f48a0d1f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdad3e00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae009c770 con 0x562efa4c7820 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562efa4418b0 con 0x562efa4c7820 2024-04-26T01:51:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fdae001a070 con 0x562efa4c7820 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdae00b6050 con 0x562efa4c7820 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaf8389740 10 client.25846 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.714+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562efa4d5630 con 0x562efa4c7820 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562efa4418b0 con 0x562efa4c7820 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fdae0099480 con 0x562efa4c7820 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 10 client.25846 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 10 client.25846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 20 client.25846 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa4e29d0 0x562efa4e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 4 client.25846 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562efa43b320 con 0x562efa4e29d0 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.715+0000 7fdae5e00640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa4e29d0 0x562efa4e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.716+0000 7fdae5e00640 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa4e29d0 0x562efa4e4dc0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55c7914deb0c493 server_cookie=4c50a7945512c34e in_seq=0 out_seq=0 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.716+0000 7fdad3e00640 10 client.25846 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdad3e00640 1 -- 172.21.15.52:0/771672155 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562efa43b320 con 0x562efa4e29d0 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdad3e00640 10 client.25846 handle_command_reply: tid=0 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 client.25846 shutdown 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 msgr2=0x562efa4cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdae000eb90 tx=0x7fdae0018600 comp rx=0 tx=0).stop 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 shutdown_connections 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa4e29d0 0x562efa4e4dc0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4d2a00 0x562efa4d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c8140 0x562efa4cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 --2- 172.21.15.52:0/771672155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4cf910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 >> 172.21.15.52:0/771672155 conn(0x562efa4c3d30 msgr2=0x562efa4c9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdad3e00640 0 client.25846 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 shutdown_connections 2024-04-26T01:51:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.717+0000 7fdaf8389740 1 -- 172.21.15.52:0/771672155 wait complete. 2024-04-26T01:51:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.718+0000 7fdaf8389740 20 client.25846 trim_cache size 0 max 16384 2024-04-26T01:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 Processor -- start 2024-04-26T01:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 -- start start 2024-04-26T01:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4de4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4d2a00 0x562efa4eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01b25d0 con 0x562efa4c8140 2024-04-26T01:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01afea0 con 0x562efa4d2a00 2024-04-26T01:51:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.734+0000 7fdaf8389740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01adcf0 con 0x562efa4c7820 2024-04-26T01:51:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57712/0 (socket says 172.21.15.52:57712) 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdae5e00640 1 -- 172.21.15.52:0/3809804974 learned_addr learned my addr 172.21.15.52:0/3809804974 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4de4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaef200640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4d2a00 0x562efa4eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 -- 172.21.15.52:0/3809804974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4d2a00 msgr2=0x562efa4eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4d2a00 0x562efa4eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 -- 172.21.15.52:0/3809804974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 msgr2=0x562efa4ea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 -- 172.21.15.52:0/3809804974 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562efa4c8ad0 con 0x562efa4c7820 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdae5e00640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdaee800640 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4de4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdae009b0c0 tx=0x7fdae009b320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f7740ccfea9a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.735+0000 7fdadea00640 1 -- 172.21.15.52:0/3809804974 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae009c950 con 0x562efa4c7820 2024-04-26T01:51:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdadea00640 1 -- 172.21.15.52:0/3809804974 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae00d7d30 con 0x562efa4c7820 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdadea00640 1 -- 172.21.15.52:0/3809804974 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae009cbe0 con 0x562efa4c7820 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 -- 172.21.15.52:0/3809804974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 msgr2=0x562efa4de4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4de4f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdae009b0c0 tx=0x7fdae009b320 comp rx=0 tx=0).stop 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 -- 172.21.15.52:0/3809804974 shutdown_connections 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4d2a00 0x562efa4eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3809804974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4c7820 0x562efa4de4f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 -- 172.21.15.52:0/3809804974 >> 172.21.15.52:0/3809804974 conn(0x562efa4c3d30 msgr2=0x562efa4c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 -- 172.21.15.52:0/3809804974 shutdown_connections 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.736+0000 7fdaf8389740 1 -- 172.21.15.52:0/3809804974 wait complete. 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 -- start start 2024-04-26T01:51:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 Processor -- start 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4d83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4d2a00 0x562efa4d8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01ae360 con 0x562efa4c8140 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01af7d0 con 0x562efa4c7820 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf01aea30 con 0x562efa4d2a00 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.737+0000 7fdaf8389740 10 client.0 _ll_register_callbacks cb 0x7fdae00d4de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57726/0 (socket says 172.21.15.52:57726) 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 -- 172.21.15.52:0/3988068043 learned_addr learned my addr 172.21.15.52:0/3988068043 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdaee800640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4d83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdaef200640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4d2a00 0x562efa4d8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 -- 172.21.15.52:0/3988068043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4d2a00 msgr2=0x562efa4d8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4d2a00 0x562efa4d8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 -- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 msgr2=0x562efa4d83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4d83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 -- 172.21.15.52:0/3988068043 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562efa547e10 con 0x562efa4c8140 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdaee800640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4d83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdae5e00640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009530 tx=0x7fdad80044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a8b514777130aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.738+0000 7fdad3e00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8020310 con 0x562efa4c8140 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fdad80204b0 con 0x562efa4c8140 2024-04-26T01:51:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae00d7d30 con 0x562efa4c8140 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdad8030050 con 0x562efa4c8140 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdaf8389740 10 client.15738 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562efa5490f0 con 0x562efa4c8140 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.739+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae00d7d30 con 0x562efa4c8140 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fdad8010c50 con 0x562efa4c8140 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 10 client.15738 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 20 client.15738 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa5498a0 0x562efa54bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 4 client.15738 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562efa435190 con 0x562efa5498a0 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.740+0000 7fdaee800640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa5498a0 0x562efa54bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.741+0000 7fdaee800640 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa5498a0 0x562efa54bcc0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c5867600e79504 server_cookie=d736ac9a30246168 in_seq=0 out_seq=0 2024-04-26T01:51:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.741+0000 7fdad3e00640 10 client.15738 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdad3e00640 1 -- 172.21.15.52:0/3988068043 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562efa435190 con 0x562efa5498a0 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdad3e00640 10 client.15738 handle_command_reply: tid=0 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 client.15738 shutdown 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 msgr2=0x562efa4d8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009530 tx=0x7fdad80044d0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 shutdown_connections 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562efa5498a0 0x562efa54bcc0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562efa4d2a00 0x562efa4d8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562efa4c8140 0x562efa4d8920 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 --2- 172.21.15.52:0/3988068043 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562efa4c7820 0x562efa4d83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdad3e00640 0 client.15738 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 >> 172.21.15.52:0/3988068043 conn(0x562efa4c3d30 msgr2=0x562efa4cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 shutdown_connections 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.775+0000 7fdaf8389740 1 -- 172.21.15.52:0/3988068043 wait complete. 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf8389740 20 client.15738 trim_cache size 0 max 16384 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:51:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 msgr2=0x7fdad40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 0x7fdad40a6280 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdae8005ab0 tx=0x7fdae8005a40 comp rx=0 tx=0).stop 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 msgr2=0x7fdaf0100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdae000ebc0 tx=0x7fdae000c6a0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 shutdown_connections 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdad40a3dc0 0x7fdad40a6280 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdaf010c5d0 0x7fdaf01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0103b20 0x7fdaf0101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.776+0000 7fdaf5400640 1 --2- 172.21.15.52:0/731303614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdaf0103200 0x7fdaf0100e50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.777+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 >> 172.21.15.52:0/731303614 conn(0x7fdaf00771d0 msgr2=0x7fdaf00fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.777+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 shutdown_connections 2024-04-26T01:51:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:09.777+0000 7fdaf5400640 1 -- 172.21.15.52:0/731303614 wait complete. 2024-04-26T01:51:10.092 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.50 MB/sec execute 149 sec latency 6365.898 ms 2024-04-26T01:51:11.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.48 MB/sec execute 150 sec latency 7366.051 ms 2024-04-26T01:51:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:11 smithi191 ceph-mon[32745]: pgmap v189: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:11.743 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:11 smithi052 ceph-mon[27851]: pgmap v189: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:11.744 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:11 smithi183 ceph-mon[32752]: pgmap v189: 97 pgs: 97 active+clean; 72 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 158 op/s 2024-04-26T01:51:12.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.45 MB/sec execute 151 sec latency 8366.250 ms 2024-04-26T01:51:13.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.43 MB/sec execute 152 sec latency 9366.419 ms 2024-04-26T01:51:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:13 smithi191 ceph-mon[32745]: pgmap v190: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:51:13.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:13 smithi183 ceph-mon[32752]: pgmap v190: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:51:13.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:13 smithi052 ceph-mon[27851]: pgmap v190: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-04-26T01:51:14.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.41 MB/sec execute 153 sec latency 10366.585 ms 2024-04-26T01:51:15.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.39 MB/sec execute 154 sec latency 11366.705 ms 2024-04-26T01:51:15.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:15 smithi052 ceph-mon[27851]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-26T01:51:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:15 smithi191 ceph-mon[32745]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-26T01:51:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:15 smithi183 ceph-mon[32752]: pgmap v191: 97 pgs: 97 active+clean; 68 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-26T01:51:16.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.36 MB/sec execute 155 sec latency 12366.811 ms 2024-04-26T01:51:17.093 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.34 MB/sec execute 156 sec latency 13366.969 ms 2024-04-26T01:51:17.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:17 smithi191 ceph-mon[32745]: pgmap v192: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-26T01:51:17.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:17 smithi183 ceph-mon[32752]: pgmap v192: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-26T01:51:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:17 smithi052 ceph-mon[27851]: pgmap v192: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-26T01:51:18.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.32 MB/sec execute 157 sec latency 14367.142 ms 2024-04-26T01:51:18.549 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:18 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:18.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:18 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:18.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:18 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:51:19.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.30 MB/sec execute 158 sec latency 15367.237 ms 2024-04-26T01:51:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:19 smithi191 ceph-mon[32745]: pgmap v193: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:19 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:19 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:19.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:19 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:51:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:19 smithi183 ceph-mon[32752]: pgmap v193: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:19 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:19 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:19 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:51:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:19 smithi052 ceph-mon[27851]: pgmap v193: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:19 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:51:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:19 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:51:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:19 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:51:20.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.28 MB/sec execute 159 sec latency 16367.396 ms 2024-04-26T01:51:21.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.26 MB/sec execute 160 sec latency 17367.541 ms 2024-04-26T01:51:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:21 smithi191 ceph-mon[32745]: pgmap v194: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:21 smithi183 ceph-mon[32752]: pgmap v194: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:21.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:21 smithi052 ceph-mon[27851]: pgmap v194: 97 pgs: 97 active+clean; 67 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 746 KiB/s wr, 52 op/s 2024-04-26T01:51:22.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.24 MB/sec execute 161 sec latency 18367.705 ms 2024-04-26T01:51:22.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.837+0000 7fb65ea00700 1 -- 172.21.15.52:0/3965325423 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e10dee28b0 con 0x55e10df9bf10 2024-04-26T01:51:22.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.837+0000 7fb65ea00700 10 client.25826 handle_command_reply: tid=0 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.837+0000 7fb68a666b80 1 client.25826 shutdown 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 msgr2=0x55e10dfaf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e10df85b80 tx=0x7fb664009290 comp rx=0 tx=0).stop 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 shutdown_connections 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e10df9bf10 0x55e10e015010 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e10df88850 0x55e10dfaff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e10deb70e0 0x55e10dfaf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 --2- 172.21.15.52:0/3965325423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e10df87f30 0x55e10dfaf9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb65ea00700 0 client.25826 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:22.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 >> 172.21.15.52:0/3965325423 conn(0x55e10df94500 msgr2=0x55e10df97ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 14, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "93ad249f": { 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 14, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:51:22.840 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 70.0, 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:22.841 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:51:22.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 shutdown_connections 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.838+0000 7fb68a666b80 1 -- 172.21.15.52:0/3965325423 wait complete. 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.839+0000 7fb68a666b80 20 client.25826 trim_cache size 0 max 16384 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 msgr2=0x7fb6600a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 0x7fb6600a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b2e0 tx=0x7fb664005960 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 msgr2=0x7fb6780fd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00e9f0 tx=0x7fb66c00ed00 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 shutdown_connections 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb6600a3ed0 0x7fb6600a6390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb678107780 0x7fb6780fe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb678101780 0x7fb6780fd740 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 --2- 172.21.15.52:0/468855301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb678102140 0x7fb6780fdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.840+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 >> 172.21.15.52:0/468855301 conn(0x7fb6780fd290 msgr2=0x7fb67806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.841+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 shutdown_connections 2024-04-26T01:51:22.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:22.841+0000 7fb67e600700 1 -- 172.21.15.52:0/468855301 wait complete. 2024-04-26T01:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 -- 172.21.15.52:0/3327778063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 msgr2=0x7fbe0c1007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 --2- 172.21.15.52:0/3327778063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1007c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009900 tx=0x7fbdfc02a410 comp rx=0 tx=0).stop 2024-04-26T01:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 -- 172.21.15.52:0/3327778063 shutdown_connections 2024-04-26T01:51:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 --2- 172.21.15.52:0/3327778063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 --2- 172.21.15.52:0/3327778063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c100d90 0x7fbe0c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 --2- 172.21.15.52:0/3327778063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1007c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 -- 172.21.15.52:0/3327778063 >> 172.21.15.52:0/3327778063 conn(0x7fbe0c076f50 msgr2=0x7fbe0c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 -- 172.21.15.52:0/3327778063 shutdown_connections 2024-04-26T01:51:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.937+0000 7fbe11a00640 1 -- 172.21.15.52:0/3327778063 wait complete. 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.938+0000 7fbe11a00640 1 Processor -- start 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.938+0000 7fbe11a00640 1 -- start start 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c19a770 0x7fbe0c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c10ffd0 con 0x7fbe0c1003e0 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c10fe50 con 0x7fbe0c19a770 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe11a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c110150 con 0x7fbe0c1054b0 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44762/0 (socket says 172.21.15.52:44762) 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe03e00640 1 -- 172.21.15.52:0/4194135749 learned_addr learned my addr 172.21.15.52:0/4194135749 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.939+0000 7fbe0b400640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c19a770 0x7fbe0c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 -- 172.21.15.52:0/4194135749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c19a770 msgr2=0x7fbe0c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c19a770 0x7fbe0c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 -- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 msgr2=0x7fbe0c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 -- 172.21.15.52:0/4194135749 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c19b370 con 0x7fbe0c1054b0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03e00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400e9c0 tx=0x7fbdf400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03400640 1 -- 172.21.15.52:0/4194135749 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf400cd80 con 0x7fbe0c1054b0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c1a7450 con 0x7fbe0c1054b0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03400640 1 -- 172.21.15.52:0/4194135749 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdf40054d0 con 0x7fbe0c1054b0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe03400640 1 -- 172.21.15.52:0/4194135749 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4010430 con 0x7fbe0c1054b0 2024-04-26T01:51:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.940+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c1a7850 con 0x7fbe0c1054b0 2024-04-26T01:51:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.944+0000 7fbe14792740 1 Processor -- start 2024-04-26T01:51:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.944+0000 7fbe03400640 1 -- 172.21.15.52:0/4194135749 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbdf4004fd0 con 0x7fbe0c1054b0 2024-04-26T01:51:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.944+0000 7fbe14792740 1 -- start start 2024-04-26T01:51:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.944+0000 7fbe14792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.944+0000 7fbe14792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5184c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe14792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe14792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5637eb473900 con 0x5637eb50d820 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe14792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637eb419ef0 con 0x5637eb518a00 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe14792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637eb46f8c0 con 0x5637eb50e140 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56924/0 (socket says 172.21.15.52:56924) 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5184c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44768/0 (socket says 172.21.15.52:44768) 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 -- 172.21.15.52:0/3012319598 learned_addr learned my addr 172.21.15.52:0/3012319598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 -- 172.21.15.52:0/3012319598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 msgr2=0x5637eb5184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5184c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 -- 172.21.15.52:0/3012319598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 msgr2=0x5637eb51adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe0b400640 1 -- 172.21.15.52:0/3012319598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637eb186210 con 0x5637eb50d820 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.945+0000 7fbe03e00640 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbe0b400640 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009b20 tx=0x7fbe0400c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b093d202b1f52f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbe03400640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 0x7fbdf00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbdfbe00640 1 -- 172.21.15.52:0/3012319598 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0401b070 con 0x5637eb50d820 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbe03400640 1 -- 172.21.15.52:0/4194135749 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbdf40d0bd0 con 0x7fbe0c1054b0 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbdfbe00640 1 -- 172.21.15.52:0/3012319598 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe04003e20 con 0x5637eb50d820 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbdfbe00640 1 -- 172.21.15.52:0/3012319598 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe040047f0 con 0x5637eb50d820 2024-04-26T01:51:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.946+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 0x7fbdf00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 -- 172.21.15.52:0/3012319598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 msgr2=0x5637eb50dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009b20 tx=0x7fbe0400c040 comp rx=0 tx=0).stop 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 -- 172.21.15.52:0/3012319598 shutdown_connections 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 --2- 172.21.15.52:0/3012319598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb50dc00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 -- 172.21.15.52:0/3012319598 >> 172.21.15.52:0/3012319598 conn(0x5637eb509d30 msgr2=0x5637eb50a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 -- 172.21.15.52:0/3012319598 shutdown_connections 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 0x7fbdf00a6280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc031440 tx=0x7fbdfc005b50 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.947+0000 7fbe14792740 1 -- 172.21.15.52:0/3012319598 wait complete. 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 -- start start 2024-04-26T01:51:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 Processor -- start 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 0x5637eb515940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb5163c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5637eb51b630 con 0x5637eb50e140 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637eb42d310 con 0x5637eb518a00 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637eb51b7b0 con 0x5637eb50d820 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.948+0000 7fbe14792740 10 client.0 _ll_register_callbacks cb 0x5637eb3fbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 0x5637eb515940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56932/0 (socket says 172.21.15.52:56932) 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe0b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb5163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 -- 172.21.15.52:0/2387928188 learned_addr learned my addr 172.21.15.52:0/2387928188 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 0x5637eb515940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44784/0 (socket says 172.21.15.52:44784) 2024-04-26T01:51:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 -- 172.21.15.52:0/2387928188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 msgr2=0x5637eb515940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 0x5637eb515940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 -- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 msgr2=0x5637eb5163c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb5163c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 -- 172.21.15.52:0/2387928188 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637eb252620 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbe03e00640 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdf409cb30 tx=0x7fbdf4005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4249335a1e71309b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbdf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.949+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf409c6e0 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbdf40ce5d0 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe04003e20 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbdf409c500 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 10 client.15750 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5637eb5278a0 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe04003e20 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fbdf40a5380 con 0x5637eb50e140 2024-04-26T01:51:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 10 client.15750 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 10 client.15750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 20 client.15750 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb528d60 0x5637eb52b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 4 client.15750 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.950+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637eb4878b0 con 0x5637eb528d60 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.951+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb528d60 0x5637eb52b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.951+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb528d60 0x5637eb52b150 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=504f05fd26e98b1b server_cookie=786b8798d57a40fe in_seq=0 out_seq=0 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.951+0000 7fbdf8c00640 10 client.15750 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbdf8c00640 1 -- 172.21.15.52:0/2387928188 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637eb4878b0 con 0x5637eb528d60 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbdf8c00640 10 client.15750 handle_command_reply: tid=0 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 client.15750 shutdown 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 msgr2=0x5637eb515e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbdf409cb30 tx=0x7fbdf4005680 comp rx=0 tx=0).stop 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 shutdown_connections 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb528d60 0x5637eb52b150 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb5163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50e140 0x5637eb515e80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 --2- 172.21.15.52:0/2387928188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50d820 0x5637eb515940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbdf8c00640 0 client.15750 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.952+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 >> 172.21.15.52:0/2387928188 conn(0x5637eb509d30 msgr2=0x5637eb50fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.953+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 shutdown_connections 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.953+0000 7fbe14792740 1 -- 172.21.15.52:0/2387928188 wait complete. 2024-04-26T01:51:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.953+0000 7fbe14792740 20 client.15750 trim_cache size 0 max 16384 2024-04-26T01:51:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 Processor -- start 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 -- start start 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb530a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1b2580 con 0x5637eb50d820 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1afe50 con 0x5637eb518a00 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.969+0000 7fbe14792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1adca0 con 0x5637eb50e140 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb530a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56936/0 (socket says 172.21.15.52:56936) 2024-04-26T01:51:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 -- 172.21.15.52:0/3447840641 learned_addr learned my addr 172.21.15.52:0/3447840641 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 -- 172.21.15.52:0/3447840641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 msgr2=0x5637eb5101e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5101e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 -- 172.21.15.52:0/3447840641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 msgr2=0x5637eb530a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb530a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 -- 172.21.15.52:0/3447840641 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637eb510720 con 0x5637eb50d820 2024-04-26T01:51:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe03e00640 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.970+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc02a930 tx=0x7fbdfc03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4929365b3e252e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.971+0000 7fbdfb400640 1 -- 172.21.15.52:0/3447840641 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc035040 con 0x5637eb50d820 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.971+0000 7fbdfb400640 1 -- 172.21.15.52:0/3447840641 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc059070 con 0x5637eb50d820 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.971+0000 7fbdfb400640 1 -- 172.21.15.52:0/3447840641 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc03d450 con 0x5637eb50d820 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.971+0000 7fbe14792740 1 -- 172.21.15.52:0/3447840641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 msgr2=0x5637eb5244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.971+0000 7fbe14792740 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc02a930 tx=0x7fbdfc03f000 comp rx=0 tx=0).stop 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 -- 172.21.15.52:0/3447840641 shutdown_connections 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb530a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb5101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 --2- 172.21.15.52:0/3447840641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb5244f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 -- 172.21.15.52:0/3447840641 >> 172.21.15.52:0/3447840641 conn(0x5637eb509d30 msgr2=0x5637eb508e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 -- 172.21.15.52:0/3447840641 shutdown_connections 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 -- 172.21.15.52:0/3447840641 wait complete. 2024-04-26T01:51:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 -- start start 2024-04-26T01:51:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.972+0000 7fbe14792740 1 Processor -- start 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb51e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1ae310 con 0x5637eb50d820 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1af780 con 0x5637eb518a00 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1ae9e0 con 0x5637eb50e140 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe14792740 10 client.0 _ll_register_callbacks cb 0x7fbdf4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56944/0 (socket says 172.21.15.52:56944) 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe0aa00640 1 -- 172.21.15.52:0/853653962 learned_addr learned my addr 172.21.15.52:0/853653962 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe03e00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb51e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.973+0000 7fbe0b400640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 -- 172.21.15.52:0/853653962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 msgr2=0x5637eb51e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb51e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 -- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 msgr2=0x5637eb51e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 -- 172.21.15.52:0/853653962 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637eb51eeb0 con 0x5637eb50d820 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe0aa00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc031a90 tx=0x7fbdfc005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d676950cacb4101 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbdf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc062020 con 0x5637eb50d820 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fbdfc004d40 con 0x5637eb50d820 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.974+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdfc059070 con 0x5637eb50d820 2024-04-26T01:51:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbdfc0533b0 con 0x5637eb50d820 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 10 client.15756 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5637eb58f2c0 con 0x5637eb50d820 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdfc059070 con 0x5637eb50d820 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fbdfc03d980 con 0x5637eb50d820 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 10 client.15756 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 20 client.15756 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb58f9d0 0x5637eb591dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 4 client.15756 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5637eb481320 con 0x5637eb58f9d0 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.975+0000 7fbe03e00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb58f9d0 0x5637eb591dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.976+0000 7fbe03e00640 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb58f9d0 0x5637eb591dc0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d255f9d8cbdea69 server_cookie=ad685ed808b6b180 in_seq=0 out_seq=0 2024-04-26T01:51:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:22.976+0000 7fbdf8c00640 10 client.15756 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.002+0000 7fbdf8c00640 1 -- 172.21.15.52:0/853653962 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5637eb481320 con 0x5637eb58f9d0 2024-04-26T01:51:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbdf8c00640 10 client.15756 handle_command_reply: tid=0 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 client.15756 shutdown 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 msgr2=0x5637eb520db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc031a90 tx=0x7fbdfc005cb0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 shutdown_connections 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637eb58f9d0 0x5637eb591dc0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637eb518a00 0x5637eb51e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637eb50e140 0x5637eb51e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 --2- 172.21.15.52:0/853653962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637eb50d820 0x5637eb520db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbdf8c00640 0 client.15756 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 >> 172.21.15.52:0/853653962 conn(0x5637eb509d30 msgr2=0x5637eb508e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 shutdown_connections 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 1 -- 172.21.15.52:0/853653962 wait complete. 2024-04-26T01:51:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.003+0000 7fbe14792740 20 client.15756 trim_cache size 0 max 16384 2024-04-26T01:51:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:51:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:51:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:51:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 msgr2=0x7fbdf00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 0x7fbdf00a6280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc031440 tx=0x7fbdfc005b50 comp rx=0 tx=0).stop 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 msgr2=0x7fbe0c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400e9c0 tx=0x7fbdf400ee80 comp rx=0 tx=0).stop 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 shutdown_connections 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbdf00a3dc0 0x7fbdf00a6280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe0c19a770 0x7fbe0c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe0c1054b0 0x7fbe0c1a0bf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 --2- 172.21.15.52:0/4194135749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe0c1003e0 0x7fbe0c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 >> 172.21.15.52:0/4194135749 conn(0x7fbe0c076f50 msgr2=0x7fbe0c103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.004+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 shutdown_connections 2024-04-26T01:51:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:23.005+0000 7fbe11a00640 1 -- 172.21.15.52:0/4194135749 wait complete. 2024-04-26T01:51:23.094 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.22 MB/sec execute 162 sec latency 19367.847 ms 2024-04-26T01:51:23.580 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:23 smithi052 ceph-mon[27851]: pgmap v195: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-26T01:51:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:23 smithi191 ceph-mon[32745]: pgmap v195: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-26T01:51:23.654 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '93ad249f', quiesce took 22.1 seconds. Will release after: 10 2024-04-26T01:51:23.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:23 smithi183 ceph-mon[32752]: pgmap v195: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-26T01:51:24.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.20 MB/sec execute 163 sec latency 20367.996 ms 2024-04-26T01:51:25.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.18 MB/sec execute 164 sec latency 21368.145 ms 2024-04-26T01:51:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:25 smithi191 ceph-mon[32745]: pgmap v196: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 568 KiB/s wr, 33 op/s 2024-04-26T01:51:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:25 smithi183 ceph-mon[32752]: pgmap v196: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 568 KiB/s wr, 33 op/s 2024-04-26T01:51:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:25 smithi052 ceph-mon[27851]: pgmap v196: 97 pgs: 97 active+clean; 67 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 568 KiB/s wr, 33 op/s 2024-04-26T01:51:26.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.16 MB/sec execute 165 sec latency 22368.288 ms 2024-04-26T01:51:27.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.14 MB/sec execute 166 sec latency 23368.422 ms 2024-04-26T01:51:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:27 smithi191 ceph-mon[32745]: pgmap v197: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 981 KiB/s wr, 55 op/s 2024-04-26T01:51:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:27 smithi183 ceph-mon[32752]: pgmap v197: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 981 KiB/s wr, 55 op/s 2024-04-26T01:51:27.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:27 smithi052 ceph-mon[27851]: pgmap v197: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 981 KiB/s wr, 55 op/s 2024-04-26T01:51:28.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.12 MB/sec execute 167 sec latency 24368.561 ms 2024-04-26T01:51:29.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.10 MB/sec execute 168 sec latency 25368.652 ms 2024-04-26T01:51:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:29 smithi191 ceph-mon[32745]: pgmap v198: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:29.667 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:29 smithi052 ceph-mon[27851]: pgmap v198: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:29.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:29 smithi183 ceph-mon[32752]: pgmap v198: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:30.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.08 MB/sec execute 169 sec latency 26368.742 ms 2024-04-26T01:51:31.095 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.07 MB/sec execute 170 sec latency 27368.880 ms 2024-04-26T01:51:31.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:31 smithi191 ceph-mon[32745]: pgmap v199: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:31 smithi183 ceph-mon[32752]: pgmap v199: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:31.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:31 smithi052 ceph-mon[27851]: pgmap v199: 97 pgs: 97 active+clean; 67 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 698 KiB/s wr, 41 op/s 2024-04-26T01:51:32.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.05 MB/sec execute 171 sec latency 28369.042 ms 2024-04-26T01:51:33.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.03 MB/sec execute 172 sec latency 29369.253 ms 2024-04-26T01:51:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:33 smithi191 ceph-mon[32745]: pgmap v200: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-04-26T01:51:33.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:51:33.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:33 smithi183 ceph-mon[32752]: pgmap v200: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-04-26T01:51:33.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:33 smithi052 ceph-mon[27851]: pgmap v200: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 1.1 MiB/s wr, 56 op/s 2024-04-26T01:51:34.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.01 MB/sec execute 173 sec latency 30369.340 ms 2024-04-26T01:51:34.126 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:35.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 3.00 MB/sec execute 174 sec latency 31369.436 ms 2024-04-26T01:51:35.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:35 smithi052 ceph-mon[27851]: pgmap v201: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 804 KiB/s wr, 37 op/s 2024-04-26T01:51:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:35 smithi191 ceph-mon[32745]: pgmap v201: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 804 KiB/s wr, 37 op/s 2024-04-26T01:51:35.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:35 smithi183 ceph-mon[32752]: pgmap v201: 97 pgs: 97 active+clean; 67 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 804 KiB/s wr, 37 op/s 2024-04-26T01:51:36.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 2.98 MB/sec execute 175 sec latency 32369.565 ms 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.154+0000 7f3f7b200640 1 -- 172.21.15.52:0/3153998162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 msgr2=0x7f3f741012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.154+0000 7f3f7b200640 1 --2- 172.21.15.52:0/3153998162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f741012b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009900 tx=0x7f3f6802a410 comp rx=0 tx=0).stop 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 -- 172.21.15.52:0/3153998162 shutdown_connections 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 --2- 172.21.15.52:0/3153998162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f74105fa0 0x7f3f7410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 --2- 172.21.15.52:0/3153998162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74101880 0x7f3f741058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 --2- 172.21.15.52:0/3153998162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f741012b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 -- 172.21.15.52:0/3153998162 >> 172.21.15.52:0/3153998162 conn(0x7f3f740fc730 msgr2=0x7f3f740feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 -- 172.21.15.52:0/3153998162 shutdown_connections 2024-04-26T01:51:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.155+0000 7f3f7b200640 1 -- 172.21.15.52:0/3153998162 wait complete. 2024-04-26T01:51:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 Processor -- start 2024-04-26T01:51:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 -- start start 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74105fa0 0x7f3f74102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 0x7f3f741a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74110f00 con 0x7f3f74100ed0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74110d80 con 0x7f3f741033e0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.156+0000 7f3f7b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74111080 con 0x7f3f74105fa0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32954/0 (socket says 172.21.15.52:32954) 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 -- 172.21.15.52:0/4057330974 learned_addr learned my addr 172.21.15.52:0/4057330974 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f7a800640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 0x7f3f741a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79400640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74105fa0 0x7f3f74102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 -- 172.21.15.52:0/4057330974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74105fa0 msgr2=0x7f3f74102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74105fa0 0x7f3f74102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 -- 172.21.15.52:0/4057330974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 msgr2=0x7f3f741a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 0x7f3f741a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f79e00640 1 -- 172.21.15.52:0/4057330974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f741a6ea0 con 0x7f3f74100ed0 2024-04-26T01:51:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.157+0000 7f3f7a800640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 0x7f3f741a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f79e00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3f68005940 tx=0x7f3f68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f6ea00640 1 -- 172.21.15.52:0/4057330974 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6803f030 con 0x7f3f74100ed0 2024-04-26T01:51:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f6ea00640 1 -- 172.21.15.52:0/4057330974 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f68003aa0 con 0x7f3f74100ed0 2024-04-26T01:51:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f741a7130 con 0x7f3f74100ed0 2024-04-26T01:51:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f6ea00640 1 -- 172.21.15.52:0/4057330974 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68035070 con 0x7f3f74100ed0 2024-04-26T01:51:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.158+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f741a7670 con 0x7f3f74100ed0 2024-04-26T01:51:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 Processor -- start 2024-04-26T01:51:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 -- start start 2024-04-26T01:51:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f6ea00640 1 -- 172.21.15.52:0/4057330974 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3f68042030 con 0x7f3f74100ed0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc852d0 0x55624fc8f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc91f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55624fbea900 con 0x55624fc8fb90 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55624fb90ef0 con 0x55624fc849b0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.162+0000 7f3f7f604740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55624fbe68c0 con 0x55624fc852d0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f79400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc91f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47806/0 (socket says 172.21.15.52:47806) 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 -- 172.21.15.52:0/3865068703 learned_addr learned my addr 172.21.15.52:0/3865068703 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f79e00640 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc852d0 0x55624fc8f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 -- 172.21.15.52:0/3865068703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc852d0 msgr2=0x55624fc8f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc852d0 0x55624fc8f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 -- 172.21.15.52:0/3865068703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 msgr2=0x55624fc91f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc91f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f7a800640 1 -- 172.21.15.52:0/3865068703 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55624f8fd210 con 0x55624fc849b0 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.163+0000 7f3f79400640 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc91f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.164+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011790 tx=0x7f3f70011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5873924935454713 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.164+0000 7f3f63000640 1 -- 172.21.15.52:0/3865068703 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70003c30 con 0x55624fc849b0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.164+0000 7f3f63000640 1 -- 172.21.15.52:0/3865068703 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7000fe50 con 0x55624fc849b0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.164+0000 7f3f63000640 1 -- 172.21.15.52:0/3865068703 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70004840 con 0x55624fc849b0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.164+0000 7f3f6ea00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 0x7f3f5c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.165+0000 7f3f79e00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 0x7f3f5c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.165+0000 7f3f6ea00640 1 -- 172.21.15.52:0/4057330974 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f3f680f0da0 con 0x7f3f74100ed0 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.165+0000 7f3f7f604740 1 -- 172.21.15.52:0/3865068703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 msgr2=0x55624fc84d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.165+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011790 tx=0x7f3f70011c50 comp rx=0 tx=0).stop 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.165+0000 7f3f7f604740 1 -- 172.21.15.52:0/3865068703 shutdown_connections 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc91f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc852d0 0x55624fc8f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3865068703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc84d90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 -- 172.21.15.52:0/3865068703 >> 172.21.15.52:0/3865068703 conn(0x55624fc80ec0 msgr2=0x55624fc812a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 -- 172.21.15.52:0/3865068703 shutdown_connections 2024-04-26T01:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f7f604740 1 -- 172.21.15.52:0/3865068703 wait complete. 2024-04-26T01:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.166+0000 7f3f79e00640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 0x7f3f5c0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55624fc85f50 tx=0x7f3f680f4850 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 -- start start 2024-04-26T01:51:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 Processor -- start 2024-04-26T01:51:36.167 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc8cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 0x55624fc8d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fc8d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55624fc927c0 con 0x55624fc852d0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55624fba4310 con 0x55624fc849b0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.167+0000 7f3f7f604740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55624fc92940 con 0x55624fc8fb90 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7f604740 10 client.0 _ll_register_callbacks cb 0x55624fb72cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc8cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc8cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47822/0 (socket says 172.21.15.52:47822) 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 -- 172.21.15.52:0/745213199 learned_addr learned my addr 172.21.15.52:0/745213199 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fc8d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f79400640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 0x55624fc8d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 -- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 msgr2=0x55624fc8cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc8cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 -- 172.21.15.52:0/745213199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 msgr2=0x55624fc8d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 0x55624fc8d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.168+0000 7f3f7a800640 1 -- 172.21.15.52:0/745213199 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55624f9c9620 con 0x55624fc8fb90 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f79400640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 0x55624fc8d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f7a800640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fc8d5a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3f700115a0 tx=0x7f3f70005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14db589d45c08d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f57400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70003a30 con 0x55624fc8fb90 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7000fe50 con 0x55624fc8fb90 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.169+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f3f70003bd0 con 0x55624fc8fb90 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3f70003dd0 con 0x55624fc8fb90 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 10 client.25882 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:36.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55624fc9ea60 con 0x55624fc8fb90 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f7000fe50 con 0x55624fc8fb90 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3f70018070 con 0x55624fc8fb90 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 10 client.25882 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 20 client.25882 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:36.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fc9fe60 0x55624fca2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 4 client.25882 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.170+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55624fbfe8b0 con 0x55624fc9fe60 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.171+0000 7f3f79400640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fc9fe60 0x55624fca2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.171+0000 7f3f79400640 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fc9fe60 0x55624fca2250 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cd588a19aa7964 server_cookie=2f7f5502634f8e3e in_seq=0 out_seq=0 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.171+0000 7f3f57400640 10 client.25882 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.172+0000 7f3f57400640 1 -- 172.21.15.52:0/745213199 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55624fbfe8b0 con 0x55624fc9fe60 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.172+0000 7f3f57400640 10 client.25882 handle_command_reply: tid=0 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.172+0000 7f3f7f604740 1 client.25882 shutdown 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 msgr2=0x55624fc8d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fc8d5a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3f700115a0 tx=0x7f3f70005fd0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 shutdown_connections 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fc9fe60 0x55624fca2250 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fc8d5a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc852d0 0x55624fc8d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 --2- 172.21.15.52:0/745213199 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc849b0 0x55624fc8cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 >> 172.21.15.52:0/745213199 conn(0x55624fc80ec0 msgr2=0x55624fc86bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f57400640 0 client.25882 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 shutdown_connections 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 1 -- 172.21.15.52:0/745213199 wait complete. 2024-04-26T01:51:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.173+0000 7f3f7f604740 20 client.25882 trim_cache size 0 max 16384 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.189+0000 7f3f7f604740 1 Processor -- start 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.189+0000 7f3f7f604740 1 -- start start 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc8d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fca7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741b24b0 con 0x55624fc849b0 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741afd80 con 0x55624fc852d0 2024-04-26T01:51:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7f604740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741adbf0 con 0x55624fc8fb90 2024-04-26T01:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32980/0 (socket says 172.21.15.52:32980) 2024-04-26T01:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f79400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc8d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.190+0000 7f3f7a800640 1 -- 172.21.15.52:0/2118799143 learned_addr learned my addr 172.21.15.52:0/2118799143 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f7a800640 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fca7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 -- 172.21.15.52:0/2118799143 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 msgr2=0x55624fca7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fca7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 -- 172.21.15.52:0/2118799143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 msgr2=0x55624fc9b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 -- 172.21.15.52:0/2118799143 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55624fc7b400 con 0x55624fc852d0 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f7a800640 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f79400640 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc8d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55624fc822e0 tx=0x7f3f6400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff77213f4e76f309 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f62600640 1 -- 172.21.15.52:0/2118799143 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64003f00 con 0x55624fc852d0 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f62600640 1 -- 172.21.15.52:0/2118799143 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f640054d0 con 0x55624fc852d0 2024-04-26T01:51:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.191+0000 7f3f62600640 1 -- 172.21.15.52:0/2118799143 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6401f350 con 0x55624fc852d0 2024-04-26T01:51:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 -- 172.21.15.52:0/2118799143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 msgr2=0x55624fc8d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc8d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55624fc822e0 tx=0x7f3f6400b390 comp rx=0 tx=0).stop 2024-04-26T01:51:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 -- 172.21.15.52:0/2118799143 shutdown_connections 2024-04-26T01:51:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc8fb90 0x55624fca7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc8d3c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 --2- 172.21.15.52:0/2118799143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc849b0 0x55624fc9b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 -- 172.21.15.52:0/2118799143 >> 172.21.15.52:0/2118799143 conn(0x55624fc80ec0 msgr2=0x55624fc80020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 -- 172.21.15.52:0/2118799143 shutdown_connections 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.192+0000 7f3f7f604740 1 -- 172.21.15.52:0/2118799143 wait complete. 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.193+0000 7f3f7f604740 1 -- start start 2024-04-26T01:51:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.193+0000 7f3f7f604740 1 Processor -- start 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc849b0 0x55624fc980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc95c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741ae240 con 0x55624fc8fb90 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741af6b0 con 0x55624fc852d0 2024-04-26T01:51:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741ae910 con 0x55624fc849b0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7f604740 10 client.0 _ll_register_callbacks cb 0x7f3f700048f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc849b0 0x55624fc980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f79400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f79400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47838/0 (socket says 172.21.15.52:47838) 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f79400640 1 -- 172.21.15.52:0/3343679423 learned_addr learned my addr 172.21.15.52:0/3343679423 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.194+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc95c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 -- 172.21.15.52:0/3343679423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc849b0 msgr2=0x55624fc980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc849b0 0x55624fc980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 -- 172.21.15.52:0/3343679423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 msgr2=0x55624fc95c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc95c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 -- 172.21.15.52:0/3343679423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55624fd05210 con 0x55624fc852d0 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc95c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f79400640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400b7c0 tx=0x7f3f640045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c68498de3c91264a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f57400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64003d00 con 0x55624fc852d0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f640054d0 con 0x55624fc852d0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.195+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f3f64003ea0 con 0x55624fc852d0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.196+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3f64037050 con 0x55624fc852d0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.196+0000 7f3f7f604740 10 client.25880 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.196+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55624fd06360 con 0x55624fc852d0 2024-04-26T01:51:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.196+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f640054d0 con 0x55624fc852d0 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3f6401f450 con 0x55624fc852d0 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 10 client.25880 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 10 client.25880 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 20 client.25880 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fd06ae0 0x55624fd08f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 4 client.25880 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55624fbf8320 con 0x55624fd06ae0 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.197+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fd06ae0 0x55624fd08f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.198+0000 7f3f7a800640 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fd06ae0 0x55624fd08f00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ad232fe90dcbf4b server_cookie=897191e5587eb3b8 in_seq=0 out_seq=0 2024-04-26T01:51:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.198+0000 7f3f57400640 10 client.25880 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f57400640 1 -- 172.21.15.52:0/3343679423 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55624fbf8320 con 0x55624fd06ae0 2024-04-26T01:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f57400640 10 client.25880 handle_command_reply: tid=0 2024-04-26T01:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 client.25880 shutdown 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 msgr2=0x55624fc954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400b7c0 tx=0x7f3f640045e0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 shutdown_connections 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55624fd06ae0 0x55624fd08f00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55624fc8fb90 0x55624fc95c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55624fc852d0 0x55624fc954d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 --2- 172.21.15.52:0/3343679423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55624fc849b0 0x55624fc980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f57400640 0 client.25880 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.217+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 >> 172.21.15.52:0/3343679423 conn(0x55624fc80ec0 msgr2=0x55624fc7f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 shutdown_connections 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7f604740 1 -- 172.21.15.52:0/3343679423 wait complete. 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7f604740 20 client.25880 trim_cache size 0 max 16384 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:51:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 msgr2=0x7f3f5c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 0x7f3f5c0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55624fc85f50 tx=0x7f3f680f4850 comp rx=0 tx=0).stop 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 msgr2=0x7f3f74102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.218+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3f68005940 tx=0x7f3f68004270 comp rx=0 tx=0).stop 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 shutdown_connections 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f5c0a3dc0 0x7f3f5c0a6280 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f741033e0 0x7f3f741a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f74105fa0 0x7f3f74102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 --2- 172.21.15.52:0/4057330974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f74100ed0 0x7f3f74102960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 >> 172.21.15.52:0/4057330974 conn(0x7f3f740fc730 msgr2=0x7f3f7406a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 shutdown_connections 2024-04-26T01:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:36.219+0000 7f3f7b200640 1 -- 172.21.15.52:0/4057330974 wait complete. 2024-04-26T01:51:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.482+0000 7fec57a00700 1 -- 172.21.15.52:0/3831746251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006a750 msgr2=0x7fec5006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.482+0000 7fec57a00700 1 --2- 172.21.15.52:0/3831746251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006a750 0x7fec5006abb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009990 tx=0x7fec3c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 -- 172.21.15.52:0/3831746251 shutdown_connections 2024-04-26T01:51:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 --2- 172.21.15.52:0/3831746251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec5006b0f0 0x7fec5010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 --2- 172.21.15.52:0/3831746251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec50108a30 0x7fec50108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 --2- 172.21.15.52:0/3831746251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006a750 0x7fec5006abb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 -- 172.21.15.52:0/3831746251 >> 172.21.15.52:0/3831746251 conn(0x7fec500788f0 msgr2=0x7fec50078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 -- 172.21.15.52:0/3831746251 shutdown_connections 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.483+0000 7fec57a00700 1 -- 172.21.15.52:0/3831746251 wait complete. 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.484+0000 7fec57a00700 1 Processor -- start 2024-04-26T01:51:36.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 -- start start 2024-04-26T01:51:36.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 0x7fec501991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec50108a30 0x7fec5019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fec50112c60 con 0x7fec5006b0f0 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec50112ae0 con 0x7fec5006a750 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec57a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec50112de0 con 0x7fec50108a30 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec4fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec4fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33000/0 (socket says 172.21.15.52:33000) 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec4fe00700 1 -- 172.21.15.52:0/49035058 learned_addr learned my addr 172.21.15.52:0/49035058 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.485+0000 7fec55000700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 0x7fec501991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec50108a30 0x7fec5019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 -- 172.21.15.52:0/49035058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec50108a30 msgr2=0x7fec5019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec50108a30 0x7fec5019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 -- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 msgr2=0x7fec50199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 -- 172.21.15.52:0/49035058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5019e190 con 0x7fec5006a750 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec4fe00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.486+0000 7fec55000700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 0x7fec501991d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fec4400e9f0 tx=0x7fec4400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.487+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4400c9c0 con 0x7fec5006a750 2024-04-26T01:51:36.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.487+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4400cb60 con 0x7fec5006a750 2024-04-26T01:51:36.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.487+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5019e480 con 0x7fec5006a750 2024-04-26T01:51:36.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.487+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44018680 con 0x7fec5006a750 2024-04-26T01:51:36.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.487+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec501a5d00 con 0x7fec5006a750 2024-04-26T01:51:36.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.488+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec50069730 con 0x7fec5006a750 2024-04-26T01:51:36.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.490+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fec44010430 con 0x7fec5006a750 2024-04-26T01:51:36.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.491+0000 7fec4d600700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 0x7fec380a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:36.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.492+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fec440d09c0 con 0x7fec5006a750 2024-04-26T01:51:36.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.492+0000 7fec4fe00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 0x7fec380a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:36.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.493+0000 7fec4fe00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 0x7fec380a62f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec50108030 tx=0x7fec3c00b280 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:36.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.494+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec4409d030 con 0x7fec5006a750 2024-04-26T01:51:36.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.786+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec5002df80 con 0x7fec5006a750 2024-04-26T01:51:36.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.787+0000 7fec4d600700 1 -- 172.21.15.52:0/49035058 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7fec4409c930 con 0x7fec5006a750 2024-04-26T01:51:36.786 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:51:36.786 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:51:36.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 msgr2=0x7fec380a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 0x7fec380a62f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec50108030 tx=0x7fec3c00b280 comp rx=0 tx=0).stop 2024-04-26T01:51:36.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 msgr2=0x7fec501991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:36.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 0x7fec501991d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fec4400e9f0 tx=0x7fec4400ed00 comp rx=0 tx=0).stop 2024-04-26T01:51:36.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 shutdown_connections 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec380a3e30 0x7fec380a62f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec50108a30 0x7fec5019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec5006a750 0x7fec501991d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 --2- 172.21.15.52:0/49035058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec5006b0f0 0x7fec50199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 >> 172.21.15.52:0/49035058 conn(0x7fec500788f0 msgr2=0x7fec500ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.791+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 shutdown_connections 2024-04-26T01:51:36.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:36.792+0000 7fec57a00700 1 -- 172.21.15.52:0/49035058 wait complete. 2024-04-26T01:51:36.791 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:51:37.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 2.96 MB/sec execute 176 sec latency 33369.687 ms 2024-04-26T01:51:37.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:37 smithi052 ceph-mon[27851]: pgmap v202: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-04-26T01:51:37.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:37 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/49035058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:37 smithi183 ceph-mon[32752]: pgmap v202: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-04-26T01:51:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:37 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/49035058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:37.690 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 93ad249f --release --await' 2024-04-26T01:51:37.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 93ad249f --release --await 2024-04-26T01:51:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:37 smithi191 ceph-mon[32745]: pgmap v202: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-04-26T01:51:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:37 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/49035058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:51:38.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 2.95 MB/sec execute 177 sec latency 34369.768 ms 2024-04-26T01:51:38.165 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:51:39.096 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 2.93 MB/sec execute 178 sec latency 35369.909 ms 2024-04-26T01:51:39.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:39 smithi183 ceph-mon[32752]: pgmap v203: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:39 smithi052 ceph-mon[27851]: pgmap v203: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:39.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:39 smithi191 ceph-mon[32745]: pgmap v203: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:40.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150403 2.91 MB/sec execute 179 sec latency 36370.030 ms 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 -- 172.21.15.52:0/3247237968 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c076f70 msgr2=0x7f596c077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 --2- 172.21.15.52:0/3247237968 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c076f70 0x7f596c077350 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f595c009990 tx=0x7f595c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 -- 172.21.15.52:0/3247237968 shutdown_connections 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 --2- 172.21.15.52:0/3247237968 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c113920 0x7f596c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 --2- 172.21.15.52:0/3247237968 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c076f70 0x7f596c077350 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 --2- 172.21.15.52:0/3247237968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c077890 0x7f596c1133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 -- 172.21.15.52:0/3247237968 >> 172.21.15.52:0/3247237968 conn(0x7f596c100770 msgr2=0x7f596c102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 -- 172.21.15.52:0/3247237968 shutdown_connections 2024-04-26T01:51:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.475+0000 7f5971a00700 1 -- 172.21.15.52:0/3247237968 wait complete. 2024-04-26T01:51:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.476+0000 7f5971a00700 1 Processor -- start 2024-04-26T01:51:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 -- start start 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c076f70 0x7f596c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 0x7f596c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f596c119310 con 0x7f596c077890 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f596c119190 con 0x7f596c076f70 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f5971a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f596c119490 con 0x7f596c113920 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f596a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.477+0000 7f596a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33018/0 (socket says 172.21.15.52:33018) 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596a000700 1 -- 172.21.15.52:0/2120255996 learned_addr learned my addr 172.21.15.52:0/2120255996 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 0x7f596c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c076f70 0x7f596c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 -- 172.21.15.52:0/2120255996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c076f70 msgr2=0x7f596c1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c076f70 0x7f596c1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 -- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 msgr2=0x7f596c1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 -- 172.21.15.52:0/2120255996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c19fd90 con 0x7f596c113920 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596a000700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.478+0000 7f596b400700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 0x7f596c19f630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f596c07b260 tx=0x7f596400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.479+0000 7f5963400700 1 -- 172.21.15.52:0/2120255996 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964004db0 con 0x7f596c113920 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.479+0000 7f5963400700 1 -- 172.21.15.52:0/2120255996 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5964003a00 con 0x7f596c113920 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.479+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596c1a0080 con 0x7f596c113920 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.479+0000 7f5963400700 1 -- 172.21.15.52:0/2120255996 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59640044b0 con 0x7f596c113920 2024-04-26T01:51:40.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.479+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596c1ac2f0 con 0x7f596c113920 2024-04-26T01:51:40.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.484+0000 7f5963400700 1 -- 172.21.15.52:0/2120255996 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f596401c020 con 0x7f596c113920 2024-04-26T01:51:40.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.485+0000 7f597c404b80 1 Processor -- start 2024-04-26T01:51:40.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.485+0000 7f597c404b80 1 -- start start 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f5963400700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 0x7f59500a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da620c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da6216f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557da615ef90 con 0x557da62174d0 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557da6105830 con 0x557da620bd40 2024-04-26T01:51:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f597c404b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557da615af50 con 0x557da620c660 2024-04-26T01:51:40.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f5963400700 1 -- 172.21.15.52:0/2120255996 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f59640cff20 con 0x7f596c113920 2024-04-26T01:51:40.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596a000700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 0x7f59500a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da620c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33024/0 (socket says 172.21.15.52:33024) 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da620c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47896/0 (socket says 172.21.15.52:47896) 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596b400700 1 -- 172.21.15.52:0/3871649892 learned_addr learned my addr 172.21.15.52:0/3871649892 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.486+0000 7f596a000700 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da6216f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 -- 172.21.15.52:0/3871649892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 msgr2=0x557da6216f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da6216f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 -- 172.21.15.52:0/3871649892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 msgr2=0x557da620c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da620c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 -- 172.21.15.52:0/3871649892 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557da60fb780 con 0x557da62174d0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596b400700 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5964098860 tx=0x7f596409b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8f06db91dafef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f596a000700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 0x7f59500a6270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59580098a0 tx=0x7f5958009e30 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f5957e00700 1 -- 172.21.15.52:0/3871649892 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964019040 con 0x557da62174d0 2024-04-26T01:51:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f5957e00700 1 -- 172.21.15.52:0/3871649892 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59640a0240 con 0x557da62174d0 2024-04-26T01:51:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.487+0000 7f5957e00700 1 -- 172.21.15.52:0/3871649892 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59640cdac0 con 0x557da62174d0 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 -- 172.21.15.52:0/3871649892 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 msgr2=0x557da62198c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5964098860 tx=0x7f596409b7f0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 -- 172.21.15.52:0/3871649892 shutdown_connections 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62198c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da620c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 --2- 172.21.15.52:0/3871649892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da6216f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 -- 172.21.15.52:0/3871649892 >> 172.21.15.52:0/3871649892 conn(0x557da613b050 msgr2=0x557da6208250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 -- 172.21.15.52:0/3871649892 shutdown_connections 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.488+0000 7f597c404b80 1 -- 172.21.15.52:0/3871649892 wait complete. 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.489+0000 7f597c404b80 1 -- start start 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.489+0000 7f597c404b80 1 Processor -- start 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620c660 0x557da6213f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62144a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557da6219fc0 con 0x557da62174d0 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557da6118960 con 0x557da620c660 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557da621a140 con 0x557da620bd40 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f597c404b80 10 client.0 _ll_register_callbacks cb 0x557da60f7820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f596aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f596aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43526/0 (socket says 172.21.15.52:43526) 2024-04-26T01:51:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f596aa00700 1 -- 172.21.15.52:0/1154044120 learned_addr learned my addr 172.21.15.52:0/1154044120 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f596b400700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.490+0000 7f596a000700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620c660 0x557da6213f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 -- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620c660 msgr2=0x557da6213f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620c660 0x557da6213f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 -- 172.21.15.52:0/1154044120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 msgr2=0x557da62144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 -- 172.21.15.52:0/1154044120 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557da60ac9d0 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596b400700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f596aa00700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f595c0057e0 tx=0x7f595c005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78c0b2c50330400d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f5954c00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c029060 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59640a0240 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.491+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f595c00bd30 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f595c003680 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f597c404b80 10 client.25906 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557da60f5bf0 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59640a0240 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f595c024890 con 0x557da620bd40 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f597c404b80 10 client.25906 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.492+0000 7f597c404b80 10 client.25906 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.493+0000 7f597c404b80 20 client.25906 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.493+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da6226db0 0x557da6229220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.493+0000 7f597c404b80 4 client.25906 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:40.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.493+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557da616c780 con 0x557da6226db0 2024-04-26T01:51:40.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.493+0000 7f596a000700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da6226db0 0x557da6229220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.494+0000 7f596a000700 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da6226db0 0x557da6229220 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50b60ad67c625ac server_cookie=f45acb74985f39f5 in_seq=0 out_seq=0 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.494+0000 7f5954c00700 10 client.25906 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f5954c00700 1 -- 172.21.15.52:0/1154044120 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557da616c780 con 0x557da6226db0 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f5954c00700 10 client.25906 handle_command_reply: tid=0 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 client.25906 shutdown 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 msgr2=0x557da6216bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f595c0057e0 tx=0x7f595c005810 comp rx=0 tx=0).stop 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 shutdown_connections 2024-04-26T01:51:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da6226db0 0x557da6229220 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da62144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da6216bf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 --2- 172.21.15.52:0/1154044120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620c660 0x557da6213f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f5954c00700 0 client.25906 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 >> 172.21.15.52:0/1154044120 conn(0x557da613b050 msgr2=0x557da6207e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 shutdown_connections 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 1 -- 172.21.15.52:0/1154044120 wait complete. 2024-04-26T01:51:40.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.495+0000 7f597c404b80 20 client.25906 trim_cache size 0 max 16384 2024-04-26T01:51:40.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.526+0000 7f597c404b80 1 Processor -- start 2024-04-26T01:51:40.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.526+0000 7f597c404b80 1 -- start start 2024-04-26T01:51:40.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da6212f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da62083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b6070 con 0x557da62174d0 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b3900 con 0x557da620bd40 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f597c404b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b1750 con 0x557da620c660 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f596b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f596b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33044/0 (socket says 172.21.15.52:33044) 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f596b400700 1 -- 172.21.15.52:0/1444503468 learned_addr learned my addr 172.21.15.52:0/1444503468 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f596aa00700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da6212f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.527+0000 7f596a000700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da62083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 -- 172.21.15.52:0/1444503468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 msgr2=0x557da62083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da62083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 -- 172.21.15.52:0/1444503468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 msgr2=0x557da6212f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da6212f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 -- 172.21.15.52:0/1444503468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557da62298a0 con 0x557da62174d0 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596a000700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da62083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596aa00700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da6212f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f596b400700 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f596409bb90 tx=0x7f596409ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604387c9e20307bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f5957400700 1 -- 172.21.15.52:0/1444503468 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964019040 con 0x557da62174d0 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f5957400700 1 -- 172.21.15.52:0/1444503468 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59640a4b00 con 0x557da62174d0 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.528+0000 7f5957400700 1 -- 172.21.15.52:0/1444503468 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596401c310 con 0x557da62174d0 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 -- 172.21.15.52:0/1444503468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 msgr2=0x557da6222ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f596409bb90 tx=0x7f596409ba80 comp rx=0 tx=0).stop 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 -- 172.21.15.52:0/1444503468 shutdown_connections 2024-04-26T01:51:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da62174d0 0x557da6222ec0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da620bd40 0x557da6212f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 --2- 172.21.15.52:0/1444503468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620c660 0x557da62083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 -- 172.21.15.52:0/1444503468 >> 172.21.15.52:0/1444503468 conn(0x557da613b050 msgr2=0x557da6207c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 -- 172.21.15.52:0/1444503468 shutdown_connections 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.529+0000 7f597c404b80 1 -- 172.21.15.52:0/1444503468 wait complete. 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.530+0000 7f597c404b80 1 -- start start 2024-04-26T01:51:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.530+0000 7f597c404b80 1 Processor -- start 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da620c660 0x557da621c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da62174d0 0x557da621ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b1dc0 con 0x557da620c660 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b3230 con 0x557da62174d0 2024-04-26T01:51:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b2490 con 0x557da620bd40 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f597c404b80 10 client.0 _ll_register_callbacks cb 0x557da6219e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f596aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f596a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da620c660 0x557da621c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f596aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43544/0 (socket says 172.21.15.52:43544) 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f596aa00700 1 -- 172.21.15.52:0/1276416064 learned_addr learned my addr 172.21.15.52:0/1276416064 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.531+0000 7f596b400700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da62174d0 0x557da621ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 -- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da62174d0 msgr2=0x557da621ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da62174d0 0x557da621ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 -- 172.21.15.52:0/1276416064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da620c660 msgr2=0x557da621c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da620c660 0x557da621c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 -- 172.21.15.52:0/1276416064 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557da6208d20 con 0x557da620bd40 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f596aa00700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f595c005580 tx=0x7f595c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be72d64bd0dda69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f5954c00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c029050 con 0x557da620bd40 2024-04-26T01:51:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f595c024480 con 0x557da620bd40 2024-04-26T01:51:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.532+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59640a4b00 con 0x557da620bd40 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f595c024680 con 0x557da620bd40 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 10 client.25912 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557da628e3b0 con 0x557da620bd40 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59640a4b00 con 0x557da620bd40 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f595c00f710 con 0x557da620bd40 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 10 client.25912 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 10 client.25912 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 20 client.25912 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da628e9c0 0x557da6290de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 4 client.25912 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "93ad249f", "release": true, "await": true}] 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.533+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "93ad249f", "release": true, "await": true}) -- 0x557da6216680 con 0x557da628e9c0 2024-04-26T01:51:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.534+0000 7f596a000700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da628e9c0 0x557da6290de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.534+0000 7f596a000700 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da628e9c0 0x557da6290de0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cefd938a17cfa8b3 server_cookie=1a18af20acc0def6 in_seq=0 out_seq=0 2024-04-26T01:51:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.534+0000 7f5954c00700 10 client.25912 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:40.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f5954c00700 1 -- 172.21.15.52:0/1276416064 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557da6216680 con 0x557da628e9c0 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f5954c00700 10 client.25912 handle_command_reply: tid=0 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 client.25912 shutdown 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 msgr2=0x557da621f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f595c005580 tx=0x7f595c009310 comp rx=0 tx=0).stop 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 shutdown_connections 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557da628e9c0 0x557da6290de0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557da62174d0 0x557da621ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557da620bd40 0x557da621f8c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f5954c00700 0 client.25912 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 --2- 172.21.15.52:0/1276416064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557da620c660 0x557da621c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.539+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 >> 172.21.15.52:0/1276416064 conn(0x557da613b050 msgr2=0x557da6221eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.540+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 shutdown_connections 2024-04-26T01:51:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.540+0000 7f597c404b80 1 -- 172.21.15.52:0/1276416064 wait complete. 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.540+0000 7f597c404b80 20 client.25912 trim_cache size 0 max 16384 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 16, 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "93ad249f": { 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 16, 2024-04-26T01:51:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 70.0, 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.540+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 msgr2=0x7f59500a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 0x7f59500a6270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59580098a0 tx=0x7f5958009e30 comp rx=0 tx=0).stop 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 msgr2=0x7f596c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 0x7f596c19f630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f596c07b260 tx=0x7f596400d3b0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 shutdown_connections 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f59500a3db0 0x7f59500a6270 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f596c113920 0x7f596c19f630 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f596c076f70 0x7f596c1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 --2- 172.21.15.52:0/2120255996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f596c077890 0x7f596c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 >> 172.21.15.52:0/2120255996 conn(0x7f596c100770 msgr2=0x7f596c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 shutdown_connections 2024-04-26T01:51:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:51:40.541+0000 7f5971a00700 1 -- 172.21.15.52:0/2120255996 wait complete. 2024-04-26T01:51:41.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 150979 2.90 MB/sec execute 180 sec latency 36814.874 ms 2024-04-26T01:51:41.478 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '93ad249f', seconds elapsed: 40.0 2024-04-26T01:51:41.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:41 smithi183 ceph-mon[32752]: pgmap v204: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:41.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:41 smithi052 ceph-mon[27851]: pgmap v204: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:41.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:41 smithi191 ceph-mon[32745]: pgmap v204: 97 pgs: 97 active+clean; 68 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 663 KiB/s wr, 26 op/s 2024-04-26T01:51:42.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 152508 2.95 MB/sec execute 181 sec latency 21.381 ms 2024-04-26T01:51:43.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 152888 2.94 MB/sec execute 182 sec latency 25.371 ms 2024-04-26T01:51:43.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:43 smithi183 ceph-mon[32752]: pgmap v205: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 726 KiB/s wr, 44 op/s 2024-04-26T01:51:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:43 smithi052 ceph-mon[27851]: pgmap v205: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 726 KiB/s wr, 44 op/s 2024-04-26T01:51:43.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:43 smithi191 ceph-mon[32745]: pgmap v205: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 726 KiB/s wr, 44 op/s 2024-04-26T01:51:44.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 153267 2.93 MB/sec execute 183 sec latency 28.125 ms 2024-04-26T01:51:45.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 154010 2.94 MB/sec execute 184 sec latency 33.131 ms 2024-04-26T01:51:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:45 smithi183 ceph-mon[32752]: pgmap v206: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 336 KiB/s wr, 29 op/s 2024-04-26T01:51:45.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:45 smithi052 ceph-mon[27851]: pgmap v206: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 336 KiB/s wr, 29 op/s 2024-04-26T01:51:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:45 smithi191 ceph-mon[32745]: pgmap v206: 97 pgs: 97 active+clean; 67 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 336 KiB/s wr, 29 op/s 2024-04-26T01:51:46.097 INFO:tasks.workunit.client.0.smithi052.stdout: 1 155267 2.95 MB/sec execute 185 sec latency 21.282 ms 2024-04-26T01:51:47.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 156307 2.98 MB/sec execute 186 sec latency 45.054 ms 2024-04-26T01:51:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:47 smithi052 ceph-mon[27851]: pgmap v207: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 672 KiB/s wr, 74 op/s 2024-04-26T01:51:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:47 smithi191 ceph-mon[32745]: pgmap v207: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 672 KiB/s wr, 74 op/s 2024-04-26T01:51:47.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:47 smithi183 ceph-mon[32752]: pgmap v207: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 672 KiB/s wr, 74 op/s 2024-04-26T01:51:48.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 156541 2.97 MB/sec execute 187 sec latency 26.234 ms 2024-04-26T01:51:49.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 157314 2.96 MB/sec execute 188 sec latency 22.394 ms 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.370+0000 7fb532c00640 1 -- 172.21.15.52:0/1532559166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 msgr2=0x7fb52c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.370+0000 7fb532c00640 1 --2- 172.21.15.52:0/1532559166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c101290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb520009ca0 tx=0x7fb52002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 -- 172.21.15.52:0/1532559166 shutdown_connections 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 --2- 172.21.15.52:0/1532559166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c105f80 0x7fb52c10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 --2- 172.21.15.52:0/1532559166 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c101860 0x7fb52c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 --2- 172.21.15.52:0/1532559166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c101290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 -- 172.21.15.52:0/1532559166 >> 172.21.15.52:0/1532559166 conn(0x7fb52c0fc770 msgr2=0x7fb52c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 -- 172.21.15.52:0/1532559166 shutdown_connections 2024-04-26T01:51:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.371+0000 7fb532c00640 1 -- 172.21.15.52:0/1532559166 wait complete. 2024-04-26T01:51:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.372+0000 7fb532c00640 1 Processor -- start 2024-04-26T01:51:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.372+0000 7fb532c00640 1 -- start start 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.372+0000 7fb532c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c105f80 0x7fb52c198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 0x7fb52c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c110f30 con 0x7fb52c19cec0 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c110db0 con 0x7fb52c105f80 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1110b0 con 0x7fb52c100eb0 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56324/0 (socket says 172.21.15.52:56324) 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 -- 172.21.15.52:0/1830360853 learned_addr learned my addr 172.21.15.52:0/1830360853 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb530e00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c105f80 0x7fb52c198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532200640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 0x7fb52c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 -- 172.21.15.52:0/1830360853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c105f80 msgr2=0x7fb52c198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c105f80 0x7fb52c198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 -- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 msgr2=0x7fb52c19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 0x7fb52c19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb531800640 1 -- 172.21.15.52:0/1830360853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c19d9b0 con 0x7fb52c100eb0 2024-04-26T01:51:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.373+0000 7fb532200640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 0x7fb52c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:51:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb531800640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb520005e30 tx=0x7fb520031840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb52a000640 1 -- 172.21.15.52:0/1830360853 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520003b70 con 0x7fb52c100eb0 2024-04-26T01:51:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb52a000640 1 -- 172.21.15.52:0/1830360853 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb520003d10 con 0x7fb52c100eb0 2024-04-26T01:51:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb52a000640 1 -- 172.21.15.52:0/1830360853 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52002fa50 con 0x7fb52c100eb0 2024-04-26T01:51:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c19dc40 con 0x7fb52c100eb0 2024-04-26T01:51:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.374+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c1a5460 con 0x7fb52c100eb0 2024-04-26T01:51:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.377+0000 7fb52a000640 1 -- 172.21.15.52:0/1830360853 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb52002fbf0 con 0x7fb52c100eb0 2024-04-26T01:51:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.378+0000 7fb537081740 1 Processor -- start 2024-04-26T01:51:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.378+0000 7fb537081740 1 -- start start 2024-04-26T01:51:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdb9b0 0x562544bdbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562544b41900 con 0x562544bdc2d0 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562544ae7ef0 con 0x562544bdb9b0 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb537081740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562544b3d8c0 con 0x562544be6b90 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb530e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb530e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56332/0 (socket says 172.21.15.52:56332) 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb530e00640 1 -- 172.21.15.52:0/3354021655 learned_addr learned my addr 172.21.15.52:0/3354021655 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb532200640 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdb9b0 0x562544bdbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.379+0000 7fb531800640 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 -- 172.21.15.52:0/3354021655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdb9b0 msgr2=0x562544bdbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdb9b0 0x562544bdbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 -- 172.21.15.52:0/3354021655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 msgr2=0x562544be6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 -- 172.21.15.52:0/3354021655 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562544854210 con 0x562544be6b90 2024-04-26T01:51:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb52a000640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 0x7fb5140a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb530e00640 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb518009c60 tx=0x7fb518011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24eeb3d65307270a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb52a000640 1 -- 172.21.15.52:0/1830360853 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb5200f1b70 con 0x7fb52c100eb0 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb532200640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 0x7fb5140a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb51ea00640 1 -- 172.21.15.52:0/3354021655 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518004230 con 0x562544be6b90 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb51ea00640 1 -- 172.21.15.52:0/3354021655 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5180043d0 con 0x562544be6b90 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.380+0000 7fb51ea00640 1 -- 172.21.15.52:0/3354021655 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180196d0 con 0x562544be6b90 2024-04-26T01:51:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 -- 172.21.15.52:0/3354021655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 msgr2=0x562544be8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb518009c60 tx=0x7fb518011810 comp rx=0 tx=0).stop 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb532200640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 0x7fb5140a61d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb52400bce0 tx=0x7fb52400b4d0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 -- 172.21.15.52:0/3354021655 shutdown_connections 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544be8f80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 --2- 172.21.15.52:0/3354021655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdb9b0 0x562544bdbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 -- 172.21.15.52:0/3354021655 >> 172.21.15.52:0/3354021655 conn(0x562544bd7ec0 msgr2=0x562544bd82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 -- 172.21.15.52:0/3354021655 shutdown_connections 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.381+0000 7fb537081740 1 -- 172.21.15.52:0/3354021655 wait complete. 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.382+0000 7fb537081740 1 -- start start 2024-04-26T01:51:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.382+0000 7fb537081740 1 Processor -- start 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.382+0000 7fb537081740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.382+0000 7fb537081740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.382+0000 7fb537081740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544be44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb537081740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562544be97c0 con 0x562544bdc2d0 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb537081740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562544afb310 con 0x562544be6b90 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb537081740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562544be9940 con 0x562544bdb9b0 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb537081740 10 client.0 _ll_register_callbacks cb 0x562544ac9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56340/0 (socket says 172.21.15.52:56340) 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 -- 172.21.15.52:0/3692255303 learned_addr learned my addr 172.21.15.52:0/3692255303 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb530e00640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb532200640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544be44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 -- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 msgr2=0x562544be44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544be44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 -- 172.21.15.52:0/3692255303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 msgr2=0x562544be3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.383+0000 7fb531800640 1 -- 172.21.15.52:0/3692255303 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562544920620 con 0x562544bdb9b0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb531800640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5200ef400 tx=0x7fb520009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35175f524e8bce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5200de270 con 0x562544bdb9b0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb5200ef510 con 0x562544bdb9b0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5180043d0 con 0x562544bdb9b0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb5200ef710 con 0x562544bdb9b0 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb537081740 10 client.25924 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562544be97c0 con 0x562544bdb9b0 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5180043d0 con 0x562544bdb9b0 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.384+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fb5200bde50 con 0x562544bdb9b0 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 10 client.25924 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 10 client.25924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 20 client.25924 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544bfb210 0x562544bfd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 4 client.25924 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562544b558b0 con 0x562544bfb210 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.385+0000 7fb530e00640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544bfb210 0x562544bfd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.386+0000 7fb530e00640 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544bfb210 0x562544bfd640 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8e75109d18577e server_cookie=c9416b4026ce1f6d in_seq=0 out_seq=0 2024-04-26T01:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.386+0000 7fb513400640 10 client.25924 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.386+0000 7fb513400640 1 -- 172.21.15.52:0/3692255303 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562544b558b0 con 0x562544bfb210 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.386+0000 7fb513400640 10 client.25924 handle_command_reply: tid=0 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.386+0000 7fb537081740 1 client.25924 shutdown 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 msgr2=0x562544be3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5200ef400 tx=0x7fb520009660 comp rx=0 tx=0).stop 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 shutdown_connections 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544bfb210 0x562544bfd640 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544be44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544be3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb513400640 0 client.25924 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 --2- 172.21.15.52:0/3692255303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be3a20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 >> 172.21.15.52:0/3692255303 conn(0x562544bd7ec0 msgr2=0x562544bddc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 shutdown_connections 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 1 -- 172.21.15.52:0/3692255303 wait complete. 2024-04-26T01:51:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.387+0000 7fb537081740 20 client.25924 trim_cache size 0 max 16384 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 Processor -- start 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 -- start start 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544bde370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1adfc0 con 0x562544bdc2d0 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1ab890 con 0x562544be6b90 2024-04-26T01:51:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.406+0000 7fb537081740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1a9700 con 0x562544bdb9b0 2024-04-26T01:51:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb530e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544bde370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47460/0 (socket says 172.21.15.52:47460) 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 -- 172.21.15.52:0/678611499 learned_addr learned my addr 172.21.15.52:0/678611499 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb531800640 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 -- 172.21.15.52:0/678611499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 msgr2=0x562544be2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 -- 172.21.15.52:0/678611499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 msgr2=0x562544bde370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544bde370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 -- 172.21.15.52:0/678611499 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562544bde8b0 con 0x562544be6b90 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.407+0000 7fb532200640 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb52401e570 tx=0x7fb524015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63ec9f80f806cd08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.408+0000 7fb51e000640 1 -- 172.21.15.52:0/678611499 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52402c070 con 0x562544be6b90 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.408+0000 7fb51e000640 1 -- 172.21.15.52:0/678611499 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb52401ec60 con 0x562544be6b90 2024-04-26T01:51:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.408+0000 7fb51e000640 1 -- 172.21.15.52:0/678611499 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52401d660 con 0x562544be6b90 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- 172.21.15.52:0/678611499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 msgr2=0x562544bd6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb52401e570 tx=0x7fb524015000 comp rx=0 tx=0).stop 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- 172.21.15.52:0/678611499 shutdown_connections 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544be6b90 0x562544bd6ea0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdc2d0 0x562544bde370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 --2- 172.21.15.52:0/678611499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544bdb9b0 0x562544be2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- 172.21.15.52:0/678611499 >> 172.21.15.52:0/678611499 conn(0x562544bd7ec0 msgr2=0x562544bd6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- 172.21.15.52:0/678611499 shutdown_connections 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- 172.21.15.52:0/678611499 wait complete. 2024-04-26T01:51:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 -- start start 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.409+0000 7fb537081740 1 Processor -- start 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdc2d0 0x562544c58f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544bf03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1a9d50 con 0x562544bdb9b0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1ab1c0 con 0x562544bdc2d0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1aa420 con 0x562544be6b90 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb537081740 10 client.0 _ll_register_callbacks cb 0x7fb520009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb531800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb531800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53006/0 (socket says 172.21.15.52:53006) 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb531800640 1 -- 172.21.15.52:0/1725408411 learned_addr learned my addr 172.21.15.52:0/1725408411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb530e00640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdc2d0 0x562544c58f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.410+0000 7fb532200640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544bf03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 -- 172.21.15.52:0/1725408411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 msgr2=0x562544bf03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544bf03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 -- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdc2d0 msgr2=0x562544c58f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:51:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdc2d0 0x562544c58f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 -- 172.21.15.52:0/1725408411 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562544bf0c20 con 0x562544bdb9b0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb531800640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb5200bd8f0 tx=0x7fb5200099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec92497b3e0784c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb513400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.411+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5200c68b0 con 0x562544bdb9b0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fb520004b40 con 0x562544bdb9b0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb52401ec60 con 0x562544bdb9b0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb520087050 con 0x562544bdb9b0 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb537081740 10 client.15816 fetch_fsmap learned FSMap version 38 2024-04-26T01:51:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562544c61310 con 0x562544bdb9b0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.412+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb52401ec60 con 0x562544bdb9b0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fb520082b70 con 0x562544bdb9b0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 10 client.15816 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 20 client.15816 populate_metadata read hostname 'smithi052' 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544c618a0 0x562544c63c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 4 client.15816 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562544b4f320 con 0x562544c618a0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.413+0000 7fb530e00640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544c618a0 0x562544c63c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.414+0000 7fb530e00640 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544c618a0 0x562544c63c90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed5a15c1b54100e server_cookie=b2eb25390c07b231 in_seq=0 out_seq=0 2024-04-26T01:51:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.414+0000 7fb513400640 10 client.15816 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.450+0000 7fb513400640 1 -- 172.21.15.52:0/1725408411 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562544b4f320 con 0x562544c618a0 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.450+0000 7fb513400640 10 client.15816 handle_command_reply: tid=0 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 client.15816 shutdown 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 msgr2=0x562544c58a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb5200bd8f0 tx=0x7fb5200099b0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 shutdown_connections 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562544c618a0 0x562544c63c90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562544be6b90 0x562544bf03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562544bdc2d0 0x562544c58f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 --2- 172.21.15.52:0/1725408411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562544bdb9b0 0x562544c58a50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb513400640 0 client.15816 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 >> 172.21.15.52:0/1725408411 conn(0x562544bd7ec0 msgr2=0x562544bd7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 shutdown_connections 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 1 -- 172.21.15.52:0/1725408411 wait complete. 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.451+0000 7fb537081740 20 client.15816 trim_cache size 0 max 16384 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:51:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:51:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:51:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 msgr2=0x7fb5140a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 0x7fb5140a61d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb52400bce0 tx=0x7fb52400b4d0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 msgr2=0x7fb52c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb520005e30 tx=0x7fb520031840 comp rx=0 tx=0).stop 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 shutdown_connections 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5140a3d10 0x7fb5140a61d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb52c19cec0 0x7fb52c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb52c105f80 0x7fb52c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 --2- 172.21.15.52:0/1830360853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c100eb0 0x7fb52c198600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.452+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 >> 172.21.15.52:0/1830360853 conn(0x7fb52c0fc770 msgr2=0x7fb52c0699e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.453+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 shutdown_connections 2024-04-26T01:51:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:51:49.453+0000 7fb532c00640 1 -- 172.21.15.52:0/1830360853 wait complete. 2024-04-26T01:51:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:49 smithi052 ceph-mon[27851]: pgmap v208: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:49 smithi191 ceph-mon[32745]: pgmap v208: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:49.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:49 smithi183 ceph-mon[32752]: pgmap v208: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:50.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 158572 2.97 MB/sec execute 189 sec latency 68.235 ms 2024-04-26T01:51:51.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 159777 3.01 MB/sec execute 190 sec latency 23.979 ms 2024-04-26T01:51:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:51 smithi052 ceph-mon[27851]: pgmap v209: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:51 smithi191 ceph-mon[32745]: pgmap v209: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:51.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:51 smithi183 ceph-mon[32752]: pgmap v209: 97 pgs: 97 active+clean; 71 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 399 KiB/s wr, 63 op/s 2024-04-26T01:51:52.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 160056 3.00 MB/sec execute 191 sec latency 32.353 ms 2024-04-26T01:51:53.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 160781 3.00 MB/sec execute 192 sec latency 29.267 ms 2024-04-26T01:51:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:53 smithi052 ceph-mon[27851]: pgmap v210: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 794 KiB/s wr, 127 op/s 2024-04-26T01:51:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:53 smithi191 ceph-mon[32745]: pgmap v210: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 794 KiB/s wr, 127 op/s 2024-04-26T01:51:53.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:53 smithi183 ceph-mon[32752]: pgmap v210: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 794 KiB/s wr, 127 op/s 2024-04-26T01:51:54.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 161937 3.01 MB/sec execute 193 sec latency 24.601 ms 2024-04-26T01:51:55.098 INFO:tasks.workunit.client.0.smithi052.stdout: 1 163257 3.05 MB/sec execute 194 sec latency 23.536 ms 2024-04-26T01:51:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:55 smithi052 ceph-mon[27851]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 731 KiB/s wr, 109 op/s 2024-04-26T01:51:55.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:55 smithi191 ceph-mon[32745]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 731 KiB/s wr, 109 op/s 2024-04-26T01:51:55.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:55 smithi183 ceph-mon[32752]: pgmap v211: 97 pgs: 97 active+clean; 70 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 731 KiB/s wr, 109 op/s 2024-04-26T01:51:56.099 INFO:tasks.workunit.client.0.smithi052.stdout: 1 163595 3.04 MB/sec execute 195 sec latency 39.592 ms 2024-04-26T01:51:57.099 INFO:tasks.workunit.client.0.smithi052.stdout: 1 164256 3.03 MB/sec execute 196 sec latency 22.623 ms 2024-04-26T01:51:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:57 smithi052 ceph-mon[27851]: pgmap v212: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-04-26T01:51:57.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:57 smithi191 ceph-mon[32745]: pgmap v212: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-04-26T01:51:57.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:57 smithi183 ceph-mon[32752]: pgmap v212: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-04-26T01:51:58.099 INFO:tasks.workunit.client.0.smithi052.stdout: 1 165324 3.04 MB/sec execute 197 sec latency 25.131 ms 2024-04-26T01:51:59.099 INFO:tasks.workunit.client.0.smithi052.stdout: 1 166733 3.08 MB/sec execute 198 sec latency 29.654 ms 2024-04-26T01:51:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:51:59 smithi052 ceph-mon[27851]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:51:59.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:51:59 smithi191 ceph-mon[32745]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:51:59.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:51:59 smithi183 ceph-mon[32752]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:52:00.051 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 15.9 2024-04-26T01:52:00.052 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:52:00.099 INFO:tasks.workunit.client.0.smithi052.stdout: 1 167138 3.07 MB/sec execute 199 sec latency 22.787 ms 2024-04-26T01:52:00.553 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:01.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 167761 3.07 MB/sec execute 200 sec latency 44.126 ms 2024-04-26T01:52:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:01 smithi052 ceph-mon[27851]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:52:01.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:01 smithi191 ceph-mon[32745]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:52:01.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:01 smithi183 ceph-mon[32752]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 111 op/s 2024-04-26T01:52:02.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 168291 3.07 MB/sec execute 201 sec latency 34.767 ms 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 -- 172.21.15.52:0/3782257000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 msgr2=0x7fc2b010bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/3782257000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b010bfe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc298009900 tx=0x7fc29802a510 comp rx=0 tx=0).stop 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 -- 172.21.15.52:0/3782257000 shutdown_connections 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/3782257000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b010c520 0x7fc2b010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/3782257000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b010bfe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/3782257000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b01031e0 0x7fc2b01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 -- 172.21.15.52:0/3782257000 >> 172.21.15.52:0/3782257000 conn(0x7fc2b0077fd0 msgr2=0x7fc2b00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 -- 172.21.15.52:0/3782257000 shutdown_connections 2024-04-26T01:52:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.602+0000 7fc2b6a00640 1 -- 172.21.15.52:0/3782257000 wait complete. 2024-04-26T01:52:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.603+0000 7fc2b6a00640 1 Processor -- start 2024-04-26T01:52:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.603+0000 7fc2b6a00640 1 -- start start 2024-04-26T01:52:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 0x7fc2b01a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b010c520 0x7fc2b019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01121e0 con 0x7fc2b01031e0 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0112060 con 0x7fc2b0103b00 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b6a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0112360 con 0x7fc2b010c520 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 0x7fc2b01a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 0x7fc2b01a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40518/0 (socket says 172.21.15.52:40518) 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2afe00640 1 -- 172.21.15.52:0/1761328580 learned_addr learned my addr 172.21.15.52:0/1761328580 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.604+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b010c520 0x7fc2b019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 -- 172.21.15.52:0/1761328580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b010c520 msgr2=0x7fc2b019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b010c520 0x7fc2b019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 -- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 msgr2=0x7fc2b01a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 0x7fc2b01a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 -- 172.21.15.52:0/1761328580 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b019afd0 con 0x7fc2b0103b00 2024-04-26T01:52:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b01a0b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc29802ab00 tx=0x7fc298003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2ad600640 1 -- 172.21.15.52:0/1761328580 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29803f070 con 0x7fc2b0103b00 2024-04-26T01:52:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2ad600640 1 -- 172.21.15.52:0/1761328580 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc298031b50 con 0x7fc2b0103b00 2024-04-26T01:52:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2ad600640 1 -- 172.21.15.52:0/1761328580 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298035030 con 0x7fc2b0103b00 2024-04-26T01:52:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b019b260 con 0x7fc2b0103b00 2024-04-26T01:52:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.605+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b01a7490 con 0x7fc2b0103b00 2024-04-26T01:52:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.609+0000 7fc2ad600640 1 -- 172.21.15.52:0/1761328580 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc298042070 con 0x7fc2b0103b00 2024-04-26T01:52:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.609+0000 7fc2b998b740 1 Processor -- start 2024-04-26T01:52:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 -- start start 2024-04-26T01:52:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e5bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e68df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1dc1900 con 0x55fbb1e5b820 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1d67ef0 con 0x55fbb1e66a00 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b998b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1dbd8c0 con 0x55fbb1e5c140 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e5bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e5bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40534/0 (socket says 172.21.15.52:40534) 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2b4a00640 1 -- 172.21.15.52:0/1037297016 learned_addr learned my addr 172.21.15.52:0/1037297016 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.610+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:51454/0 (socket says 172.21.15.52:51454) 2024-04-26T01:52:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e68df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 -- 172.21.15.52:0/1037297016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 msgr2=0x55fbb1e68df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e68df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 -- 172.21.15.52:0/1037297016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 msgr2=0x55fbb1e5bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e5bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 -- 172.21.15.52:0/1037297016 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb1ad4210 con 0x55fbb1e5c140 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e68df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0004590 tx=0x7fc2a000bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fce983009994236 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2ad600640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 0x7fc2940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.611+0000 7fc2a5200640 1 -- 172.21.15.52:0/1037297016 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0003bd0 con 0x55fbb1e5c140 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2ad600640 1 -- 172.21.15.52:0/1761328580 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fc2980f09f0 con 0x7fc2b0103b00 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2a5200640 1 -- 172.21.15.52:0/1037297016 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0015070 con 0x55fbb1e5c140 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2a5200640 1 -- 172.21.15.52:0/1037297016 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0004dd0 con 0x55fbb1e5c140 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 0x7fc2940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2b998b740 1 -- 172.21.15.52:0/1037297016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 msgr2=0x55fbb1e664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.612+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0004590 tx=0x7fc2a000bbb0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 0x7fc2940a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8011630 tx=0x7fc2a8009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 -- 172.21.15.52:0/1037297016 shutdown_connections 2024-04-26T01:52:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e68df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e664c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1037297016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e5bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 -- 172.21.15.52:0/1037297016 >> 172.21.15.52:0/1037297016 conn(0x55fbb1e57d30 msgr2=0x55fbb1e58110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 -- 172.21.15.52:0/1037297016 shutdown_connections 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 -- 172.21.15.52:0/1037297016 wait complete. 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.613+0000 7fc2b998b740 1 -- start start 2024-04-26T01:52:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 Processor -- start 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e63960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e643e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1e69630 con 0x55fbb1e66a00 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1d7b310 con 0x55fbb1e5b820 2024-04-26T01:52:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fbb1e697b0 con 0x55fbb1e5c140 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b998b740 10 client.0 _ll_register_callbacks cb 0x55fbb1d49cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e63960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e63960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46374/0 (socket says 172.21.15.52:46374) 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:51470/0 (socket says 172.21.15.52:51470) 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2afe00640 1 -- 172.21.15.52:0/447964936 learned_addr learned my addr 172.21.15.52:0/447964936 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.614+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 -- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 msgr2=0x55fbb1e63960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e63960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 -- 172.21.15.52:0/447964936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 msgr2=0x55fbb1e643e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e643e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 -- 172.21.15.52:0/447964936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb1ba0620 con 0x55fbb1e5c140 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2a7400640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2980ee110 tx=0x7fc2980b8d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a08274df27a0bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc29d600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298049070 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a0015070 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.615+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fc2980ee390 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc2980ee590 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 10 client.25942 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fbb1e69680 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0015070 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fc2980bcc70 con 0x55fbb1e5c140 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 10 client.25942 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 10 client.25942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 20 client.25942 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1e76cd0 0x55fbb1e790c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 4 client.25942 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.616+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbb1dd58b0 con 0x55fbb1e76cd0 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.617+0000 7fc2afe00640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1e76cd0 0x55fbb1e790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.617+0000 7fc2afe00640 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1e76cd0 0x55fbb1e790c0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c9e1333c9869f0b server_cookie=6706ee4653f06428 in_seq=0 out_seq=0 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.617+0000 7fc29d600640 10 client.25942 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc29d600640 1 -- 172.21.15.52:0/447964936 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbb1dd58b0 con 0x55fbb1e76cd0 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc29d600640 10 client.25942 handle_command_reply: tid=0 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 client.25942 shutdown 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 msgr2=0x55fbb1e63ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2980ee110 tx=0x7fc2980b8d70 comp rx=0 tx=0).stop 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 shutdown_connections 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1e76cd0 0x55fbb1e790c0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e63ea0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 --2- 172.21.15.52:0/447964936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e63960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc29d600640 0 client.25942 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.618+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 >> 172.21.15.52:0/447964936 conn(0x55fbb1e57d30 msgr2=0x55fbb1e5dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.619+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 shutdown_connections 2024-04-26T01:52:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.619+0000 7fc2b998b740 1 -- 172.21.15.52:0/447964936 wait complete. 2024-04-26T01:52:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.619+0000 7fc2b998b740 20 client.25942 trim_cache size 0 max 16384 2024-04-26T01:52:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.638+0000 7fc2b998b740 1 Processor -- start 2024-04-26T01:52:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 -- start start 2024-04-26T01:52:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e57610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e7e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01b2570 con 0x55fbb1e66a00 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01afe40 con 0x55fbb1e5b820 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b998b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01adc90 con 0x55fbb1e5c140 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e7e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46390/0 (socket says 172.21.15.52:46390) 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2afe00640 1 -- 172.21.15.52:0/3609637680 learned_addr learned my addr 172.21.15.52:0/3609637680 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.639+0000 7fc2a7400640 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e57610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 -- 172.21.15.52:0/3609637680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 msgr2=0x55fbb1e57610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e57610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 -- 172.21.15.52:0/3609637680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 msgr2=0x55fbb1e7e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e7e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 -- 172.21.15.52:0/3609637680 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb1e648c0 con 0x55fbb1e5b820 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2a7400640 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc2afe00640 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00044e0 tx=0x7fc2a0003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae7636ccec705f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc29fe00640 1 -- 172.21.15.52:0/3609637680 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0003dd0 con 0x55fbb1e5b820 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc29fe00640 1 -- 172.21.15.52:0/3609637680 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0015070 con 0x55fbb1e5b820 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.640+0000 7fc29fe00640 1 -- 172.21.15.52:0/3609637680 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0019420 con 0x55fbb1e5b820 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.641+0000 7fc2b998b740 1 -- 172.21.15.52:0/3609637680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 msgr2=0x55fbb1e724f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.641+0000 7fc2b998b740 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00044e0 tx=0x7fc2a0003af0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 -- 172.21.15.52:0/3609637680 shutdown_connections 2024-04-26T01:52:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e66a00 0x55fbb1e7e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 --2- 172.21.15.52:0/3609637680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e5b820 0x55fbb1e724f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 -- 172.21.15.52:0/3609637680 >> 172.21.15.52:0/3609637680 conn(0x55fbb1e57d30 msgr2=0x55fbb1e641b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 -- 172.21.15.52:0/3609637680 shutdown_connections 2024-04-26T01:52:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 -- 172.21.15.52:0/3609637680 wait complete. 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 -- start start 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.642+0000 7fc2b998b740 1 Processor -- start 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e6c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01ae300 con 0x55fbb1e5b820 2024-04-26T01:52:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01af770 con 0x55fbb1e66a00 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01ae9d0 con 0x55fbb1e5c140 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b998b740 10 client.0 _ll_register_callbacks cb 0x7fc298009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46400/0 (socket says 172.21.15.52:46400) 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40548/0 (socket says 172.21.15.52:40548) 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2b4a00640 1 -- 172.21.15.52:0/1487178816 learned_addr learned my addr 172.21.15.52:0/1487178816 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.643+0000 7fc2a7400640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e6c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 -- 172.21.15.52:0/1487178816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 msgr2=0x55fbb1e6c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e6c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 -- 172.21.15.52:0/1487178816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 msgr2=0x55fbb1e6c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 -- 172.21.15.52:0/1487178816 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb1edc250 con 0x55fbb1e66a00 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc2b4a00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80041e0 tx=0x7fc2a8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffa7f12c56f954bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc29d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.644+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a802e060 con 0x55fbb1e66a00 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a0015070 con 0x55fbb1e66a00 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7fc2a800f770 con 0x55fbb1e66a00 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc2a8004810 con 0x55fbb1e66a00 2024-04-26T01:52:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 10 client.25940 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fbb1edd5b0 con 0x55fbb1e66a00 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0015070 con 0x55fbb1e66a00 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fc2a8028030 con 0x55fbb1e66a00 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 10 client.25940 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 20 client.25940 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1eddc80 0x55fbb1ee00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 4 client.25940 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.645+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbb1dcf320 con 0x55fbb1eddc80 2024-04-26T01:52:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.646+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1eddc80 0x55fbb1ee00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.646+0000 7fc2afe00640 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1eddc80 0x55fbb1ee00a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bed01ad9a64eb6 server_cookie=50a53b315d0eca21 in_seq=0 out_seq=0 2024-04-26T01:52:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.646+0000 7fc29d600640 10 client.25940 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc29d600640 1 -- 172.21.15.52:0/1487178816 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fbb1dcf320 con 0x55fbb1eddc80 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc29d600640 10 client.25940 handle_command_reply: tid=0 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 client.25940 shutdown 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 msgr2=0x55fbb1e6ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80041e0 tx=0x7fc2a8004a40 comp rx=0 tx=0).stop 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 shutdown_connections 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fbb1eddc80 0x55fbb1ee00a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fbb1e66a00 0x55fbb1e6ce60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fbb1e5c140 0x55fbb1e6c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 --2- 172.21.15.52:0/1487178816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbb1e5b820 0x55fbb1e6c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 >> 172.21.15.52:0/1487178816 conn(0x55fbb1e57d30 msgr2=0x55fbb1e5d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.692+0000 7fc29d600640 0 client.25940 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.693+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 shutdown_connections 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.693+0000 7fc2b998b740 1 -- 172.21.15.52:0/1487178816 wait complete. 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.693+0000 7fc2b998b740 20 client.25940 trim_cache size 0 max 16384 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:52:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 msgr2=0x7fc2940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 0x7fc2940a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8011630 tx=0x7fc2a8009500 comp rx=0 tx=0).stop 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 msgr2=0x7fc2b01a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b01a0b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc29802ab00 tx=0x7fc298003b90 comp rx=0 tx=0).stop 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 shutdown_connections 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc2940a3e40 0x7fc2940a6300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc2b010c520 0x7fc2b019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0103b00 0x7fc2b01a0b80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 --2- 172.21.15.52:0/1761328580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2b01031e0 0x7fc2b01a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.695+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 >> 172.21.15.52:0/1761328580 conn(0x7fc2b0077fd0 msgr2=0x7fc2b01084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.696+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 shutdown_connections 2024-04-26T01:52:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:02.696+0000 7fc2b6a00640 1 -- 172.21.15.52:0/1761328580 wait complete. 2024-04-26T01:52:03.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 169725 3.09 MB/sec execute 202 sec latency 41.278 ms 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 -- 172.21.15.52:0/3848240440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8105890 msgr2=0x7f56d81098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 --2- 172.21.15.52:0/3848240440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8105890 0x7f56d81098e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56c40099c0 tx=0x7f56c4009cd0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 -- 172.21.15.52:0/3848240440 shutdown_connections 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 --2- 172.21.15.52:0/3848240440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8109fc0 0x7f56d8111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 --2- 172.21.15.52:0/3848240440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d81052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 --2- 172.21.15.52:0/3848240440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8105890 0x7f56d81098e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 -- 172.21.15.52:0/3848240440 >> 172.21.15.52:0/3848240440 conn(0x7f56d8100770 msgr2=0x7f56d8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:03.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 -- 172.21.15.52:0/3848240440 shutdown_connections 2024-04-26T01:52:03.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.177+0000 7f56dd800700 1 -- 172.21.15.52:0/3848240440 wait complete. 2024-04-26T01:52:03.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.178+0000 7f56dd800700 1 Processor -- start 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.178+0000 7f56dd800700 1 -- start start 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8105890 0x7f56d819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f56d8114e50 con 0x7f56d8104ee0 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f56d8114cd0 con 0x7f56d8105890 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56dd800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f56d8114fd0 con 0x7f56d8109fc0 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d7400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d7400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:51516/0 (socket says 172.21.15.52:51516) 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d6000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8105890 0x7f56d819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:03.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d7400700 1 -- 172.21.15.52:0/2611504614 learned_addr learned my addr 172.21.15.52:0/2611504614 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d6a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40568/0 (socket says 172.21.15.52:40568) 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d7400700 1 -- 172.21.15.52:0/2611504614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8105890 msgr2=0x7f56d819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.179+0000 7f56d7400700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8105890 0x7f56d819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56d7400700 1 -- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 msgr2=0x7f56d819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56d7400700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56d7400700 1 -- 172.21.15.52:0/2611504614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d81a02f0 con 0x7f56d8109fc0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56d7400700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f56d000ef30 tx=0x7f56d000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d00092e0 con 0x7f56d8109fc0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d0004bb0 con 0x7f56d8109fc0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d81a0580 con 0x7f56d8109fc0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0004020 con 0x7f56d8109fc0 2024-04-26T01:52:03.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.180+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d81a7e60 con 0x7f56d8109fc0 2024-04-26T01:52:03.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.182+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56d8069730 con 0x7f56d8109fc0 2024-04-26T01:52:03.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.184+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f56d001b030 con 0x7f56d8109fc0 2024-04-26T01:52:03.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.185+0000 7f56cf400700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 0x7f56bc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:03.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.186+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8477+0+0 (secure 0 0 0) 0x7f56d00d0190 con 0x7f56d8109fc0 2024-04-26T01:52:03.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.186+0000 7f56d6a00700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 0x7f56bc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:03.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.186+0000 7f56d6a00700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 0x7f56bc0a6270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009c60 tx=0x7f56c8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:03.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.190+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56d0022020 con 0x7f56d8109fc0 2024-04-26T01:52:03.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.516+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56d802df80 con 0x7f56d8109fc0 2024-04-26T01:52:03.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.517+0000 7f56cf400700 1 -- 172.21.15.52:0/2611504614 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f56d0010040 con 0x7f56d8109fc0 2024-04-26T01:52:03.515 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:52:03.516 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:52:03.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.520+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 msgr2=0x7f56bc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:03.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 0x7f56bc0a6270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009c60 tx=0x7f56c8009260 comp rx=0 tx=0).stop 2024-04-26T01:52:03.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 msgr2=0x7f56d819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:03.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f56d000ef30 tx=0x7f56d000c3a0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 shutdown_connections 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f56bc0a3db0 0x7f56bc0a6270 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56d8109fc0 0x7f56d819fbb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d8104ee0 0x7f56d819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 --2- 172.21.15.52:0/2611504614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f56d8105890 0x7f56d819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 >> 172.21.15.52:0/2611504614 conn(0x7f56d8100770 msgr2=0x7f56d8101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 shutdown_connections 2024-04-26T01:52:03.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:03.521+0000 7f56dd800700 1 -- 172.21.15.52:0/2611504614 wait complete. 2024-04-26T01:52:03.521 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:52:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:03 smithi191 ceph-mon[32745]: pgmap v215: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-04-26T01:52:03.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:03 smithi183 ceph-mon[32752]: pgmap v215: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-04-26T01:52:03.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:03 smithi052 ceph-mon[27851]: pgmap v215: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 182 op/s 2024-04-26T01:52:04.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 170588 3.11 MB/sec execute 203 sec latency 42.859 ms 2024-04-26T01:52:04.563 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 75.9 --await-for 42.0' 2024-04-26T01:52:04.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 75.9 --await-for 42.0 2024-04-26T01:52:04.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:04 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2611504614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:04.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:04 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2611504614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:04.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:04 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2611504614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:05.078 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:05.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 171180 3.10 MB/sec execute 204 sec latency 25.809 ms 2024-04-26T01:52:05.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:05 smithi052 ceph-mon[27851]: pgmap v216: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 KiB/s wr, 119 op/s 2024-04-26T01:52:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:05 smithi191 ceph-mon[32745]: pgmap v216: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 KiB/s wr, 119 op/s 2024-04-26T01:52:05.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:05 smithi183 ceph-mon[32752]: pgmap v216: 97 pgs: 97 active+clean; 78 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 KiB/s wr, 119 op/s 2024-04-26T01:52:06.100 INFO:tasks.workunit.client.0.smithi052.stdout: 1 172165 3.10 MB/sec execute 205 sec latency 26.058 ms 2024-04-26T01:52:07.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 173784 3.15 MB/sec execute 206 sec latency 30.042 ms 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 -- 172.21.15.52:0/1458488238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8102af0 msgr2=0x7f0bc8102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 --2- 172.21.15.52:0/1458488238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8102af0 0x7f0bc8102f70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40099c0 tx=0x7f0bb4009cd0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 -- 172.21.15.52:0/1458488238 shutdown_connections 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 --2- 172.21.15.52:0/1458488238 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc81034b0 0x7f0bc810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 --2- 172.21.15.52:0/1458488238 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8108a30 0x7f0bc8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 --2- 172.21.15.52:0/1458488238 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8102af0 0x7f0bc8102f70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 -- 172.21.15.52:0/1458488238 >> 172.21.15.52:0/1458488238 conn(0x7f0bc8079b10 msgr2=0x7f0bc8079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 -- 172.21.15.52:0/1458488238 shutdown_connections 2024-04-26T01:52:07.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.677+0000 7f0bce400700 1 -- 172.21.15.52:0/1458488238 wait complete. 2024-04-26T01:52:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.678+0000 7f0bce400700 1 Processor -- start 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.678+0000 7f0bce400700 1 -- start start 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc81034b0 0x7f0bc8199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8108a30 0x7f0bc819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc8112c60 con 0x7f0bc81034b0 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc8112ae0 con 0x7f0bc8108a30 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bce400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc8112de0 con 0x7f0bc8102af0 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bc7400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bc7400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50358/0 (socket says 172.21.15.52:50358) 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bc7400700 1 -- 172.21.15.52:0/935450505 learned_addr learned my addr 172.21.15.52:0/935450505 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:07.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.679+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8108a30 0x7f0bc819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 -- 172.21.15.52:0/935450505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8108a30 msgr2=0x7f0bc819d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8108a30 0x7f0bc819d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 -- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc81034b0 msgr2=0x7f0bc8199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc81034b0 0x7f0bc8199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 -- 172.21.15.52:0/935450505 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc819e0b0 con 0x7f0bc8102af0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.680+0000 7f0bc7400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e9c0 tx=0x7f0bb800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.681+0000 7f0bbfe00700 1 -- 172.21.15.52:0/935450505 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb800cbd0 con 0x7f0bc8102af0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.681+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc819e3a0 con 0x7f0bc8102af0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.681+0000 7f0bbfe00700 1 -- 172.21.15.52:0/935450505 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bb800cd70 con 0x7f0bc8102af0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.681+0000 7f0bbfe00700 1 -- 172.21.15.52:0/935450505 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8005610 con 0x7f0bc8102af0 2024-04-26T01:52:07.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.681+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc8074c70 con 0x7f0bc8102af0 2024-04-26T01:52:07.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.686+0000 7f0bbfe00700 1 -- 172.21.15.52:0/935450505 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0bb8004830 con 0x7f0bc8102af0 2024-04-26T01:52:07.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.687+0000 7f0bd8e11b80 1 Processor -- start 2024-04-26T01:52:07.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.687+0000 7f0bd8e11b80 1 -- start start 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 0x559e11ddb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11ddd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bbfe00700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 0x7f0bac0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559e11d23020 con 0x559e11dcfdd0 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559e11cc98c0 con 0x559e11ddb560 2024-04-26T01:52:07.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559e11d1efe0 con 0x559e11dd06f0 2024-04-26T01:52:07.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bbfe00700 1 -- 172.21.15.52:0/935450505 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8565+0+0 (secure 0 0 0) 0x7f0bb80d0aa0 con 0x7f0bc8102af0 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56036/0 (socket says 172.21.15.52:56036) 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.688+0000 7f0bc7e00700 1 -- 172.21.15.52:0/3779451846 learned_addr learned my addr 172.21.15.52:0/3779451846 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7400700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 0x559e11ddb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11ddd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 -- 172.21.15.52:0/3779451846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 msgr2=0x559e11ddb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 0x559e11ddb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 -- 172.21.15.52:0/3779451846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 msgr2=0x559e11ddd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11ddd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 -- 172.21.15.52:0/3779451846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e11cbf810 con 0x559e11dcfdd0 2024-04-26T01:52:07.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7400700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 0x559e11ddb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:07.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.689+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009ec0 tx=0x7f0bc0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe31f10808cadcd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.690+0000 7f0bbcc00700 1 -- 172.21.15.52:0/3779451846 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0009310 con 0x559e11dcfdd0 2024-04-26T01:52:07.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.690+0000 7f0bbcc00700 1 -- 172.21.15.52:0/3779451846 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bc0013870 con 0x559e11dcfdd0 2024-04-26T01:52:07.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.690+0000 7f0bbcc00700 1 -- 172.21.15.52:0/3779451846 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0004220 con 0x559e11dcfdd0 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/3779451846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 msgr2=0x559e11dd01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009ec0 tx=0x7f0bc0009920 comp rx=0 tx=0).stop 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/3779451846 shutdown_connections 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11ddd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11dd01b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/3779451846 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dd06f0 0x559e11ddb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.691+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/3779451846 >> 172.21.15.52:0/3779451846 conn(0x559e11cff0e0 msgr2=0x559e11dcc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.692+0000 7f0bc6a00700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 0x7f0bac0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.692+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/3779451846 shutdown_connections 2024-04-26T01:52:07.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.692+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/3779451846 wait complete. 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.692+0000 7f0bc6a00700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 0x7f0bac0a6390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098e0 tx=0x7f0bb4000bc0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.693+0000 7f0bd8e11b80 1 -- start start 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.693+0000 7f0bd8e11b80 1 Processor -- start 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.693+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11ddac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dd7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559e11dddff0 con 0x559e11dcfdd0 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559e11cdc9f0 con 0x559e11dd06f0 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559e11dde170 con 0x559e11ddb560 2024-04-26T01:52:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bd8e11b80 10 client.0 _ll_register_callbacks cb 0x559e11cbb8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bc7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11ddac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50374/0 (socket says 172.21.15.52:50374) 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bc7e00700 1 -- 172.21.15.52:0/1182462781 learned_addr learned my addr 172.21.15.52:0/1182462781 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.694+0000 7f0bc6a00700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dd7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 -- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 msgr2=0x559e11dd7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dd7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 -- 172.21.15.52:0/1182462781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 msgr2=0x559e11ddac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11ddac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 -- 172.21.15.52:0/1182462781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e11c70a60 con 0x559e11ddb560 2024-04-26T01:52:07.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000f670 tx=0x7f0bc0003c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c446c876750def93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bb1600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.695+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0027020 con 0x559e11ddb560 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bc0013870 con 0x559e11ddb560 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8565+0+0 (secure 0 0 0) 0x7f0bc0013a20 con 0x559e11ddb560 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0bc0032050 con 0x559e11ddb560 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bd8e11b80 10 client.25963 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:07.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559e11cb9c80 con 0x559e11ddb560 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bc0013870 con 0x559e11ddb560 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.696+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f0bc0004740 con 0x559e11ddb560 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 10 client.25963 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 10 client.25963 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 20 client.25963 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11df3440 0x559e11df5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 4 client.25963 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:07.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e11d30810 con 0x559e11df3440 2024-04-26T01:52:07.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bc7400700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11df3440 0x559e11df5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bc7400700 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11df3440 0x559e11df5830 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1290349a289a5735 server_cookie=be95ee2a50396878 in_seq=0 out_seq=0 2024-04-26T01:52:07.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.697+0000 7f0bb1600700 10 client.25963 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:07.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.698+0000 7f0bb1600700 1 -- 172.21.15.52:0/1182462781 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e11d30810 con 0x559e11df3440 2024-04-26T01:52:07.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.698+0000 7f0bb1600700 10 client.25963 handle_command_reply: tid=0 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.698+0000 7f0bd8e11b80 1 client.25963 shutdown 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 msgr2=0x559e11dd84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000f670 tx=0x7f0bc0003c30 comp rx=0 tx=0).stop 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 shutdown_connections 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11df3440 0x559e11df5830 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd84c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11ddac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/1182462781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dd7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 >> 172.21.15.52:0/1182462781 conn(0x559e11cff0e0 msgr2=0x559e11dcbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bb1600700 0 client.25963 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:07.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 shutdown_connections 2024-04-26T01:52:07.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/1182462781 wait complete. 2024-04-26T01:52:07.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.699+0000 7f0bd8e11b80 20 client.25963 trim_cache size 0 max 16384 2024-04-26T01:52:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.723+0000 7f0bd8e11b80 1 Processor -- start 2024-04-26T01:52:07.721 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:07 smithi052 ceph-mon[27851]: pgmap v217: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-04-26T01:52:07.722 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:07 smithi052 ceph-mon[27851]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:52:07.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.723+0000 7f0bd8e11b80 1 -- start start 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.723+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dcfdd0 0x559e11dcc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 0x559e11dfae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81ada50 con 0x559e11dd06f0 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81ab2e0 con 0x559e11ddb560 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81a9150 con 0x559e11dcfdd0 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60078/0 (socket says 172.21.15.52:60078) 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dcfdd0 0x559e11dcc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2812090046 learned_addr learned my addr 172.21.15.52:0/2812090046 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc6a00700 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 0x559e11dfae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2812090046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dcfdd0 msgr2=0x559e11dcc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dcfdd0 0x559e11dcc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2812090046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 msgr2=0x559e11dfae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 0x559e11dfae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2812090046 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e11dd64f0 con 0x559e11ddb560 2024-04-26T01:52:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.724+0000 7f0bc6a00700 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 0x559e11dfae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.725+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00075a0 tx=0x7f0bc0009050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f37f8acba6e80d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.725+0000 7f0bb3e00700 1 -- 172.21.15.52:0/2812090046 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc001ae60 con 0x559e11ddb560 2024-04-26T01:52:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.725+0000 7f0bb3e00700 1 -- 172.21.15.52:0/2812090046 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bc0013d20 con 0x559e11ddb560 2024-04-26T01:52:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.725+0000 7f0bb3e00700 1 -- 172.21.15.52:0/2812090046 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0027040 con 0x559e11ddb560 2024-04-26T01:52:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2812090046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 msgr2=0x559e11dfb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00075a0 tx=0x7f0bc0009050 comp rx=0 tx=0).stop 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2812090046 shutdown_connections 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11ddb560 0x559e11dfb370 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11dcfdd0 0x559e11dcc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2812090046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dd06f0 0x559e11dfae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2812090046 >> 172.21.15.52:0/2812090046 conn(0x559e11cff0e0 msgr2=0x559e11dd9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2812090046 shutdown_connections 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.726+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2812090046 wait complete. 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.727+0000 7f0bd8e11b80 1 -- start start 2024-04-26T01:52:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.727+0000 7f0bd8e11b80 1 Processor -- start 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11debc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81a97a0 con 0x559e11dcfdd0 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81aac10 con 0x559e11dd06f0 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81a9e70 con 0x559e11ddb560 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bd8e11b80 10 client.0 _ll_register_callbacks cb 0x559e11cbb8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bc6a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bc6a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dec1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60080/0 (socket says 172.21.15.52:60080) 2024-04-26T01:52:07.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bc6a00700 1 -- 172.21.15.52:0/2109816452 learned_addr learned my addr 172.21.15.52:0/2109816452 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:07.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bc7400700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11debc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.728+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 msgr2=0x559e11dec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dec1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2109816452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 msgr2=0x559e11debc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11debc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 -- 172.21.15.52:0/2109816452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e11dd4c00 con 0x559e11ddb560 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7400700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11debc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bc7e00700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd43c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00090e0 tx=0x7f0bc0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea115dcb0c07bfe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.729+0000 7f0bb1600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:07.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0043060 con 0x559e11ddb560 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8565+0+0 (secure 0 0 0) 0x7f0bc0013a70 con 0x559e11ddb560 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bc0013d20 con 0x559e11ddb560 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0bc0007e60 con 0x559e11ddb560 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bd8e11b80 10 client.25969 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.730+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559e11dd5e50 con 0x559e11ddb560 2024-04-26T01:52:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.731+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bc0013d20 con 0x559e11ddb560 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.731+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f0bc0004ce0 con 0x559e11ddb560 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 10 client.25969 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 10 client.25969 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 20 client.25969 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11e5a9b0 0x559e11e5cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 4 client.25969 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.9, "await_for": 42.0}] 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 75.9, "await_for": 42.0}) -- 0x559e11d2a8b0 con 0x559e11e5a9b0 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.732+0000 7f0bc7400700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11e5a9b0 0x559e11e5cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.733+0000 7f0bc7400700 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11e5a9b0 0x559e11e5cdd0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=380cafa5318d3322 server_cookie=4bd65e451e35e355 in_seq=0 out_seq=0 2024-04-26T01:52:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.733+0000 7f0bb1600700 10 client.25969 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:07.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.824+0000 7f0bb1600700 1 -- 172.21.15.52:0/2109816452 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559e11d2a8b0 con 0x559e11e5a9b0 2024-04-26T01:52:07.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.824+0000 7f0bb1600700 10 client.25969 handle_command_reply: tid=0 2024-04-26T01:52:07.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.824+0000 7f0bd8e11b80 1 client.25969 shutdown 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 msgr2=0x559e11dd43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd43c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00090e0 tx=0x7f0bc0009580 comp rx=0 tx=0).stop 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 shutdown_connections 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559e11e5a9b0 0x559e11e5cdd0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559e11ddb560 0x559e11dd43c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559e11dcfdd0 0x559e11debc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 --2- 172.21.15.52:0/2109816452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559e11dd06f0 0x559e11dec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 >> 172.21.15.52:0/2109816452 conn(0x559e11cff0e0 msgr2=0x559e11dcb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bb1600700 0 client.25969 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 shutdown_connections 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 1 -- 172.21.15.52:0/2109816452 wait complete. 2024-04-26T01:52:07.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.825+0000 7f0bd8e11b80 20 client.25969 trim_cache size 0 max 16384 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 18, 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "afeaef32": { 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 18, 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:52:07.825 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 75.9, 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:07.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:07.827 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:52:07.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 msgr2=0x7f0bac0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 0x7f0bac0a6390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098e0 tx=0x7f0bb4000bc0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 msgr2=0x7f0bc81990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:07.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e9c0 tx=0x7f0bb800ecd0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 shutdown_connections 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0bac0a3ed0 0x7f0bac0a6390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0bc8108a30 0x7f0bc819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bc8102af0 0x7f0bc81990f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 --2- 172.21.15.52:0/935450505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0bc81034b0 0x7f0bc8199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 >> 172.21.15.52:0/935450505 conn(0x7f0bc8079b10 msgr2=0x7f0bc81017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 shutdown_connections 2024-04-26T01:52:07.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:07.827+0000 7f0bce400700 1 -- 172.21.15.52:0/935450505 wait complete. 2024-04-26T01:52:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:07 smithi191 ceph-mon[32745]: pgmap v217: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-04-26T01:52:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:07 smithi191 ceph-mon[32745]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:52:07.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:07 smithi183 ceph-mon[32752]: pgmap v217: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 209 op/s 2024-04-26T01:52:07.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:07 smithi183 ceph-mon[32752]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T01:52:08.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.14 MB/sec execute 207 sec latency 354.734 ms 2024-04-26T01:52:08.806 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'afeaef32', quiesce took 8.8 seconds. Will release after: 15.9 2024-04-26T01:52:09.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.12 MB/sec execute 208 sec latency 1354.826 ms 2024-04-26T01:52:09.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:09 smithi191 ceph-mon[32745]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:09 smithi183 ceph-mon[32752]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:09 smithi052 ceph-mon[27851]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:10.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.11 MB/sec execute 209 sec latency 2354.988 ms 2024-04-26T01:52:11.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.09 MB/sec execute 210 sec latency 3355.133 ms 2024-04-26T01:52:11.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:11 smithi191 ceph-mon[32745]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:11.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:11 smithi183 ceph-mon[32752]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:11.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:11 smithi052 ceph-mon[27851]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 194 op/s 2024-04-26T01:52:12.101 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.08 MB/sec execute 211 sec latency 4355.315 ms 2024-04-26T01:52:13.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.07 MB/sec execute 212 sec latency 5355.435 ms 2024-04-26T01:52:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:13 smithi191 ceph-mon[32745]: pgmap v221: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 769 KiB/s wr, 140 op/s 2024-04-26T01:52:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:13 smithi191 ceph-mon[32745]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:52:13.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:13 smithi183 ceph-mon[32752]: pgmap v221: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 769 KiB/s wr, 140 op/s 2024-04-26T01:52:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:13 smithi183 ceph-mon[32752]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:52:13.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:13 smithi052 ceph-mon[27851]: pgmap v221: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 769 KiB/s wr, 140 op/s 2024-04-26T01:52:13.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:13 smithi052 ceph-mon[27851]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T01:52:14.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.05 MB/sec execute 213 sec latency 6355.630 ms 2024-04-26T01:52:15.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.04 MB/sec execute 214 sec latency 7355.799 ms 2024-04-26T01:52:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 -- 172.21.15.52:0/1402114106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16881052f0 msgr2=0x7f168810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 --2- 172.21.15.52:0/1402114106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16881052f0 0x7f168810cec0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f168002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 -- 172.21.15.52:0/1402114106 shutdown_connections 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 --2- 172.21.15.52:0/1402114106 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16881052f0 0x7f168810cec0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 --2- 172.21.15.52:0/1402114106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100bd0 0x7f1688104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 --2- 172.21.15.52:0/1402114106 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1688100220 0x7f1688100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 -- 172.21.15.52:0/1402114106 >> 172.21.15.52:0/1402114106 conn(0x7f1688076db0 msgr2=0x7f16880fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 -- 172.21.15.52:0/1402114106 shutdown_connections 2024-04-26T01:52:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.844+0000 7f1687200640 1 -- 172.21.15.52:0/1402114106 wait complete. 2024-04-26T01:52:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.845+0000 7f1687200640 1 Processor -- start 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.845+0000 7f1687200640 1 -- start start 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16881052f0 0x7f168819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f168819f010 0x7f168819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f168810fe50 con 0x7f168819f010 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f168810fcd0 con 0x7f16881052f0 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1687200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f168810ffd0 con 0x7f1688100220 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1685e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f168819f010 0x7f168819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1685e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50580/0 (socket says 172.21.15.52:50580) 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f1685e00640 1 -- 172.21.15.52:0/2807019528 learned_addr learned my addr 172.21.15.52:0/2807019528 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.846+0000 7f167fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16881052f0 0x7f168819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 -- 172.21.15.52:0/2807019528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16881052f0 msgr2=0x7f168819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16881052f0 0x7f168819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 -- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f168819f010 msgr2=0x7f168819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f168819f010 0x7f168819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 -- 172.21.15.52:0/2807019528 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168819fb00 con 0x7f1688100220 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1685e00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f167800d950 tx=0x7f167800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f167ea00640 1 -- 172.21.15.52:0/2807019528 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1678017030 con 0x7f1688100220 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f167ea00640 1 -- 172.21.15.52:0/2807019528 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1678004020 con 0x7f1688100220 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168819fdf0 con 0x7f1688100220 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f167ea00640 1 -- 172.21.15.52:0/2807019528 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16780047a0 con 0x7f1688100220 2024-04-26T01:52:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.847+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a75e0 con 0x7f1688100220 2024-04-26T01:52:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.850+0000 7f168fb9c740 1 Processor -- start 2024-04-26T01:52:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f167ea00640 1 -- 172.21.15.52:0/2807019528 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1678009a30 con 0x7f1688100220 2024-04-26T01:52:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 -- start start 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437959650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d437959b90 0x55d43795bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d4378b4900 con 0x55d43794e9b0 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d43785aef0 con 0x55d43794f2d0 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f168fb9c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d4378b08c0 con 0x55d437959b90 2024-04-26T01:52:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48388/0 (socket says 172.21.15.52:48388) 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.851+0000 7f1686800640 1 -- 172.21.15.52:0/11789135 learned_addr learned my addr 172.21.15.52:0/11789135 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1685e00640 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437959650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f167fe00640 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d437959b90 0x55d43795bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 -- 172.21.15.52:0/11789135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d437959b90 msgr2=0x55d43795bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d437959b90 0x55d43795bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 -- 172.21.15.52:0/11789135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 msgr2=0x55d437959650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437959650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 -- 172.21.15.52:0/11789135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4375c7210 con 0x55d43794e9b0 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f1686800640 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f168002f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc3f104f2fb26a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.852+0000 7f167ea00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 0x7f16680a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f1677000640 1 -- 172.21.15.52:0/11789135 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680031db0 con 0x55d43794e9b0 2024-04-26T01:52:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f1677000640 1 -- 172.21.15.52:0/11789135 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680036030 con 0x55d43794e9b0 2024-04-26T01:52:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f1677000640 1 -- 172.21.15.52:0/11789135 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16800042a0 con 0x55d43794e9b0 2024-04-26T01:52:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f167ea00640 1 -- 172.21.15.52:0/2807019528 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f16780d0230 con 0x7f1688100220 2024-04-26T01:52:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f1685e00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 0x7f16680a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f168fb9c740 1 -- 172.21.15.52:0/11789135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 msgr2=0x55d43794ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f168fb9c740 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f168002f3c0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.853+0000 7f1685e00640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 0x7f16680a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55d43794ff50 tx=0x7f16780d3ce0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 -- 172.21.15.52:0/11789135 shutdown_connections 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d437959b90 0x55d43795bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437959650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 --2- 172.21.15.52:0/11789135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d43794ed90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 -- 172.21.15.52:0/11789135 >> 172.21.15.52:0/11789135 conn(0x55d43794aec0 msgr2=0x55d43794b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 -- 172.21.15.52:0/11789135 shutdown_connections 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 -- 172.21.15.52:0/11789135 wait complete. 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 -- start start 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.854+0000 7f168fb9c740 1 Processor -- start 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d4379570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d4379575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d43795c5e0 con 0x55d43794e9b0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d43786e310 con 0x55d437959b90 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d43795c760 con 0x55d43794f2d0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f168fb9c740 10 client.0 _ll_register_callbacks cb 0x55d43783ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48394/0 (socket says 172.21.15.52:48394) 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f1686800640 1 -- 172.21.15.52:0/2898303729 learned_addr learned my addr 172.21.15.52:0/2898303729 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f167fe00640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d4379570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.855+0000 7f1686800640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d4379575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 -- 172.21.15.52:0/2898303729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 msgr2=0x55d4379570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d4379570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 -- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 msgr2=0x55d4379575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d4379575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 -- 172.21.15.52:0/2898303729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d437693620 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f1686800640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1680004af0 tx=0x7f1680003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4477d1bb1af69de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f166f400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680058060 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f1680031860 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.856+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1680036030 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f168002f610 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 10 client.15861 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d43795c5e0 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680036030 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f1680043070 con 0x55d43794e9b0 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 10 client.15861 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 20 client.15861 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d437969a10 0x55d43796be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 4 client.15861 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.857+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4378c88b0 con 0x55d437969a10 2024-04-26T01:52:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.858+0000 7f167fe00640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d437969a10 0x55d43796be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.858+0000 7f167fe00640 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d437969a10 0x55d43796be00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=937534cb19e7867a server_cookie=903ef7e8eb1243db in_seq=0 out_seq=0 2024-04-26T01:52:15.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.858+0000 7f166f400640 10 client.15861 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.859+0000 7f166f400640 1 -- 172.21.15.52:0/2898303729 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4378c88b0 con 0x55d437969a10 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.859+0000 7f166f400640 10 client.15861 handle_command_reply: tid=0 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 client.15861 shutdown 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 msgr2=0x55d437956b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1680004af0 tx=0x7f1680003df0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 shutdown_connections 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d437969a10 0x55d43796be00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d4379575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d4379570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 --2- 172.21.15.52:0/2898303729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437956b60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f166f400640 0 client.15861 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 >> 172.21.15.52:0/2898303729 conn(0x55d43794aec0 msgr2=0x55d437950c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 shutdown_connections 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 1 -- 172.21.15.52:0/2898303729 wait complete. 2024-04-26T01:52:15.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.860+0000 7f168fb9c740 20 client.15861 trim_cache size 0 max 16384 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.876+0000 7f168fb9c740 1 Processor -- start 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 -- start start 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437950a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d437959b90 0x55d4379716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f16881b2430 con 0x55d437959b90 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f16881afd00 con 0x55d43794f2d0 2024-04-26T01:52:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f168fb9c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f16881adb70 con 0x55d43794e9b0 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f1686800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d437959b90 0x55d4379716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f1686800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50596/0 (socket says 172.21.15.52:50596) 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.877+0000 7f1686800640 1 -- 172.21.15.52:0/3084815895 learned_addr learned my addr 172.21.15.52:0/3084815895 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f167fe00640 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437950a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 -- 172.21.15.52:0/3084815895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 msgr2=0x55d437950a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437950a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 -- 172.21.15.52:0/3084815895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d437959b90 msgr2=0x55d4379716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d437959b90 0x55d4379716e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 -- 172.21.15.52:0/3084815895 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4379550b0 con 0x55d43794e9b0 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1686800640 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1680004430 tx=0x7f1680003df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edc72d1b27aba0d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.878+0000 7f1676600640 1 -- 172.21.15.52:0/3084815895 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680058070 con 0x55d43794e9b0 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f1676600640 1 -- 172.21.15.52:0/3084815895 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680036040 con 0x55d43794e9b0 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f1676600640 1 -- 172.21.15.52:0/3084815895 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1680031660 con 0x55d43794e9b0 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 -- 172.21.15.52:0/3084815895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 msgr2=0x55d4379654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1680004430 tx=0x7f1680003df0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 -- 172.21.15.52:0/3084815895 shutdown_connections 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d437959b90 0x55d4379716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d43794f2d0 0x55d437950a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 --2- 172.21.15.52:0/3084815895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794e9b0 0x55d4379654a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 -- 172.21.15.52:0/3084815895 >> 172.21.15.52:0/3084815895 conn(0x55d43794aec0 msgr2=0x55d437950360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 -- 172.21.15.52:0/3084815895 shutdown_connections 2024-04-26T01:52:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.879+0000 7f168fb9c740 1 -- 172.21.15.52:0/3084815895 wait complete. 2024-04-26T01:52:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.880+0000 7f168fb9c740 1 -- start start 2024-04-26T01:52:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.880+0000 7f168fb9c740 1 Processor -- start 2024-04-26T01:52:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d43795f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d43795f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f16881ae1c0 con 0x55d43794e9b0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f16881af630 con 0x55d437959b90 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f16881ae890 con 0x55d43794f2d0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f168fb9c740 10 client.0 _ll_register_callbacks cb 0x7f168002f310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f1686800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48402/0 (socket says 172.21.15.52:48402) 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f1686800640 1 -- 172.21.15.52:0/1975158734 learned_addr learned my addr 172.21.15.52:0/1975158734 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f167fe00640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d43795f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.881+0000 7f1686800640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d43795f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 -- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 msgr2=0x55d43795f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d43795f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 -- 172.21.15.52:0/1975158734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 msgr2=0x55d437961f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 -- 172.21.15.52:0/1975158734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4379cef60 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f1686800640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f167fe00640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d43795f290 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55d4379563c0 tx=0x7f1670005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbddcc160d986a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f166f400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670018b90 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f1670018d30 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.882+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1680036040 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f167002f050 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 10 client.25987 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d4379d0170 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680036040 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f1670004a10 con 0x55d43794f2d0 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 10 client.25987 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 20 client.25987 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d4379d0870 0x55d4379d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 4 client.25987 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.883+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d4378c2320 con 0x55d4379d0870 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.884+0000 7f1686800640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d4379d0870 0x55d4379d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.884+0000 7f1686800640 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d4379d0870 0x55d4379d2c90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0dc3bfed25f9751 server_cookie=eb00c43352a292ee in_seq=0 out_seq=0 2024-04-26T01:52:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.884+0000 7f166f400640 10 client.25987 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:15.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:15 smithi191 ceph-mon[32745]: pgmap v223: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 257 KiB/s wr, 40 op/s 2024-04-26T01:52:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f166f400640 1 -- 172.21.15.52:0/1975158734 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d4378c2320 con 0x55d4379d0870 2024-04-26T01:52:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f166f400640 10 client.25987 handle_command_reply: tid=0 2024-04-26T01:52:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 client.25987 shutdown 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 msgr2=0x55d43795f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d43795f290 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55d4379563c0 tx=0x7f1670005f90 comp rx=0 tx=0).stop 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 shutdown_connections 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d4379d0870 0x55d4379d2c90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d437959b90 0x55d43795f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d43794f2d0 0x55d43795f290 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 --2- 172.21.15.52:0/1975158734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d43794e9b0 0x55d437961f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f166f400640 0 client.25987 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 >> 172.21.15.52:0/1975158734 conn(0x55d43794aec0 msgr2=0x55d4379573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 shutdown_connections 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 1 -- 172.21.15.52:0/1975158734 wait complete. 2024-04-26T01:52:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.919+0000 7f168fb9c740 20 client.25987 trim_cache size 0 max 16384 2024-04-26T01:52:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:52:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:52:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:52:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 msgr2=0x7f16680a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 0x7f16680a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55d43794ff50 tx=0x7f16780d3ce0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 msgr2=0x7f168819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f167800d950 tx=0x7f167800de10 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 shutdown_connections 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f16680a3d10 0x7f16680a61d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f168819f010 0x7f168819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16881052f0 0x7f168819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 --2- 172.21.15.52:0/2807019528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1688100220 0x7f168819a750 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 >> 172.21.15.52:0/2807019528 conn(0x7f1688076db0 msgr2=0x7f16881031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.920+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 shutdown_connections 2024-04-26T01:52:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:15.921+0000 7f1687200640 1 -- 172.21.15.52:0/2807019528 wait complete. 2024-04-26T01:52:15.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:15 smithi183 ceph-mon[32752]: pgmap v223: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 257 KiB/s wr, 40 op/s 2024-04-26T01:52:15.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:15 smithi052 ceph-mon[27851]: pgmap v223: 97 pgs: 97 active+clean; 76 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 257 KiB/s wr, 40 op/s 2024-04-26T01:52:16.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.02 MB/sec execute 215 sec latency 8355.953 ms 2024-04-26T01:52:17.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 3.01 MB/sec execute 216 sec latency 9356.093 ms 2024-04-26T01:52:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:17 smithi191 ceph-mon[32745]: pgmap v224: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 448 KiB/s wr, 66 op/s 2024-04-26T01:52:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:17 smithi183 ceph-mon[32752]: pgmap v224: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 448 KiB/s wr, 66 op/s 2024-04-26T01:52:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:17 smithi052 ceph-mon[27851]: pgmap v224: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 448 KiB/s wr, 66 op/s 2024-04-26T01:52:18.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.99 MB/sec execute 217 sec latency 10356.220 ms 2024-04-26T01:52:19.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.98 MB/sec execute 218 sec latency 11356.353 ms 2024-04-26T01:52:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:19 smithi191 ceph-mon[32745]: pgmap v225: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:19 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:19.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:19 smithi183 ceph-mon[32752]: pgmap v225: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:19 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:19.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:19 smithi052 ceph-mon[27851]: pgmap v225: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:19.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:19 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:52:20.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.97 MB/sec execute 219 sec latency 12356.428 ms 2024-04-26T01:52:20.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:20 smithi191 ceph-mon[32745]: pgmap v226: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:20.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:20 smithi183 ceph-mon[32752]: pgmap v226: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:20 smithi052 ceph-mon[27851]: pgmap v226: 97 pgs: 97 active+clean; 77 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 427 KiB/s wr, 62 op/s 2024-04-26T01:52:21.102 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.95 MB/sec execute 220 sec latency 13356.565 ms 2024-04-26T01:52:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:21 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:21 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:21 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:52:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:21 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:21 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:21 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:52:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:21 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:52:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:21 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:52:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:21 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:52:22.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.94 MB/sec execute 221 sec latency 14356.700 ms 2024-04-26T01:52:22.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:22 smithi191 ceph-mon[32745]: pgmap v227: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 709 KiB/s wr, 51 op/s 2024-04-26T01:52:22.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:22 smithi183 ceph-mon[32752]: pgmap v227: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 709 KiB/s wr, 51 op/s 2024-04-26T01:52:22.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:22 smithi052 ceph-mon[27851]: pgmap v227: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 709 KiB/s wr, 51 op/s 2024-04-26T01:52:23.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.93 MB/sec execute 222 sec latency 15356.858 ms 2024-04-26T01:52:24.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.91 MB/sec execute 223 sec latency 16356.952 ms 2024-04-26T01:52:24.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:52:25.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.90 MB/sec execute 224 sec latency 17357.036 ms 2024-04-26T01:52:25.186 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:25.347 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:25 smithi052 ceph-mon[27851]: pgmap v228: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 652 KiB/s wr, 47 op/s 2024-04-26T01:52:25.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:25 smithi191 ceph-mon[32745]: pgmap v228: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 652 KiB/s wr, 47 op/s 2024-04-26T01:52:25.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:25 smithi183 ceph-mon[32752]: pgmap v228: 97 pgs: 97 active+clean; 72 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 652 KiB/s wr, 47 op/s 2024-04-26T01:52:26.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.89 MB/sec execute 225 sec latency 18357.145 ms 2024-04-26T01:52:27.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.88 MB/sec execute 226 sec latency 19357.262 ms 2024-04-26T01:52:27.247 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:27 smithi052 ceph-mon[27851]: pgmap v229: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 865 KiB/s wr, 57 op/s 2024-04-26T01:52:27.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:27 smithi191 ceph-mon[32745]: pgmap v229: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 865 KiB/s wr, 57 op/s 2024-04-26T01:52:27.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:27 smithi183 ceph-mon[32752]: pgmap v229: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 865 KiB/s wr, 57 op/s 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 -- 172.21.15.52:0/1907630643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44075590 msgr2=0x7f8c441136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 --2- 172.21.15.52:0/1907630643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44075590 0x7f8c441136d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009990 tx=0x7f8c2c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 -- 172.21.15.52:0/1907630643 shutdown_connections 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 --2- 172.21.15.52:0/1907630643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44113c10 0x7f8c44116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 --2- 172.21.15.52:0/1907630643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44074c70 0x7f8c44075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 --2- 172.21.15.52:0/1907630643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44075590 0x7f8c441136d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 -- 172.21.15.52:0/1907630643 >> 172.21.15.52:0/1907630643 conn(0x7f8c44079ba0 msgr2=0x7f8c44079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 -- 172.21.15.52:0/1907630643 shutdown_connections 2024-04-26T01:52:27.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.667+0000 7f8c49200700 1 -- 172.21.15.52:0/1907630643 wait complete. 2024-04-26T01:52:27.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 Processor -- start 2024-04-26T01:52:27.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 -- start start 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44075590 0x7f8c441a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44113c10 0x7f8c4419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44119250 con 0x7f8c44074c70 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441190d0 con 0x7f8c44113c10 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c49200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441193d0 con 0x7f8c44075590 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.669+0000 7f8c41e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50278/0 (socket says 172.21.15.52:50278) 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 -- 172.21.15.52:0/165528437 learned_addr learned my addr 172.21.15.52:0/165528437 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c3be00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44075590 0x7f8c441a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:27.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c42800700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44113c10 0x7f8c4419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:27.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 -- 172.21.15.52:0/165528437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44075590 msgr2=0x7f8c441a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:27.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44075590 0x7f8c441a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 -- 172.21.15.52:0/165528437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44113c10 msgr2=0x7f8c4419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44113c10 0x7f8c4419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 -- 172.21.15.52:0/165528437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4419fd50 con 0x7f8c44074c70 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.670+0000 7f8c41e00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d950 tx=0x7f8c3400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.671+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34016070 con 0x7f8c44074c70 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.671+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c3400fd40 con 0x7f8c44074c70 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.671+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c441a0040 con 0x7f8c44074c70 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.671+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400e3b0 con 0x7f8c44074c70 2024-04-26T01:52:27.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.671+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c441ac300 con 0x7f8c44074c70 2024-04-26T01:52:27.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.672+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c44069730 con 0x7f8c44074c70 2024-04-26T01:52:27.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.674+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8c34019070 con 0x7f8c44074c70 2024-04-26T01:52:27.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.676+0000 7f8c3aa00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 0x7f8c280a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:27.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.676+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f8c340d0320 con 0x7f8c44074c70 2024-04-26T01:52:27.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.676+0000 7f8c3be00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 0x7f8c280a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:27.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.677+0000 7f8c3be00700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 0x7f8c280a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c44101170 tx=0x7f8c2c005fd0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:27.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.687+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c3409c960 con 0x7f8c44074c70 2024-04-26T01:52:27.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.971+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c441ac530 con 0x7f8c44074c70 2024-04-26T01:52:27.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.973+0000 7f8c3aa00700 1 -- 172.21.15.52:0/165528437 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f8c3409c260 con 0x7f8c44074c70 2024-04-26T01:52:27.972 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:52:27.972 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 msgr2=0x7f8c280a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 0x7f8c280a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c44101170 tx=0x7f8c2c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 msgr2=0x7f8c441a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d950 tx=0x7f8c3400dc60 comp rx=0 tx=0).stop 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 shutdown_connections 2024-04-26T01:52:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8c280a3db0 0x7f8c280a6270 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c44113c10 0x7f8c4419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c44074c70 0x7f8c441a5430 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 --2- 172.21.15.52:0/165528437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c44075590 0x7f8c441a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.976+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 >> 172.21.15.52:0/165528437 conn(0x7f8c44079ba0 msgr2=0x7f8c4410f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.977+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 shutdown_connections 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:27.977+0000 7f8c49200700 1 -- 172.21.15.52:0/165528437 wait complete. 2024-04-26T01:52:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:52:28.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.86 MB/sec execute 227 sec latency 20357.377 ms 2024-04-26T01:52:28.317 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:28 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/165528437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:28.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:28 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/165528437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:28.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:28 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/165528437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:52:28.876 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id afeaef32 --release --await' 2024-04-26T01:52:28.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id afeaef32 --release --await 2024-04-26T01:52:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 -- 172.21.15.52:0/2837284327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0102980 msgr2=0x7f9fb01069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 --2- 172.21.15.52:0/2837284327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0102980 0x7f9fb01069c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009900 tx=0x7f9f9c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 -- 172.21.15.52:0/2837284327 shutdown_connections 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 --2- 172.21.15.52:0/2837284327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 --2- 172.21.15.52:0/2837284327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0102980 0x7f9fb01069c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 --2- 172.21.15.52:0/2837284327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 0x7f9fb01023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 -- 172.21.15.52:0/2837284327 >> 172.21.15.52:0/2837284327 conn(0x7f9fb00fd8d0 msgr2=0x7f9fb00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 -- 172.21.15.52:0/2837284327 shutdown_connections 2024-04-26T01:52:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.070+0000 7f9fb6200640 1 -- 172.21.15.52:0/2837284327 wait complete. 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.071+0000 7f9fb6200640 1 Processor -- start 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.071+0000 7f9fb6200640 1 -- start start 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb0112030 con 0x7f9fb0074cd0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb0111eb0 con 0x7f9fb0101fd0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb6200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01121b0 con 0x7f9fb01070a0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb5800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52962/0 (socket says 172.21.15.52:52962) 2024-04-26T01:52:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fafe00640 1 -- 172.21.15.52:0/1692638361 learned_addr learned my addr 172.21.15.52:0/1692638361 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.072+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50310/0 (socket says 172.21.15.52:50310) 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 -- 172.21.15.52:0/1692638361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 msgr2=0x7f9fb0074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 -- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 msgr2=0x7f9fb01a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 -- 172.21.15.52:0/1692638361 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb01a6ec0 con 0x7f9fb01070a0 2024-04-26T01:52:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005ab0 tx=0x7f9f9c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fad600640 1 -- 172.21.15.52:0/1692638361 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c03f070 con 0x7f9fb01070a0 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb01a7150 con 0x7f9fb01070a0 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.073+0000 7f9fad600640 1 -- 172.21.15.52:0/1692638361 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f9c031b50 con 0x7f9fb01070a0 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.074+0000 7f9fad600640 1 -- 172.21.15.52:0/1692638361 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c035030 con 0x7f9fb01070a0 2024-04-26T01:52:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.074+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb01a7640 con 0x7f9fb01070a0 2024-04-26T01:52:29.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.077+0000 7f9fad600640 1 -- 172.21.15.52:0/1692638361 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9f9c042070 con 0x7f9fb01070a0 2024-04-26T01:52:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fad600640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 0x7f9f980a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 Processor -- start 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fad600640 1 -- 172.21.15.52:0/1692638361 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f9f9c0f0e30 con 0x7f9fb01070a0 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 -- start start 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 0x7f9f980a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 0x55737e785d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e792f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55737e6eb900 con 0x55737e790b90 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55737e691ef0 con 0x55737e7862d0 2024-04-26T01:52:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.079+0000 7f9fba578740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55737e6e78c0 con 0x55737e7859b0 2024-04-26T01:52:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fb5800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e792f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 0x55737e785d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 0x55737e785d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52964/0 (socket says 172.21.15.52:52964) 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1169667706 learned_addr learned my addr 172.21.15.52:0/1169667706 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:32952/0 (socket says 172.21.15.52:32952) 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 0x7f9f980a61d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40095f0 tx=0x7f9fa4005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 -- 172.21.15.52:0/1169667706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 msgr2=0x55737e785d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 0x55737e785d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 -- 172.21.15.52:0/1169667706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 msgr2=0x55737e792f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e792f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.080+0000 7f9fafe00640 1 -- 172.21.15.52:0/1169667706 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55737e3fe210 con 0x55737e7862d0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.081+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0ee520 tx=0x7f9f9c040fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ef4110a17471701 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.081+0000 7f9fa2000640 1 -- 172.21.15.52:0/1169667706 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c049050 con 0x55737e7862d0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.081+0000 7f9fa2000640 1 -- 172.21.15.52:0/1169667706 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f9c0c1120 con 0x55737e7862d0 2024-04-26T01:52:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.081+0000 7f9fa2000640 1 -- 172.21.15.52:0/1169667706 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0c5550 con 0x55737e7862d0 2024-04-26T01:52:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 -- 172.21.15.52:0/1169667706 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 msgr2=0x55737e790650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0ee520 tx=0x7f9f9c040fc0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 -- 172.21.15.52:0/1169667706 shutdown_connections 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e792f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e790650 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 --2- 172.21.15.52:0/1169667706 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7859b0 0x55737e785d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 -- 172.21.15.52:0/1169667706 >> 172.21.15.52:0/1169667706 conn(0x55737e781ec0 msgr2=0x55737e7822a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 -- 172.21.15.52:0/1169667706 shutdown_connections 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.082+0000 7f9fba578740 1 -- 172.21.15.52:0/1169667706 wait complete. 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.083+0000 7f9fba578740 1 -- start start 2024-04-26T01:52:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.083+0000 7f9fba578740 1 Processor -- start 2024-04-26T01:52:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e78dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e78e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55737e793580 con 0x55737e790b90 2024-04-26T01:52:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55737e6a5310 con 0x55737e7859b0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55737e793700 con 0x55737e7862d0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fba578740 10 client.0 _ll_register_callbacks cb 0x55737e673cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fb5800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e78e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fafe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52974/0 (socket says 172.21.15.52:52974) 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fafe00640 1 -- 172.21.15.52:0/2882019172 learned_addr learned my addr 172.21.15.52:0/2882019172 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.084+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e78dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 -- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 msgr2=0x55737e78dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e78dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 -- 172.21.15.52:0/2882019172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 msgr2=0x55737e78e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e78e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 -- 172.21.15.52:0/2882019172 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55737e4ca620 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e78dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fafe00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c031440 tx=0x7f9f9c005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f34c944a8408c8bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9f96a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0d61a0 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f9f9c0b9490 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.085+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c0c1120 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9f9c049020 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 10 client.26005 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55737e793580 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f9c0c1120 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f9f9c03bb90 con 0x55737e7862d0 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 10 client.26005 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 10 client.26005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 20 client.26005 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e7a09e0 0x55737e7a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 4 client.26005 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.086+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55737e6ff8b0 con 0x55737e7a09e0 2024-04-26T01:52:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.087+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e7a09e0 0x55737e7a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.088+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e7a09e0 0x55737e7a2dd0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f9f213b5201d242 server_cookie=5d71957d2255d53a in_seq=0 out_seq=0 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.088+0000 7f9f96a00640 10 client.26005 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.088+0000 7f9f96a00640 1 -- 172.21.15.52:0/2882019172 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55737e6ff8b0 con 0x55737e7a09e0 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9f96a00640 10 client.26005 handle_command_reply: tid=0 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 client.26005 shutdown 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 msgr2=0x55737e78e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c031440 tx=0x7f9f9c005b50 comp rx=0 tx=0).stop 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 shutdown_connections 2024-04-26T01:52:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e7a09e0 0x55737e7a2dd0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e78e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9f96a00640 0 client.26005 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e78e110 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 --2- 172.21.15.52:0/2882019172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e78dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 >> 172.21.15.52:0/2882019172 conn(0x55737e781ec0 msgr2=0x55737e787bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 shutdown_connections 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 1 -- 172.21.15.52:0/2882019172 wait complete. 2024-04-26T01:52:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.089+0000 7f9fba578740 20 client.26005 trim_cache size 0 max 16384 2024-04-26T01:52:29.103 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.85 MB/sec execute 228 sec latency 21357.474 ms 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.110+0000 7f9fba578740 1 Processor -- start 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.110+0000 7f9fba578740 1 -- start start 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 0x55737e7817a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e788370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 0x55737e7a8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01b24c0 con 0x55737e7859b0 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01afd90 con 0x55737e7862d0 2024-04-26T01:52:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fba578740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01adc00 con 0x55737e790b90 2024-04-26T01:52:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 0x55737e7817a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fb5800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 0x55737e7a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 0x55737e7817a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50332/0 (socket says 172.21.15.52:50332) 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1593616759 learned_addr learned my addr 172.21.15.52:0/1593616759 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.111+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e788370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 -- 172.21.15.52:0/1593616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 msgr2=0x55737e788370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e788370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 -- 172.21.15.52:0/1593616759 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 msgr2=0x55737e7817a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 0x55737e7817a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 -- 172.21.15.52:0/1593616759 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55737e7a3400 con 0x55737e790b90 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 0x55737e7a8690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55737e7832e0 tx=0x7f9fa8009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832a68689db1f9cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fa1600640 1 -- 172.21.15.52:0/1593616759 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8022070 con 0x55737e790b90 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fa1600640 1 -- 172.21.15.52:0/1593616759 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fa8003e20 con 0x55737e790b90 2024-04-26T01:52:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.112+0000 7f9fa1600640 1 -- 172.21.15.52:0/1593616759 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80042b0 con 0x55737e790b90 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.113+0000 7f9fba578740 1 -- 172.21.15.52:0/1593616759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 msgr2=0x55737e7a8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.113+0000 7f9fba578740 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 0x55737e7a8690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55737e7832e0 tx=0x7f9fa8009500 comp rx=0 tx=0).stop 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 -- 172.21.15.52:0/1593616759 shutdown_connections 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e790b90 0x55737e7a8690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7862d0 0x55737e788370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 --2- 172.21.15.52:0/1593616759 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e7859b0 0x55737e7817a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 -- 172.21.15.52:0/1593616759 >> 172.21.15.52:0/1593616759 conn(0x55737e781ec0 msgr2=0x55737e78e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 -- 172.21.15.52:0/1593616759 shutdown_connections 2024-04-26T01:52:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.114+0000 7f9fba578740 1 -- 172.21.15.52:0/1593616759 wait complete. 2024-04-26T01:52:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 -- start start 2024-04-26T01:52:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 Processor -- start 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e796b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e805f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01ae250 con 0x55737e790b90 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01af6c0 con 0x55737e7859b0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01ae920 con 0x55737e7862d0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.115+0000 7f9fba578740 10 client.0 _ll_register_callbacks cb 0x7f9f9c0bcca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:32976/0 (socket says 172.21.15.52:32976) 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1054019162 learned_addr learned my addr 172.21.15.52:0/1054019162 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb5800640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e805f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e796b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1054019162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 msgr2=0x55737e796b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e796b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1054019162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 msgr2=0x55737e805f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e805f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fb4e00640 1 -- 172.21.15.52:0/1054019162 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55737e7973d0 con 0x55737e7859b0 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.116+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e796b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.117+0000 7f9fb4e00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9fa402a500 tx=0x7f9fa4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8651839e8a26f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.117+0000 7f9f96a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:52:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.117+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40035f0 con 0x55737e7859b0 2024-04-26T01:52:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.117+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f9fa4003790 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.117+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fa8003e20 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9fa4003990 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 10 client.26006 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55737e807590 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fa8003e20 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f9fa4031070 con 0x55737e7859b0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 10 client.26006 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 10 client.26006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 20 client.26006 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e807d10 0x55737e80a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 4 client.26006 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.118+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55737e6f9320 con 0x55737e807d10 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.119+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e807d10 0x55737e80a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.119+0000 7f9fafe00640 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e807d10 0x55737e80a130 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb8b0f70df1d1dd2 server_cookie=fa7929bee766a145 in_seq=0 out_seq=0 2024-04-26T01:52:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.119+0000 7f9f96a00640 10 client.26006 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.147+0000 7f9f96a00640 1 -- 172.21.15.52:0/1054019162 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55737e6f9320 con 0x55737e807d10 2024-04-26T01:52:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.147+0000 7f9f96a00640 10 client.26006 handle_command_reply: tid=0 2024-04-26T01:52:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.147+0000 7f9fba578740 1 client.26006 shutdown 2024-04-26T01:52:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 msgr2=0x55737e796600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9fa402a500 tx=0x7f9fa4004730 comp rx=0 tx=0).stop 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 shutdown_connections 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55737e807d10 0x55737e80a130 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55737e790b90 0x55737e805f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55737e7862d0 0x55737e796b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 --2- 172.21.15.52:0/1054019162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55737e7859b0 0x55737e796600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 >> 172.21.15.52:0/1054019162 conn(0x55737e781ec0 msgr2=0x55737e78f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9f96a00640 0 client.26006 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 shutdown_connections 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 1 -- 172.21.15.52:0/1054019162 wait complete. 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.148+0000 7f9fba578740 20 client.26006 trim_cache size 0 max 16384 2024-04-26T01:52:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 msgr2=0x7f9f980a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 0x7f9f980a61d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40095f0 tx=0x7f9fa4005f90 comp rx=0 tx=0).stop 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 msgr2=0x7f9fb0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005ab0 tx=0x7f9f9c003b90 comp rx=0 tx=0).stop 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 shutdown_connections 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9f980a3d10 0x7f9f980a61d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9fb0074cd0 0x7f9fb01a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fb01070a0 0x7f9fb0074790 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 --2- 172.21.15.52:0/1692638361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.149+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 >> 172.21.15.52:0/1692638361 conn(0x7f9fb00fd8d0 msgr2=0x7f9fb00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.150+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 shutdown_connections 2024-04-26T01:52:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:29.150+0000 7f9fb6200640 1 -- 172.21.15.52:0/1692638361 wait complete. 2024-04-26T01:52:29.350 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:52:29.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:29 smithi191 ceph-mon[32745]: pgmap v230: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 680 KiB/s wr, 32 op/s 2024-04-26T01:52:29.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:29 smithi183 ceph-mon[32752]: pgmap v230: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 680 KiB/s wr, 32 op/s 2024-04-26T01:52:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:29 smithi052 ceph-mon[27851]: pgmap v230: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 680 KiB/s wr, 32 op/s 2024-04-26T01:52:30.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.84 MB/sec execute 229 sec latency 22357.584 ms 2024-04-26T01:52:31.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.83 MB/sec execute 230 sec latency 23357.739 ms 2024-04-26T01:52:31.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:31 smithi052 ceph-mon[27851]: pgmap v231: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 681 KiB/s wr, 32 op/s 2024-04-26T01:52:31.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:31 smithi191 ceph-mon[32745]: pgmap v231: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 681 KiB/s wr, 32 op/s 2024-04-26T01:52:31.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:31 smithi183 ceph-mon[32752]: pgmap v231: 97 pgs: 97 active+clean; 72 MiB data, 942 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 681 KiB/s wr, 32 op/s 2024-04-26T01:52:32.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.080+0000 7f303aa00700 1 -- 172.21.15.52:0/1786928909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 msgr2=0x7f3034106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.080+0000 7f303aa00700 1 --2- 172.21.15.52:0/1786928909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f3034106780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3020009990 tx=0x7f3020009ca0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 -- 172.21.15.52:0/1786928909 shutdown_connections 2024-04-26T01:52:32.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 --2- 172.21.15.52:0/1786928909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 0x7f3034107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 --2- 172.21.15.52:0/1786928909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f3034101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 --2- 172.21.15.52:0/1786928909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f3034106780 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 -- 172.21.15.52:0/1786928909 >> 172.21.15.52:0/1786928909 conn(0x7f30340fd3f0 msgr2=0x7f30340ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 -- 172.21.15.52:0/1786928909 shutdown_connections 2024-04-26T01:52:32.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.081+0000 7f303aa00700 1 -- 172.21.15.52:0/1786928909 wait complete. 2024-04-26T01:52:32.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.082+0000 7f303aa00700 1 Processor -- start 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.082+0000 7f303aa00700 1 -- start start 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.082+0000 7f303aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.082+0000 7f303aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f303419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.082+0000 7f303aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 0x7f30341a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f303aa00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30341191a0 con 0x7f30341022b0 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f303aa00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3034119020 con 0x7f3034106da0 2024-04-26T01:52:32.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f303aa00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3034119320 con 0x7f3034101990 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3038c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f303419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f303a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 0x7f30341a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53014/0 (socket says 172.21.15.52:53014) 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 -- 172.21.15.52:0/1482803923 learned_addr learned my addr 172.21.15.52:0/1482803923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 -- 172.21.15.52:0/1482803923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 msgr2=0x7f30341a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 0x7f30341a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 -- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 msgr2=0x7f303419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f303419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3039600700 1 -- 172.21.15.52:0/1482803923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30341a46a0 con 0x7f3034101990 2024-04-26T01:52:32.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.083+0000 7f3038c00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f303419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f3039600700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f302800e9a0 tx=0x7f302800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f3032000700 1 -- 172.21.15.52:0/1482803923 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302800c9c0 con 0x7f3034101990 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30341a4990 con 0x7f3034101990 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f3032000700 1 -- 172.21.15.52:0/1482803923 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302800cb60 con 0x7f3034101990 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f3032000700 1 -- 172.21.15.52:0/1482803923 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3028018680 con 0x7f3034101990 2024-04-26T01:52:32.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.084+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30341ac210 con 0x7f3034101990 2024-04-26T01:52:32.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.089+0000 7f3032000700 1 -- 172.21.15.52:0/1482803923 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3028010430 con 0x7f3034101990 2024-04-26T01:52:32.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.090+0000 7f3046a7bb80 1 Processor -- start 2024-04-26T01:52:32.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.090+0000 7f3046a7bb80 1 -- start start 2024-04-26T01:52:32.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fecd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fedaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555b9fe1ff90 con 0x555b9feccea0 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b9fdc6830 con 0x555b9fed8670 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555b9fe1bf50 con 0x555b9fecd7c0 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f303a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fecd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f303a000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fecd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50358/0 (socket says 172.21.15.52:50358) 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f303a000700 1 -- 172.21.15.52:0/1978802295 learned_addr learned my addr 172.21.15.52:0/1978802295 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3039600700 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3032000700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 0x7f301c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.091+0000 7f3038c00700 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fedaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f303a000700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 0x7f301c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 -- 172.21.15.52:0/1978802295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 msgr2=0x555b9fed8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 -- 172.21.15.52:0/1978802295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 msgr2=0x555b9fecd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fecd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 -- 172.21.15.52:0/1978802295 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b9fdbc780 con 0x555b9fed8670 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3032000700 1 -- 172.21.15.52:0/1482803923 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f30280d0aa0 con 0x7f3034101990 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.092+0000 7f3038c00700 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fedaa60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f30200059d0 tx=0x7f30200096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8926b8b075ea5d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f303a000700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 0x7f301c0a62f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f80 tx=0x7f302c004b40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f3026a00700 1 -- 172.21.15.52:0/1978802295 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020027070 con 0x555b9fed8670 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f3026a00700 1 -- 172.21.15.52:0/1978802295 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3020003680 con 0x555b9fed8670 2024-04-26T01:52:32.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f3026a00700 1 -- 172.21.15.52:0/1978802295 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302000f6a0 con 0x555b9fed8670 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1978802295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 msgr2=0x555b9fedaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.093+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fedaa60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f30200059d0 tx=0x7f30200096a0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1978802295 shutdown_connections 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fedaa60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fecd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1978802295 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1978802295 >> 172.21.15.52:0/1978802295 conn(0x555b9fdfc050 msgr2=0x555b9fec93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1978802295 shutdown_connections 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1978802295 wait complete. 2024-04-26T01:52:32.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.094+0000 7f3046a7bb80 1 -- start start 2024-04-26T01:52:32.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 Processor -- start 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fed7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fed5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555b9fedb0a0 con 0x555b9feccea0 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b9fdd9960 con 0x555b9fed8670 2024-04-26T01:52:32.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555b9fedb220 con 0x555b9fecd7c0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.095+0000 7f3046a7bb80 10 client.0 _ll_register_callbacks cb 0x555b9fdb8820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3039600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fed7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53022/0 (socket says 172.21.15.52:53022) 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3039600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fed7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50370/0 (socket says 172.21.15.52:50370) 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 -- 172.21.15.52:0/729427943 learned_addr learned my addr 172.21.15.52:0/729427943 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f303a000700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fed5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 -- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 msgr2=0x555b9fed5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fed5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 -- 172.21.15.52:0/729427943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 msgr2=0x555b9fed7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fed7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 -- 172.21.15.52:0/729427943 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b9fd6d9d0 con 0x555b9fecd7c0 2024-04-26T01:52:32.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f3038c00700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3020009350 tx=0x7f302000bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c8490a1348a7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.096+0000 7f301b400700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020030060 con 0x555b9fecd7c0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f3020005270 con 0x555b9fecd7c0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3020003680 con 0x555b9fecd7c0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3020027040 con 0x555b9fecd7c0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f3046a7bb80 10 client.26023 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555b9fdb6bf0 con 0x555b9fecd7c0 2024-04-26T01:52:32.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3020003680 con 0x555b9fecd7c0 2024-04-26T01:52:32.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.097+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3020021030 con 0x555b9fecd7c0 2024-04-26T01:52:32.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 10 client.26023 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:32.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 10 client.26023 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:52:32.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 20 client.26023 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9fee7c60 0x555b9feea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 4 client.26023 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b9fe2d780 con 0x555b9fee7c60 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.098+0000 7f3039600700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9fee7c60 0x555b9feea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.099+0000 7f3039600700 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9fee7c60 0x555b9feea050 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7854ad441d25a554 server_cookie=417e385805529b37 in_seq=0 out_seq=0 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.099+0000 7f301b400700 10 client.26023 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f301b400700 1 -- 172.21.15.52:0/729427943 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b9fe2d780 con 0x555b9fee7c60 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f301b400700 10 client.26023 handle_command_reply: tid=0 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 client.26023 shutdown 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 msgr2=0x555b9fed5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3020009350 tx=0x7f302000bf30 comp rx=0 tx=0).stop 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 shutdown_connections 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9fee7c60 0x555b9feea050 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9fed5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9feccea0 0x555b9fed7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f301b400700 0 client.26023 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/729427943 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9fed5130 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 >> 172.21.15.52:0/729427943 conn(0x555b9fdfc050 msgr2=0x555b9fec8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 shutdown_connections 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.100+0000 7f3046a7bb80 1 -- 172.21.15.52:0/729427943 wait complete. 2024-04-26T01:52:32.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.101+0000 7f3046a7bb80 20 client.26023 trim_cache size 0 max 16384 2024-04-26T01:52:32.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174127 2.81 MB/sec execute 231 sec latency 24357.821 ms 2024-04-26T01:52:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 Processor -- start 2024-04-26T01:52:32.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 -- start start 2024-04-26T01:52:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 0x555b9fed7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 0x555b9feed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9feed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b5f00 con 0x555b9fecd7c0 2024-04-26T01:52:32.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b3790 con 0x555b9fed8670 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.125+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b1600 con 0x555b9feccea0 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 0x555b9feed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3039600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 0x555b9fed7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3039600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 0x555b9fed7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53034/0 (socket says 172.21.15.52:53034) 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3039600700 1 -- 172.21.15.52:0/285550927 learned_addr learned my addr 172.21.15.52:0/285550927 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f303a000700 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9feed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 -- 172.21.15.52:0/285550927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 msgr2=0x555b9fed7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 0x555b9fed7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 -- 172.21.15.52:0/285550927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 msgr2=0x555b9feed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9feed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f3038c00700 1 -- 172.21.15.52:0/285550927 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b9fecd4d0 con 0x555b9fecd7c0 2024-04-26T01:52:32.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.126+0000 7f303a000700 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9feed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:32.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.127+0000 7f3038c00700 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 0x555b9feed080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f30200098e0 tx=0x7f30200096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4112a2cb58eb270 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.127+0000 7f3026000700 1 -- 172.21.15.52:0/285550927 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020030070 con 0x555b9fecd7c0 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.127+0000 7f3026000700 1 -- 172.21.15.52:0/285550927 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3020017d70 con 0x555b9fecd7c0 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.127+0000 7f3026000700 1 -- 172.21.15.52:0/285550927 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020005680 con 0x555b9fecd7c0 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- 172.21.15.52:0/285550927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 msgr2=0x555b9feed080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 0x555b9feed080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f30200098e0 tx=0x7f30200096a0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- 172.21.15.52:0/285550927 shutdown_connections 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9fed8670 0x555b9feed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9feccea0 0x555b9fed7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/285550927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fecd7c0 0x555b9feed080 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- 172.21.15.52:0/285550927 >> 172.21.15.52:0/285550927 conn(0x555b9fdfc050 msgr2=0x555b9fed6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- 172.21.15.52:0/285550927 shutdown_connections 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- 172.21.15.52:0/285550927 wait complete. 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.128+0000 7f3046a7bb80 1 -- start start 2024-04-26T01:52:32.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 Processor -- start 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9ff47d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 0x555b9ff41dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b1c50 con 0x555b9fed8670 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b30c0 con 0x555b9feccea0 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30341b2320 con 0x555b9fecd7c0 2024-04-26T01:52:32.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.129+0000 7f3046a7bb80 10 client.0 _ll_register_callbacks cb 0x555b9fdb8820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33034/0 (socket says 172.21.15.52:33034) 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 -- 172.21.15.52:0/1428706871 learned_addr learned my addr 172.21.15.52:0/1428706871 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3038c00700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9ff47d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f303a000700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 0x555b9ff41dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 -- 172.21.15.52:0/1428706871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 msgr2=0x555b9ff47d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9ff47d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 -- 172.21.15.52:0/1428706871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 msgr2=0x555b9ff41dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 0x555b9ff41dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 -- 172.21.15.52:0/1428706871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b9ff424d0 con 0x555b9feccea0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3038c00700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9ff47d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f303a000700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 0x555b9ff41dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f3039600700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30280993b0 tx=0x7f30280bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84a24c8a6417117b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:32.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.130+0000 7f301b400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302809c340 con 0x555b9feccea0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f302809ace0 con 0x555b9feccea0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3020017d70 con 0x555b9feccea0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f30280b3080 con 0x555b9feccea0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f3046a7bb80 10 client.26024 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555b9ff4f0c0 con 0x555b9feccea0 2024-04-26T01:52:32.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3020017d70 con 0x555b9feccea0 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.131+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f30280ce410 con 0x555b9feccea0 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 10 client.26024 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 10 client.26024 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 20 client.26024 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9ff4f750 0x555b9ff51b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 4 client.26024 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "afeaef32", "release": true, "await": true}] 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "afeaef32", "release": true, "await": true}) -- 0x555b9fe27820 con 0x555b9ff4f750 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3038c00700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9ff4f750 0x555b9ff51b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:32.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f3038c00700 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9ff4f750 0x555b9ff51b70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0f88087519d4063 server_cookie=bace1725757103a in_seq=0 out_seq=0 2024-04-26T01:52:32.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.132+0000 7f301b400700 10 client.26024 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:32.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f301b400700 1 -- 172.21.15.52:0/1428706871 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x555b9fe27820 con 0x555b9ff4f750 2024-04-26T01:52:32.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f301b400700 10 client.26024 handle_command_reply: tid=0 2024-04-26T01:52:32.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f3046a7bb80 1 client.26024 shutdown 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 msgr2=0x555b9ff47850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30280993b0 tx=0x7f30280bd040 comp rx=0 tx=0).stop 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 shutdown_connections 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.138+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555b9ff4f750 0x555b9ff51b70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555b9fed8670 0x555b9ff41dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b9feccea0 0x555b9ff47850 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 --2- 172.21.15.52:0/1428706871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555b9fecd7c0 0x555b9ff47d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f301b400700 0 client.26024 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:32.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 >> 172.21.15.52:0/1428706871 conn(0x555b9fdfc050 msgr2=0x555b9fed71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 shutdown_connections 2024-04-26T01:52:32.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 1 -- 172.21.15.52:0/1428706871 wait complete. 2024-04-26T01:52:32.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.139+0000 7f3046a7bb80 20 client.26024 trim_cache size 0 max 16384 2024-04-26T01:52:32.138 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:52:32.138 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 20, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "afeaef32": { 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 20, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 75.9, 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:52:32.139 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:52:32.140 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 msgr2=0x7f301c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 0x7f301c0a62f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f302c009f80 tx=0x7f302c004b40 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 msgr2=0x7f303419f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f302800e9a0 tx=0x7f302800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 shutdown_connections 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f301c0a3e30 0x7f301c0a62f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3034106da0 0x7f30341a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3034101990 0x7f303419f690 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 --2- 172.21.15.52:0/1482803923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30341022b0 0x7f303419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.140+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 >> 172.21.15.52:0/1482803923 conn(0x7f30340fd3f0 msgr2=0x7f3034104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:32.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.141+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 shutdown_connections 2024-04-26T01:52:32.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:52:32.141+0000 7f303aa00700 1 -- 172.21.15.52:0/1482803923 wait complete. 2024-04-26T01:52:33.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 174560 2.80 MB/sec execute 232 sec latency 24395.212 ms 2024-04-26T01:52:33.117 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'afeaef32', seconds elapsed: 33.1 2024-04-26T01:52:33.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:33 smithi191 ceph-mon[32745]: pgmap v232: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 980 KiB/s wr, 52 op/s 2024-04-26T01:52:33.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:33 smithi183 ceph-mon[32752]: pgmap v232: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 980 KiB/s wr, 52 op/s 2024-04-26T01:52:33.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:33 smithi052 ceph-mon[27851]: pgmap v232: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 980 KiB/s wr, 52 op/s 2024-04-26T01:52:34.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 175411 2.81 MB/sec execute 233 sec latency 23.644 ms 2024-04-26T01:52:35.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 177093 2.84 MB/sec execute 234 sec latency 21.536 ms 2024-04-26T01:52:35.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:35 smithi191 ceph-mon[32745]: pgmap v233: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 573 KiB/s wr, 34 op/s 2024-04-26T01:52:35.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:35 smithi183 ceph-mon[32752]: pgmap v233: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 573 KiB/s wr, 34 op/s 2024-04-26T01:52:35.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:35 smithi052 ceph-mon[27851]: pgmap v233: 97 pgs: 97 active+clean; 72 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 573 KiB/s wr, 34 op/s 2024-04-26T01:52:36.104 INFO:tasks.workunit.client.0.smithi052.stdout: 1 177663 2.85 MB/sec execute 235 sec latency 47.126 ms 2024-04-26T01:52:37.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 178166 2.84 MB/sec execute 236 sec latency 39.582 ms 2024-04-26T01:52:37.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:37 smithi191 ceph-mon[32745]: pgmap v234: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-04-26T01:52:37.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:37 smithi183 ceph-mon[32752]: pgmap v234: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-04-26T01:52:37.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:37 smithi052 ceph-mon[27851]: pgmap v234: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 87 op/s 2024-04-26T01:52:38.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 179070 2.85 MB/sec execute 237 sec latency 46.112 ms 2024-04-26T01:52:39.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 180552 2.87 MB/sec execute 238 sec latency 22.199 ms 2024-04-26T01:52:39.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:39 smithi191 ceph-mon[32745]: pgmap v235: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:39.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:39 smithi183 ceph-mon[32752]: pgmap v235: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:39.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:39 smithi052 ceph-mon[27851]: pgmap v235: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:40.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 181198 2.88 MB/sec execute 239 sec latency 28.346 ms 2024-04-26T01:52:41.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 181506 2.87 MB/sec execute 240 sec latency 21.702 ms 2024-04-26T01:52:41.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:41 smithi191 ceph-mon[32745]: pgmap v236: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:41.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:41 smithi183 ceph-mon[32752]: pgmap v236: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:41.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:41 smithi052 ceph-mon[27851]: pgmap v236: 97 pgs: 97 active+clean; 75 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 870 KiB/s wr, 72 op/s 2024-04-26T01:52:42.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 182365 2.88 MB/sec execute 241 sec latency 26.825 ms 2024-04-26T01:52:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 -- 172.21.15.52:0/3498811416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c104ba0 msgr2=0x7fe81c108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 --2- 172.21.15.52:0/3498811416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c104ba0 0x7fe81c108be0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe804009900 tx=0x7fe80402a410 comp rx=0 tx=0).stop 2024-04-26T01:52:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 -- 172.21.15.52:0/3498811416 shutdown_connections 2024-04-26T01:52:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 --2- 172.21.15.52:0/3498811416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 0x7fe81c110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 --2- 172.21.15.52:0/3498811416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c104ba0 0x7fe81c108be0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 --2- 172.21.15.52:0/3498811416 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 -- 172.21.15.52:0/3498811416 >> 172.21.15.52:0/3498811416 conn(0x7fe81c0ffa30 msgr2=0x7fe81c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 -- 172.21.15.52:0/3498811416 shutdown_connections 2024-04-26T01:52:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.316+0000 7fe820e00640 1 -- 172.21.15.52:0/3498811416 wait complete. 2024-04-26T01:52:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.317+0000 7fe820e00640 1 Processor -- start 2024-04-26T01:52:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.317+0000 7fe820e00640 1 -- start start 2024-04-26T01:52:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 0x7fe81c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 0x7fe81c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c114230 con 0x7fe81c1041f0 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1140b0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe820e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1143b0 con 0x7fe81c1092c0 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe81b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe81b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60210/0 (socket says 172.21.15.52:60210) 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe81b400640 1 -- 172.21.15.52:0/2189412131 learned_addr learned my addr 172.21.15.52:0/2189412131 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 0x7fe81c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.318+0000 7fe812a00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 0x7fe81c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 -- 172.21.15.52:0/2189412131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 msgr2=0x7fe81c1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 0x7fe81c1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 -- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 msgr2=0x7fe81c1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 -- 172.21.15.52:0/2189412131 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c19b0f0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81b400640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 0x7fe81c19aa80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe818c00640 1 -- 172.21.15.52:0/2189412131 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81400ccc0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe818c00640 1 -- 172.21.15.52:0/2189412131 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe814005020 con 0x7fe81c19a5d0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe818c00640 1 -- 172.21.15.52:0/2189412131 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814004a00 con 0x7fe81c19a5d0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.319+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c19b3e0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.320+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1a7630 con 0x7fe81c19a5d0 2024-04-26T01:52:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.323+0000 7fe818c00640 1 -- 172.21.15.52:0/2189412131 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe814004ba0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.324+0000 7fe8253d5740 1 Processor -- start 2024-04-26T01:52:42.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.324+0000 7fe8253d5740 1 -- start start 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b92650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b94f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556904aed900 con 0x556904b882d0 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556904a93ef0 con 0x556904b879b0 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe8253d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556904ae98c0 con 0x556904b92b90 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe812a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe818c00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 0x7fe8000a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe812a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59472/0 (socket says 172.21.15.52:59472) 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe812a00640 1 -- 172.21.15.52:0/2986931929 learned_addr learned my addr 172.21.15.52:0/2986931929 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.325+0000 7fe812a00640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b94f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe818c00640 1 -- 172.21.15.52:0/2189412131 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fe8140d06e0 con 0x7fe81c19a5d0 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe81b400640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b92650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 0x7fe8000a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 -- 172.21.15.52:0/2986931929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 msgr2=0x556904b94f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b94f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 -- 172.21.15.52:0/2986931929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 msgr2=0x556904b92650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b92650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 -- 172.21.15.52:0/2986931929 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556904800210 con 0x556904b879b0 2024-04-26T01:52:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe81b400640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b92650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe812a00640 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x556904b85050 tx=0x7fe804005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a01befcc97b48a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.326+0000 7fe810c00640 1 -- 172.21.15.52:0/2986931929 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804035030 con 0x556904b879b0 2024-04-26T01:52:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.327+0000 7fe810c00640 1 -- 172.21.15.52:0/2986931929 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe804003d50 con 0x556904b879b0 2024-04-26T01:52:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.327+0000 7fe81be00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 0x7fe8000a6250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe8140a4d60 tx=0x7fe8140046c0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.327+0000 7fe810c00640 1 -- 172.21.15.52:0/2986931929 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804031c60 con 0x556904b879b0 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.327+0000 7fe8253d5740 1 -- 172.21.15.52:0/2986931929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 msgr2=0x556904b87d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.327+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x556904b85050 tx=0x7fe804005780 comp rx=0 tx=0).stop 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 -- 172.21.15.52:0/2986931929 shutdown_connections 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b94f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b92650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2986931929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b87d90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 -- 172.21.15.52:0/2986931929 >> 172.21.15.52:0/2986931929 conn(0x556904b83ec0 msgr2=0x556904b842a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 -- 172.21.15.52:0/2986931929 shutdown_connections 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 -- 172.21.15.52:0/2986931929 wait complete. 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 -- start start 2024-04-26T01:52:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.328+0000 7fe8253d5740 1 Processor -- start 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b90130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b92b90 0x556904b90670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556904b95760 con 0x556904b882d0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556904aa7310 con 0x556904b92b90 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556904b958e0 con 0x556904b879b0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe8253d5740 10 client.0 _ll_register_callbacks cb 0x556904a75cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe812a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe812a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:47226/0 (socket says 172.21.15.52:47226) 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe812a00640 1 -- 172.21.15.52:0/686896579 learned_addr learned my addr 172.21.15.52:0/686896579 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe81b400640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b90130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.329+0000 7fe812a00640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b92b90 0x556904b90670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 -- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b92b90 msgr2=0x556904b90670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b92b90 0x556904b90670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 -- 172.21.15.52:0/686896579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 msgr2=0x556904b90130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b90130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 -- 172.21.15.52:0/686896579 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569048cc620 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe812a00640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe804031940 tx=0x7fe804005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74f6cbee19e998e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe809600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804035070 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fe8040038a0 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.330+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe804003d50 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe8040316d0 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 10 client.26041 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556904b95760 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe804003d50 con 0x556904b879b0 2024-04-26T01:52:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fe804049040 con 0x556904b879b0 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 10 client.26041 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 10 client.26041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 20 client.26041 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904ba2c20 0x556904ba5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 4 client.26041 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.331+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556904b018b0 con 0x556904ba2c20 2024-04-26T01:52:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.332+0000 7fe81b400640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904ba2c20 0x556904ba5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.332+0000 7fe81b400640 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904ba2c20 0x556904ba5010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86719462fbc5e311 server_cookie=5f0bc87d01015339 in_seq=0 out_seq=0 2024-04-26T01:52:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.332+0000 7fe809600640 10 client.26041 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:42.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.333+0000 7fe809600640 1 -- 172.21.15.52:0/686896579 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556904b018b0 con 0x556904ba2c20 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.333+0000 7fe809600640 10 client.26041 handle_command_reply: tid=0 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.333+0000 7fe8253d5740 1 client.26041 shutdown 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 msgr2=0x556904b8fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe804031940 tx=0x7fe804005850 comp rx=0 tx=0).stop 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 shutdown_connections 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904ba2c20 0x556904ba5010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b92b90 0x556904b90670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b882d0 0x556904b90130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 --2- 172.21.15.52:0/686896579 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b879b0 0x556904b8fbf0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe809600640 0 client.26041 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 >> 172.21.15.52:0/686896579 conn(0x556904b83ec0 msgr2=0x556904b89c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 shutdown_connections 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 1 -- 172.21.15.52:0/686896579 wait complete. 2024-04-26T01:52:42.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.334+0000 7fe8253d5740 20 client.26041 trim_cache size 0 max 16384 2024-04-26T01:52:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.353+0000 7fe8253d5740 1 Processor -- start 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 -- start start 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 0x556904b8a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b92b90 0x556904b902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b2450 con 0x556904b92b90 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1afd20 con 0x556904b879b0 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe8253d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1adb90 con 0x556904b882d0 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe812a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe812a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b92b90 0x556904b902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe81b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 0x556904b8a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe812a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59492/0 (socket says 172.21.15.52:59492) 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.354+0000 7fe812a00640 1 -- 172.21.15.52:0/2436010479 learned_addr learned my addr 172.21.15.52:0/2436010479 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 -- 172.21.15.52:0/2436010479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 msgr2=0x556904b9e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 -- 172.21.15.52:0/2436010479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b92b90 msgr2=0x556904b902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b92b90 0x556904b902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 -- 172.21.15.52:0/2436010479 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556904b837a0 con 0x556904b882d0 2024-04-26T01:52:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe812a00640 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe81b400640 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 0x556904b8a3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556904ba55b0 tx=0x7fe80c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef244a44b2a6339 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe80be00640 1 -- 172.21.15.52:0/2436010479 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c024070 con 0x556904b882d0 2024-04-26T01:52:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe80be00640 1 -- 172.21.15.52:0/2436010479 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c01e430 con 0x556904b882d0 2024-04-26T01:52:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.355+0000 7fe80be00640 1 -- 172.21.15.52:0/2436010479 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c01c7c0 con 0x556904b882d0 2024-04-26T01:52:42.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 -- 172.21.15.52:0/2436010479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 msgr2=0x556904b8a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 0x556904b8a3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556904ba55b0 tx=0x7fe80c004a40 comp rx=0 tx=0).stop 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 -- 172.21.15.52:0/2436010479 shutdown_connections 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b92b90 0x556904b902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b882d0 0x556904b8a3c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 --2- 172.21.15.52:0/2436010479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b879b0 0x556904b9e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 -- 172.21.15.52:0/2436010479 >> 172.21.15.52:0/2436010479 conn(0x556904b83ec0 msgr2=0x556904b8fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 -- 172.21.15.52:0/2436010479 shutdown_connections 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.356+0000 7fe8253d5740 1 -- 172.21.15.52:0/2436010479 wait complete. 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.357+0000 7fe8253d5740 1 -- start start 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.357+0000 7fe8253d5740 1 Processor -- start 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b879b0 0x556904b9b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b98a80 0x556904b98f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ae1e0 con 0x556904b879b0 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1af650 con 0x556904b98a80 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ae8b0 con 0x556904b92b90 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe8253d5740 10 client.0 _ll_register_callbacks cb 0x7fe804004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe812a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b879b0 0x556904b9b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:47240/0 (socket says 172.21.15.52:47240) 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 -- 172.21.15.52:0/3887868066 learned_addr learned my addr 172.21.15.52:0/3887868066 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe812a00640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b98a80 0x556904b98f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 -- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b98a80 msgr2=0x556904b98f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b98a80 0x556904b98f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 -- 172.21.15.52:0/3887868066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b879b0 msgr2=0x556904b9b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:42.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b879b0 0x556904b9b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.358+0000 7fe81b400640 1 -- 172.21.15.52:0/3887868066 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556904c08160 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe81b400640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe80c022040 tx=0x7fe80c00f6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56f959f13d1bf41f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe809600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c02c030 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fe80c004510 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80c01e430 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe80c036050 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe8253d5740 10 client.26047 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.359+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556904c09450 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c01e430 con 0x556904b92b90 2024-04-26T01:52:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fe80c027070 con 0x556904b92b90 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 10 client.26047 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 10 client.26047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 20 client.26047 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904c09ac0 0x556904c0bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 4 client.26047 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556904afb320 con 0x556904c09ac0 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.360+0000 7fe812a00640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904c09ac0 0x556904c0bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.361+0000 7fe812a00640 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904c09ac0 0x556904c0bee0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33339829b3c831f6 server_cookie=7f40c6c46e71cd13 in_seq=0 out_seq=0 2024-04-26T01:52:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.361+0000 7fe809600640 10 client.26047 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.400+0000 7fe809600640 1 -- 172.21.15.52:0/3887868066 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556904afb320 con 0x556904c09ac0 2024-04-26T01:52:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.400+0000 7fe809600640 10 client.26047 handle_command_reply: tid=0 2024-04-26T01:52:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.400+0000 7fe8253d5740 1 client.26047 shutdown 2024-04-26T01:52:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 msgr2=0x556904b98430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe80c022040 tx=0x7fe80c00f6d0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 shutdown_connections 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556904c09ac0 0x556904c0bee0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556904b98a80 0x556904b98f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556904b92b90 0x556904b98430 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 --2- 172.21.15.52:0/3887868066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556904b879b0 0x556904b9b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe809600640 0 client.26047 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 >> 172.21.15.52:0/3887868066 conn(0x556904b83ec0 msgr2=0x556904b8e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 shutdown_connections 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 1 -- 172.21.15.52:0/3887868066 wait complete. 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.401+0000 7fe8253d5740 20 client.26047 trim_cache size 0 max 16384 2024-04-26T01:52:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 msgr2=0x7fe8000a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 0x7fe8000a6250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe8140a4d60 tx=0x7fe8140046c0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 msgr2=0x7fe81c19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 0x7fe81c19aa80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400ee30 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 shutdown_connections 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe8000a3d90 0x7fe8000a6250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe81c19a5d0 0x7fe81c19aa80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe81c1092c0 0x7fe81c1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 --2- 172.21.15.52:0/2189412131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe81c1041f0 0x7fe81c1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 >> 172.21.15.52:0/2189412131 conn(0x7fe81c0ffa30 msgr2=0x7fe81c100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 shutdown_connections 2024-04-26T01:52:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:42.402+0000 7fe820e00640 1 -- 172.21.15.52:0/2189412131 wait complete. 2024-04-26T01:52:43.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 183673 2.89 MB/sec execute 242 sec latency 74.491 ms 2024-04-26T01:52:43.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:43 smithi191 ceph-mon[32745]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-26T01:52:43.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:43 smithi183 ceph-mon[32752]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-26T01:52:43.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:43 smithi052 ceph-mon[27851]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-04-26T01:52:44.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 184667 2.91 MB/sec execute 243 sec latency 32.317 ms 2024-04-26T01:52:45.105 INFO:tasks.workunit.client.0.smithi052.stdout: 1 184915 2.90 MB/sec execute 244 sec latency 27.246 ms 2024-04-26T01:52:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:45 smithi191 ceph-mon[32745]: pgmap v238: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T01:52:45.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:45 smithi183 ceph-mon[32752]: pgmap v238: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T01:52:45.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:45 smithi052 ceph-mon[27851]: pgmap v238: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T01:52:46.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 185611 2.90 MB/sec execute 245 sec latency 27.178 ms 2024-04-26T01:52:47.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 186371 2.90 MB/sec execute 246 sec latency 26.713 ms 2024-04-26T01:52:47.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:47 smithi183 ceph-mon[32752]: pgmap v239: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-26T01:52:47.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:47 smithi052 ceph-mon[27851]: pgmap v239: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-26T01:52:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:47 smithi191 ceph-mon[32745]: pgmap v239: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-26T01:52:48.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 188064 2.94 MB/sec execute 247 sec latency 28.843 ms 2024-04-26T01:52:49.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 188444 2.93 MB/sec execute 248 sec latency 21.646 ms 2024-04-26T01:52:49.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:49 smithi183 ceph-mon[32752]: pgmap v240: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:49.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:49 smithi052 ceph-mon[27851]: pgmap v240: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:49 smithi191 ceph-mon[32745]: pgmap v240: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:50.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 189289 2.93 MB/sec execute 249 sec latency 21.725 ms 2024-04-26T01:52:51.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 190430 2.93 MB/sec execute 250 sec latency 26.310 ms 2024-04-26T01:52:51.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:51 smithi183 ceph-mon[32752]: pgmap v241: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:51.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:51 smithi052 ceph-mon[27851]: pgmap v241: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:51 smithi191 ceph-mon[32745]: pgmap v241: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-26T01:52:52.106 INFO:tasks.workunit.client.0.smithi052.stdout: 1 191764 2.97 MB/sec execute 251 sec latency 23.336 ms 2024-04-26T01:52:53.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 192012 2.96 MB/sec execute 252 sec latency 26.968 ms 2024-04-26T01:52:53.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:53 smithi183 ceph-mon[32752]: pgmap v242: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-04-26T01:52:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:53 smithi052 ceph-mon[27851]: pgmap v242: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-04-26T01:52:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:53 smithi191 ceph-mon[32745]: pgmap v242: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 175 op/s 2024-04-26T01:52:54.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 192805 2.96 MB/sec execute 253 sec latency 19.365 ms 2024-04-26T01:52:55.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 194325 2.98 MB/sec execute 254 sec latency 30.419 ms 2024-04-26T01:52:55.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:55 smithi183 ceph-mon[32752]: pgmap v243: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 814 KiB/s wr, 114 op/s 2024-04-26T01:52:55.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:55 smithi052 ceph-mon[27851]: pgmap v243: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 814 KiB/s wr, 114 op/s 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.556+0000 7fee2d600640 1 -- 172.21.15.52:0/444920423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28101860 msgr2=0x7fee281058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.556+0000 7fee2d600640 1 --2- 172.21.15.52:0/444920423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28101860 0x7fee281058a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fee14009900 tx=0x7fee1402a410 comp rx=0 tx=0).stop 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 -- 172.21.15.52:0/444920423 shutdown_connections 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 --2- 172.21.15.52:0/444920423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee28105f80 0x7fee2810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 --2- 172.21.15.52:0/444920423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28101860 0x7fee281058a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 --2- 172.21.15.52:0/444920423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 0x7fee28101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 -- 172.21.15.52:0/444920423 >> 172.21.15.52:0/444920423 conn(0x7fee280fc770 msgr2=0x7fee280feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 -- 172.21.15.52:0/444920423 shutdown_connections 2024-04-26T01:52:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.557+0000 7fee2d600640 1 -- 172.21.15.52:0/444920423 wait complete. 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 Processor -- start 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 -- start start 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 0x7fee28102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee281033c0 0x7fee281a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fee28110ef0 con 0x7fee28100eb0 2024-04-26T01:52:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fee28110d70 con 0x7fee28105f80 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.558+0000 7fee2d600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fee28111070 con 0x7fee281033c0 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:39704/0 (socket says 172.21.15.52:39704) 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 0x7fee28102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 -- 172.21.15.52:0/208727941 learned_addr learned my addr 172.21.15.52:0/208727941 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee2cc00640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee281033c0 0x7fee281a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 -- 172.21.15.52:0/208727941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee281033c0 msgr2=0x7fee281a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee281033c0 0x7fee281a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 -- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 msgr2=0x7fee28102940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 0x7fee28102940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.559+0000 7fee27400640 1 -- 172.21.15.52:0/208727941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee281a6e90 con 0x7fee28105f80 2024-04-26T01:52:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee27400640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fee14005ab0 tx=0x7fee14003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee24c00640 1 -- 172.21.15.52:0/208727941 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1403f070 con 0x7fee28105f80 2024-04-26T01:52:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee24c00640 1 -- 172.21.15.52:0/208727941 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee14031b50 con 0x7fee28105f80 2024-04-26T01:52:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee24c00640 1 -- 172.21.15.52:0/208727941 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14035030 con 0x7fee28105f80 2024-04-26T01:52:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee281a7120 con 0x7fee28105f80 2024-04-26T01:52:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.560+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee281a75c0 con 0x7fee28105f80 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 Processor -- start 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 -- start start 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee24c00640 1 -- 172.21.15.52:0/208727941 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fee14042070 con 0x7fee28105f80 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcda2d0 0x563c5bce4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bce6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c5bc3f900 con 0x563c5bcda2d0 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c5bbe5ef0 con 0x563c5bce4b90 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.564+0000 7fee31a31740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c5bc3b8c0 con 0x563c5bcd99b0 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee27e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcda2d0 0x563c5bce4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:47580/0 (socket says 172.21.15.52:47580) 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 -- 172.21.15.52:0/4237619927 learned_addr learned my addr 172.21.15.52:0/4237619927 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee27400640 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bce6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 -- 172.21.15.52:0/4237619927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 msgr2=0x563c5bce6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bce6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 -- 172.21.15.52:0/4237619927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcda2d0 msgr2=0x563c5bce4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcda2d0 0x563c5bce4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 -- 172.21.15.52:0/4237619927 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c5b952210 con 0x563c5bcd99b0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.565+0000 7fee2cc00640 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee20009f50 tx=0x7fee200118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c4398af02ac70b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee1d200640 1 -- 172.21.15.52:0/4237619927 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20004d20 con 0x563c5bcd99b0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee1d200640 1 -- 172.21.15.52:0/4237619927 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee20004050 con 0x563c5bcd99b0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee1d200640 1 -- 172.21.15.52:0/4237619927 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20005280 con 0x563c5bcd99b0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee24c00640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 0x7fee0c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee24c00640 1 -- 172.21.15.52:0/208727941 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fee140f0d00 con 0x7fee28105f80 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.566+0000 7fee27e00640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 0x7fee0c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 -- 172.21.15.52:0/4237619927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 msgr2=0x563c5bcd9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee20009f50 tx=0x7fee200118d0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 -- 172.21.15.52:0/4237619927 shutdown_connections 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bce6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcda2d0 0x563c5bce4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 --2- 172.21.15.52:0/4237619927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcd9d90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 -- 172.21.15.52:0/4237619927 >> 172.21.15.52:0/4237619927 conn(0x563c5bcd5ec0 msgr2=0x563c5bcd62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 -- 172.21.15.52:0/4237619927 shutdown_connections 2024-04-26T01:52:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee31a31740 1 -- 172.21.15.52:0/4237619927 wait complete. 2024-04-26T01:52:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.567+0000 7fee27e00640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 0x7fee0c0a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x563c5bcdaf50 tx=0x7fee18005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 -- start start 2024-04-26T01:52:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 Processor -- start 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcd99b0 0x563c5bce1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bce24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563c5bce77c0 con 0x563c5bce4b90 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563c5bbf9310 con 0x563c5bcd99b0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.568+0000 7fee31a31740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c5bce7940 con 0x563c5bcda2d0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee31a31740 10 client.0 _ll_register_callbacks cb 0x563c5bbc7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:47594/0 (socket says 172.21.15.52:47594) 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 -- 172.21.15.52:0/4180825005 learned_addr learned my addr 172.21.15.52:0/4180825005 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27e00640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcd99b0 0x563c5bce1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee2cc00640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bce24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 -- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcd99b0 msgr2=0x563c5bce1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcd99b0 0x563c5bce1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 -- 172.21.15.52:0/4180825005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 msgr2=0x563c5bce24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bce24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.569+0000 7fee27400640 1 -- 172.21.15.52:0/4180825005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c5ba1e620 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee27400640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fee140ee450 tx=0x7fee14031f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53125e8369adf852 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14049070 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fee140baff0 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee20004050 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fee140bcac0 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee31a31740 10 client.26059 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563c5bce7820 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee20004050 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fee140ee670 con 0x563c5bcda2d0 2024-04-26T01:52:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.570+0000 7fee31a31740 10 client.26059 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee31a31740 10 client.26059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee31a31740 20 client.26059 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bcf4ea0 0x563c5bcf72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee31a31740 4 client.26059 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c5bc538b0 con 0x563c5bcf4ea0 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee27e00640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bcf4ea0 0x563c5bcf72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee27e00640 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bcf4ea0 0x563c5bcf72d0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7d3328075e7c884 server_cookie=bbde62321a8b07ff in_seq=0 out_seq=0 2024-04-26T01:52:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.571+0000 7fee11600640 10 client.26059 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.572+0000 7fee11600640 1 -- 172.21.15.52:0/4180825005 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c5bc538b0 con 0x563c5bcf4ea0 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.572+0000 7fee11600640 10 client.26059 handle_command_reply: tid=0 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.572+0000 7fee31a31740 1 client.26059 shutdown 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.572+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 msgr2=0x563c5bce1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.572+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fee140ee450 tx=0x7fee14031f50 comp rx=0 tx=0).stop 2024-04-26T01:52:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 shutdown_connections 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bcf4ea0 0x563c5bcf72d0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bce24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bce1fb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 --2- 172.21.15.52:0/4180825005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcd99b0 0x563c5bce1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee11600640 0 client.26059 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 >> 172.21.15.52:0/4180825005 conn(0x563c5bcd5ec0 msgr2=0x563c5bcdbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 shutdown_connections 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 1 -- 172.21.15.52:0/4180825005 wait complete. 2024-04-26T01:52:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.573+0000 7fee31a31740 20 client.26059 trim_cache size 0 max 16384 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 Processor -- start 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 -- start start 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcf0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bcd4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fee281b2490 con 0x563c5bce4b90 2024-04-26T01:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fee281afd60 con 0x563c5bcda2d0 2024-04-26T01:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.591+0000 7fee31a31740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fee281adbd0 con 0x563c5bcd99b0 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:39730/0 (socket says 172.21.15.52:39730) 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcf0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 -- 172.21.15.52:0/1931089938 learned_addr learned my addr 172.21.15.52:0/1931089938 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee2cc00640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bcd4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 -- 172.21.15.52:0/1931089938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 msgr2=0x563c5bcf0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcf0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 -- 172.21.15.52:0/1931089938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 msgr2=0x563c5bcd4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bcd4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 -- 172.21.15.52:0/1931089938 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c5bcf7920 con 0x563c5bcda2d0 2024-04-26T01:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27e00640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcf0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee2cc00640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bcd4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee27400640 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee14031440 tx=0x7fee1402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e28fed1523f8fb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee13e00640 1 -- 172.21.15.52:0/1931089938 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14049070 con 0x563c5bcda2d0 2024-04-26T01:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee13e00640 1 -- 172.21.15.52:0/1931089938 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee140bccf0 con 0x563c5bcda2d0 2024-04-26T01:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.592+0000 7fee13e00640 1 -- 172.21.15.52:0/1931089938 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140bc650 con 0x563c5bcda2d0 2024-04-26T01:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 -- 172.21.15.52:0/1931089938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 msgr2=0x563c5bcdc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee14031440 tx=0x7fee1402a410 comp rx=0 tx=0).stop 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 -- 172.21.15.52:0/1931089938 shutdown_connections 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bce4b90 0x563c5bcd4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bcda2d0 0x563c5bcdc370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 --2- 172.21.15.52:0/1931089938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcd99b0 0x563c5bcf0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 -- 172.21.15.52:0/1931089938 >> 172.21.15.52:0/1931089938 conn(0x563c5bcd5ec0 msgr2=0x563c5bcd4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 -- 172.21.15.52:0/1931089938 shutdown_connections 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.593+0000 7fee31a31740 1 -- 172.21.15.52:0/1931089938 wait complete. 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.594+0000 7fee31a31740 1 -- start start 2024-04-26T01:52:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.594+0000 7fee31a31740 1 Processor -- start 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 0x563c5bcecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bcea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bcea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fee281ae220 con 0x563c5bcd99b0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fee281af690 con 0x563c5bce4b90 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fee281ae8f0 con 0x563c5bcda2d0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee31a31740 10 client.0 _ll_register_callbacks cb 0x7fee14005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee27e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 0x563c5bcecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bcea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee27400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bcea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee27e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 0x563c5bcecc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43408/0 (socket says 172.21.15.52:43408) 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee27e00640 1 -- 172.21.15.52:0/3271177604 learned_addr learned my addr 172.21.15.52:0/3271177604 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 -- 172.21.15.52:0/3271177604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 msgr2=0x563c5bcea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bcea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 -- 172.21.15.52:0/3271177604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 msgr2=0x563c5bcecc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 0x563c5bcecc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.595+0000 7fee2cc00640 1 -- 172.21.15.52:0/3271177604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c5bceabe0 con 0x563c5bce4b90 2024-04-26T01:52:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.596+0000 7fee2cc00640 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bcea6a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee2001b040 tx=0x7fee20003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb7ac86e862bf46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:52:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.596+0000 7fee11600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:52:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.596+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee200054d0 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.596+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7fee20005670 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.596+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee140bccf0 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fee20034050 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 10 client.26060 fetch_fsmap learned FSMap version 38 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563c5bd5b150 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee140bccf0 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fee2000fd00 con 0x563c5bce4b90 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 10 client.26060 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 20 client.26060 populate_metadata read hostname 'smithi052' 2024-04-26T01:52:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bd5b690 0x563c5bd5dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:52:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 4 client.26060 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:52:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c5bc4d320 con 0x563c5bd5b690 2024-04-26T01:52:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.597+0000 7fee27e00640 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bd5b690 0x563c5bd5dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:52:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.598+0000 7fee27e00640 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bd5b690 0x563c5bd5dab0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c323362d6cea08a8 server_cookie=5c67bd7b4bf45d95 in_seq=0 out_seq=0 2024-04-26T01:52:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.598+0000 7fee11600640 10 client.26060 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee11600640 1 -- 172.21.15.52:0/3271177604 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563c5bc4d320 con 0x563c5bd5b690 2024-04-26T01:52:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee11600640 10 client.26060 handle_command_reply: tid=0 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 client.26060 shutdown 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 msgr2=0x563c5bcea6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bcea6a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee2001b040 tx=0x7fee20003cd0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 shutdown_connections 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563c5bd5b690 0x563c5bd5dab0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563c5bce4b90 0x563c5bcea6a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c5bcda2d0 0x563c5bcea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 --2- 172.21.15.52:0/3271177604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563c5bcd99b0 0x563c5bcecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 >> 172.21.15.52:0/3271177604 conn(0x563c5bcd5ec0 msgr2=0x563c5bcd5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.640+0000 7fee11600640 0 client.26060 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:52:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.641+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 shutdown_connections 2024-04-26T01:52:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.641+0000 7fee31a31740 1 -- 172.21.15.52:0/3271177604 wait complete. 2024-04-26T01:52:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.641+0000 7fee31a31740 20 client.26060 trim_cache size 0 max 16384 2024-04-26T01:52:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.643+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 msgr2=0x7fee0c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.643+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 0x7fee0c0a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x563c5bcdaf50 tx=0x7fee18005f90 comp rx=0 tx=0).stop 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.643+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 msgr2=0x7fee28102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:52:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.643+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fee14005ab0 tx=0x7fee14003b90 comp rx=0 tx=0).stop 2024-04-26T01:52:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 shutdown_connections 2024-04-26T01:52:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fee0c0a3dc0 0x7fee0c0a6280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee281033c0 0x7fee281a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee28105f80 0x7fee28102e80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 --2- 172.21.15.52:0/208727941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee28100eb0 0x7fee28102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:52:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 >> 172.21.15.52:0/208727941 conn(0x7fee280fc770 msgr2=0x7fee28069a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:52:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 shutdown_connections 2024-04-26T01:52:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:52:55.644+0000 7fee2d600640 1 -- 172.21.15.52:0/208727941 wait complete. 2024-04-26T01:52:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:55 smithi191 ceph-mon[32745]: pgmap v243: 97 pgs: 97 active+clean; 82 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 814 KiB/s wr, 114 op/s 2024-04-26T01:52:56.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 195332 3.00 MB/sec execute 255 sec latency 99.911 ms 2024-04-26T01:52:57.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 195666 2.99 MB/sec execute 256 sec latency 24.161 ms 2024-04-26T01:52:57.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:57 smithi052 ceph-mon[27851]: pgmap v244: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-04-26T01:52:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:57 smithi191 ceph-mon[32745]: pgmap v244: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-04-26T01:52:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:57 smithi183 ceph-mon[32752]: pgmap v244: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 192 op/s 2024-04-26T01:52:58.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 196549 3.00 MB/sec execute 257 sec latency 19.273 ms 2024-04-26T01:52:59.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 197852 3.01 MB/sec execute 258 sec latency 22.254 ms 2024-04-26T01:52:59.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:52:59 smithi052 ceph-mon[27851]: pgmap v245: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:52:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:52:59 smithi191 ceph-mon[32745]: pgmap v245: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:52:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:52:59 smithi183 ceph-mon[32752]: pgmap v245: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:53:00.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 198882 3.03 MB/sec execute 259 sec latency 21.569 ms 2024-04-26T01:53:01.107 INFO:tasks.workunit.client.0.smithi052.stdout: 1 199210 3.02 MB/sec execute 260 sec latency 21.920 ms 2024-04-26T01:53:01.454 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:01 smithi191 ceph-mon[32745]: pgmap v246: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:53:01.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:01 smithi052 ceph-mon[27851]: pgmap v246: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:53:01.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:01 smithi183 ceph-mon[32752]: pgmap v246: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 820 KiB/s wr, 128 op/s 2024-04-26T01:53:02.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 199673 3.01 MB/sec execute 261 sec latency 58.753 ms 2024-04-26T01:53:02.646 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.9 2024-04-26T01:53:02.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:53:03.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 200449 3.01 MB/sec execute 262 sec latency 27.910 ms 2024-04-26T01:53:03.136 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:03.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:03 smithi052 ceph-mon[27851]: pgmap v247: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-04-26T01:53:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:03 smithi191 ceph-mon[32745]: pgmap v247: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-04-26T01:53:03.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:03 smithi183 ceph-mon[32752]: pgmap v247: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-04-26T01:53:04.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 202184 3.05 MB/sec execute 263 sec latency 19.634 ms 2024-04-26T01:53:05.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 202618 3.04 MB/sec execute 264 sec latency 25.295 ms 2024-04-26T01:53:05.217 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:05 smithi052 ceph-mon[27851]: pgmap v248: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 KiB/s wr, 129 op/s 2024-04-26T01:53:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:05 smithi191 ceph-mon[32745]: pgmap v248: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 KiB/s wr, 129 op/s 2024-04-26T01:53:05.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:05 smithi183 ceph-mon[32752]: pgmap v248: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 KiB/s wr, 129 op/s 2024-04-26T01:53:06.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 203423 3.04 MB/sec execute 265 sec latency 18.233 ms 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 -- 172.21.15.52:0/2557014124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2684108a30 msgr2=0x7f2684108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 --2- 172.21.15.52:0/2557014124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2684108a30 0x7f2684108e10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2678009990 tx=0x7f2678009ca0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 -- 172.21.15.52:0/2557014124 shutdown_connections 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 --2- 172.21.15.52:0/2557014124 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406b0f0 0x7f268410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 --2- 172.21.15.52:0/2557014124 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2684108a30 0x7f2684108e10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 --2- 172.21.15.52:0/2557014124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f268406a750 0x7f268406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 -- 172.21.15.52:0/2557014124 >> 172.21.15.52:0/2557014124 conn(0x7f26840788f0 msgr2=0x7f2684078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 -- 172.21.15.52:0/2557014124 shutdown_connections 2024-04-26T01:53:06.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.121+0000 7f268c200700 1 -- 172.21.15.52:0/2557014124 wait complete. 2024-04-26T01:53:06.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.122+0000 7f268c200700 1 Processor -- start 2024-04-26T01:53:06.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 -- start start 2024-04-26T01:53:06.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:06.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f268406b0f0 0x7f268419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:06.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 0x7f268419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:06.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2684112c00 con 0x7f268406a750 2024-04-26T01:53:06.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2684112a80 con 0x7f2684108a30 2024-04-26T01:53:06.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268c200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2684112d80 con 0x7f268406b0f0 2024-04-26T01:53:06.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f2689800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:06.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f2689800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59456/0 (socket says 172.21.15.52:59456) 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f2689800700 1 -- 172.21.15.52:0/14899888 learned_addr learned my addr 172.21.15.52:0/14899888 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f2688e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f268406b0f0 0x7f268419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.123+0000 7f268a200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 0x7f268419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f2689800700 1 -- 172.21.15.52:0/14899888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f268406b0f0 msgr2=0x7f268419b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f2689800700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f268406b0f0 0x7f268419b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f2689800700 1 -- 172.21.15.52:0/14899888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 msgr2=0x7f268419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f2689800700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 0x7f268419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f2689800700 1 -- 172.21.15.52:0/14899888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26841a0340 con 0x7f268406a750 2024-04-26T01:53:06.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.124+0000 7f268a200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 0x7f268419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.125+0000 7f2689800700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2678000c00 tx=0x7f26780057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.125+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678027070 con 0x7f268406a750 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.125+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2678022470 con 0x7f268406a750 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.125+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678017650 con 0x7f268406a750 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.125+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26841a05d0 con 0x7f268406a750 2024-04-26T01:53:06.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.126+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26841a7eb0 con 0x7f268406a750 2024-04-26T01:53:06.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.127+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2684104170 con 0x7f268406a750 2024-04-26T01:53:06.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.129+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f267802a070 con 0x7f268406a750 2024-04-26T01:53:06.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.130+0000 7f2682000700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 0x7f266c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:06.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.130+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8521+0+0 (secure 0 0 0) 0x7f26780d14c0 con 0x7f268406a750 2024-04-26T01:53:06.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.131+0000 7f2688e00700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 0x7f266c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:06.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.131+0000 7f2688e00700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 0x7f266c0a6270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2674009680 tx=0x7f2674009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:06.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.137+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2678013070 con 0x7f268406a750 2024-04-26T01:53:06.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.533+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f268402df80 con 0x7f268406a750 2024-04-26T01:53:06.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.535+0000 7f2682000700 1 -- 172.21.15.52:0/14899888 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f267809db90 con 0x7f268406a750 2024-04-26T01:53:06.534 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:53:06.535 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:53:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 msgr2=0x7f266c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 0x7f266c0a6270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2674009680 tx=0x7f2674009260 comp rx=0 tx=0).stop 2024-04-26T01:53:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 msgr2=0x7f268419b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2678000c00 tx=0x7f26780057c0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 shutdown_connections 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f266c0a3db0 0x7f266c0a6270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2684108a30 0x7f268419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f268406a750 0x7f268419b330 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 --2- 172.21.15.52:0/14899888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f268406b0f0 0x7f268419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 >> 172.21.15.52:0/14899888 conn(0x7f26840788f0 msgr2=0x7f268410de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 shutdown_connections 2024-04-26T01:53:06.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:06.539+0000 7f268c200700 1 -- 172.21.15.52:0/14899888 wait complete. 2024-04-26T01:53:06.539 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:53:07.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 205083 3.06 MB/sec execute 266 sec latency 23.548 ms 2024-04-26T01:53:07.331 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:07 smithi052 ceph-mon[27851]: pgmap v249: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T01:53:07.332 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:07 smithi052 ceph-mon[27851]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:53:07.332 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:07 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/14899888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:07.405 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 82.9 --await-for 42.0' 2024-04-26T01:53:07.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 82.9 --await-for 42.0 2024-04-26T01:53:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:07 smithi191 ceph-mon[32745]: pgmap v249: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T01:53:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:07 smithi191 ceph-mon[32745]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:53:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:07 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/14899888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:07 smithi183 ceph-mon[32752]: pgmap v249: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T01:53:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:07 smithi183 ceph-mon[32752]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T01:53:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:07 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/14899888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:07.947 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:08.108 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206092 3.08 MB/sec execute 267 sec latency 67.582 ms 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.817+0000 7f0e9f400640 1 -- 172.21.15.52:0/3812445451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e981043a0 msgr2=0x7f0e98104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.817+0000 7f0e9f400640 1 --2- 172.21.15.52:0/3812445451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e981043a0 0x7f0e98104780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009860 tx=0x7f0e8c02a340 comp rx=0 tx=0).stop 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 -- 172.21.15.52:0/3812445451 shutdown_connections 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 --2- 172.21.15.52:0/3812445451 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e98109470 0x7f0e98111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 --2- 172.21.15.52:0/3812445451 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98104d50 0x7f0e98108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 --2- 172.21.15.52:0/3812445451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e981043a0 0x7f0e98104780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 -- 172.21.15.52:0/3812445451 >> 172.21.15.52:0/3812445451 conn(0x7f0e980ffc40 msgr2=0x7f0e98102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 -- 172.21.15.52:0/3812445451 shutdown_connections 2024-04-26T01:53:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.818+0000 7f0e9f400640 1 -- 172.21.15.52:0/3812445451 wait complete. 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 Processor -- start 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 -- start start 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e9819a6f0 0x7f0e9819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981143b0 con 0x7f0e98109470 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98114230 con 0x7f0e981043a0 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9f400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98114530 con 0x7f0e9819a6f0 2024-04-26T01:53:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.819+0000 7f0e9ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59488/0 (socket says 172.21.15.52:59488) 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55866/0 (socket says 172.21.15.52:55866) 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e97e00640 1 -- 172.21.15.52:0/2836084392 learned_addr learned my addr 172.21.15.52:0/2836084392 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9d400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e9819a6f0 0x7f0e9819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 -- 172.21.15.52:0/2836084392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e9819a6f0 msgr2=0x7f0e9819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e9819a6f0 0x7f0e9819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 -- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 msgr2=0x7f0e981a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 -- 172.21.15.52:0/2836084392 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9819b2f0 con 0x7f0e981043a0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e97e00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e9ca00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c035040 tx=0x7f0e8c032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.820+0000 7f0e95600640 1 -- 172.21.15.52:0/2836084392 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c037040 con 0x7f0e981043a0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.821+0000 7f0e95600640 1 -- 172.21.15.52:0/2836084392 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8c004050 con 0x7f0e981043a0 2024-04-26T01:53:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.821+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9819b580 con 0x7f0e981043a0 2024-04-26T01:53:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.821+0000 7f0e95600640 1 -- 172.21.15.52:0/2836084392 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0035f0 con 0x7f0e981043a0 2024-04-26T01:53:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.821+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e981a7660 con 0x7f0e981043a0 2024-04-26T01:53:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.824+0000 7f0e95600640 1 -- 172.21.15.52:0/2836084392 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0e8c043020 con 0x7f0e981043a0 2024-04-26T01:53:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.824+0000 7f0ea1dfd740 1 Processor -- start 2024-04-26T01:53:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.824+0000 7f0ea1dfd740 1 -- start start 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55faf510f900 con 0x55faf51aa140 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55faf50b5ef0 con 0x55faf51b4a00 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55faf510b8c0 con 0x55faf51a9820 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9ca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52492/0 (socket says 172.21.15.52:52492) 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 -- 172.21.15.52:0/2340223512 learned_addr learned my addr 172.21.15.52:0/2340223512 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:08.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e97e00640 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 -- 172.21.15.52:0/2340223512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 msgr2=0x55faf51b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 -- 172.21.15.52:0/2340223512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 msgr2=0x55faf51b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.825+0000 7f0e9d400640 1 -- 172.21.15.52:0/2340223512 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faf4e22210 con 0x55faf51a9820 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e95600640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 0x7f0e800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e9d400640 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009f50 tx=0x7f0e900118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d889ef6b2d66a62e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e95600640 1 -- 172.21.15.52:0/2836084392 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f0e8c0ef850 con 0x7f0e981043a0 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e9ca00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 0x7f0e800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e89c00640 1 -- 172.21.15.52:0/2340223512 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90004d20 con 0x55faf51a9820 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e89c00640 1 -- 172.21.15.52:0/2340223512 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e90004050 con 0x55faf51a9820 2024-04-26T01:53:08.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.826+0000 7f0e89c00640 1 -- 172.21.15.52:0/2340223512 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90005280 con 0x55faf51a9820 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/2340223512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 msgr2=0x55faf51a9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009f50 tx=0x7f0e900118d0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/2340223512 shutdown_connections 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/2340223512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51a9c00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/2340223512 >> 172.21.15.52:0/2340223512 conn(0x55faf51a5d30 msgr2=0x55faf51a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/2340223512 shutdown_connections 2024-04-26T01:53:08.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/2340223512 wait complete. 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.827+0000 7f0e9ca00640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 0x7f0e800a6300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55faf51aadc0 tx=0x7f0e8c0f5530 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 -- start start 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 Processor -- start 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51a9820 0x55faf51b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 0x55faf51b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55faf51b7630 con 0x55faf51aa140 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55faf50c9310 con 0x55faf51a9820 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55faf51b77b0 con 0x55faf51b4a00 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.828+0000 7f0ea1dfd740 10 client.0 _ll_register_callbacks cb 0x55faf5097cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59498/0 (socket says 172.21.15.52:59498) 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e97e00640 1 -- 172.21.15.52:0/3277782128 learned_addr learned my addr 172.21.15.52:0/3277782128 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51a9820 0x55faf51b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 0x55faf51b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 -- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51a9820 msgr2=0x55faf51b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51a9820 0x55faf51b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 -- 172.21.15.52:0/3277782128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 msgr2=0x55faf51b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 -- 172.21.15.52:0/3277782128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faf4eee620 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e9d400640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 0x55faf51b2420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009250 tx=0x7f0e90000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2746fe4eaea859c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e7e000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.829+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9002d060 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f0e90004c40 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e90004050 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0e90003d40 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 10 client.26083 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55faf51c38b0 con 0x55faf51b4a00 2024-04-26T01:53:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e90004050 con 0x55faf51b4a00 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f0e9000f480 con 0x55faf51b4a00 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 10 client.26083 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 10 client.26083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 20 client.26083 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf51c4d10 0x55faf51c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 4 client.26083 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.830+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faf51238b0 con 0x55faf51c4d10 2024-04-26T01:53:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.831+0000 7f0e97e00640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf51c4d10 0x55faf51c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.831+0000 7f0e97e00640 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf51c4d10 0x55faf51c7100 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed68a11b2ab32876 server_cookie=11c8d8986ae4f97a in_seq=0 out_seq=0 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.831+0000 7f0e7e000640 10 client.26083 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.832+0000 7f0e7e000640 1 -- 172.21.15.52:0/3277782128 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55faf51238b0 con 0x55faf51c4d10 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.832+0000 7f0e7e000640 10 client.26083 handle_command_reply: tid=0 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.832+0000 7f0ea1dfd740 1 client.26083 shutdown 2024-04-26T01:53:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.832+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 msgr2=0x55faf51b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.832+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 0x55faf51b2420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009250 tx=0x7f0e90000f40 comp rx=0 tx=0).stop 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 shutdown_connections 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf51c4d10 0x55faf51c7100 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51b4a00 0x55faf51b2420 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51aa140 0x55faf51b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/3277782128 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51a9820 0x55faf51b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 >> 172.21.15.52:0/3277782128 conn(0x55faf51a5d30 msgr2=0x55faf51abab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0e7e000640 0 client.26083 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 shutdown_connections 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/3277782128 wait complete. 2024-04-26T01:53:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.833+0000 7f0ea1dfd740 20 client.26083 trim_cache size 0 max 16384 2024-04-26T01:53:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 Processor -- start 2024-04-26T01:53:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 -- start start 2024-04-26T01:53:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 0x55faf51a5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981b26c0 con 0x55faf51a9820 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981aff90 con 0x55faf51b4a00 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981ade00 con 0x55faf51aa140 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59506/0 (socket says 172.21.15.52:59506) 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0e9d400640 1 -- 172.21.15.52:0/765596016 learned_addr learned my addr 172.21.15.52:0/765596016 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.850+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 0x55faf51a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e9d400640 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 -- 172.21.15.52:0/765596016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 msgr2=0x55faf51cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 -- 172.21.15.52:0/765596016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 msgr2=0x55faf51c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 -- 172.21.15.52:0/765596016 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faf51aaad0 con 0x55faf51aa140 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e9d400640 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e97e00640 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 0x55faf51a5610 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55faf51c7760 tx=0x7f0e84009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb228512c22c0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.851+0000 7f0e89200640 1 -- 172.21.15.52:0/765596016 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84005020 con 0x55faf51aa140 2024-04-26T01:53:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0e89200640 1 -- 172.21.15.52:0/765596016 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e840047a0 con 0x55faf51aa140 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0e89200640 1 -- 172.21.15.52:0/765596016 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84003850 con 0x55faf51aa140 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/765596016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 msgr2=0x55faf51a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 0x55faf51a5610 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55faf51c7760 tx=0x7f0e84009260 comp rx=0 tx=0).stop 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/765596016 shutdown_connections 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51b4a00 0x55faf51cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51aa140 0x55faf51a5610 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/765596016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51a9820 0x55faf51c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.852+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/765596016 >> 172.21.15.52:0/765596016 conn(0x55faf51a5d30 msgr2=0x55faf51b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.853+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/765596016 shutdown_connections 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.853+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/765596016 wait complete. 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.853+0000 7f0ea1dfd740 1 -- start start 2024-04-26T01:53:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.853+0000 7f0ea1dfd740 1 Processor -- start 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51b4a00 0x55faf51bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981ae450 con 0x55faf51b4a00 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981af8c0 con 0x55faf51aa140 2024-04-26T01:53:08.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0e981aeb20 con 0x55faf51a9820 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0ea1dfd740 10 client.0 _ll_register_callbacks cb 0x7f0e90003c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55900/0 (socket says 172.21.15.52:55900) 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 -- 172.21.15.52:0/1056795651 learned_addr learned my addr 172.21.15.52:0/1056795651 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e9d400640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51b4a00 0x55faf51bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 -- 172.21.15.52:0/1056795651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 msgr2=0x55faf51ba670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51ba670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 -- 172.21.15.52:0/1056795651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51b4a00 msgr2=0x55faf51bb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51b4a00 0x55faf51bb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.854+0000 7f0e97e00640 1 -- 172.21.15.52:0/1056795651 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faf522a2f0 con 0x55faf51aa140 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0e97e00640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e84010740 tx=0x7f0e84009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18f7e22e10660b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0e7e000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84004da0 con 0x55faf51aa140 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f0e84005020 con 0x55faf51aa140 2024-04-26T01:53:08.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e840047a0 con 0x55faf51aa140 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0e84005220 con 0x55faf51aa140 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0ea1dfd740 10 client.26084 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.855+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55faf522b5e0 con 0x55faf51aa140 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e840047a0 con 0x55faf51aa140 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f0e8401e050 con 0x55faf51aa140 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 10 client.26084 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 10 client.26084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 20 client.26084 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf522bc40 0x55faf522e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 4 client.26084 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55faf511d320 con 0x55faf522bc40 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.856+0000 7f0e9d400640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf522bc40 0x55faf522e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.857+0000 7f0e7e000640 10 client.26084 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:08.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.857+0000 7f0e9d400640 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf522bc40 0x55faf522e030 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b015cccb0e1bd57d server_cookie=87958d49388988b8 in_seq=0 out_seq=0 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0e7e000640 1 -- 172.21.15.52:0/1056795651 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55faf511d320 con 0x55faf522bc40 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0e7e000640 10 client.26084 handle_command_reply: tid=0 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 client.26084 shutdown 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 msgr2=0x55faf51babb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e84010740 tx=0x7f0e84009f60 comp rx=0 tx=0).stop 2024-04-26T01:53:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 shutdown_connections 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55faf522bc40 0x55faf522e030 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55faf51b4a00 0x55faf51bb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55faf51aa140 0x55faf51babb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 --2- 172.21.15.52:0/1056795651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55faf51a9820 0x55faf51ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0e7e000640 0 client.26084 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 >> 172.21.15.52:0/1056795651 conn(0x55faf51a5d30 msgr2=0x55faf51b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 shutdown_connections 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 1 -- 172.21.15.52:0/1056795651 wait complete. 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.897+0000 7f0ea1dfd740 20 client.26084 trim_cache size 0 max 16384 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:53:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:53:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:53:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:53:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 msgr2=0x7f0e800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 0x7f0e800a6300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55faf51aadc0 tx=0x7f0e8c0f5530 comp rx=0 tx=0).stop 2024-04-26T01:53:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 msgr2=0x7f0e981a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c035040 tx=0x7f0e8c032d50 comp rx=0 tx=0).stop 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 shutdown_connections 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0e800a3e40 0x7f0e800a6300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e9819a6f0 0x7f0e9819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e98109470 0x7f0e981a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 --2- 172.21.15.52:0/2836084392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e981043a0 0x7f0e981a05e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 >> 172.21.15.52:0/2836084392 conn(0x7f0e980ffc40 msgr2=0x7f0e98101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 shutdown_connections 2024-04-26T01:53:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:08.898+0000 7f0e9f400640 1 -- 172.21.15.52:0/2836084392 wait complete. 2024-04-26T01:53:09.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206092 3.07 MB/sec execute 268 sec latency 1067.714 ms 2024-04-26T01:53:09.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:09 smithi052 ceph-mon[27851]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:09 smithi191 ceph-mon[32745]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:09.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:09 smithi183 ceph-mon[32752]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:10.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206092 3.06 MB/sec execute 269 sec latency 2067.862 ms 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 -- 172.21.15.52:0/3362405933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c4109c80 msgr2=0x7f34c410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3362405933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c4109c80 0x7f34c410dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f34b00099c0 tx=0x7f34b0009cd0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 -- 172.21.15.52:0/3362405933 shutdown_connections 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3362405933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c410e3b0 0x7f34c4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3362405933 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 0x7f34c41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3362405933 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c4109c80 0x7f34c410dcd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 -- 172.21.15.52:0/3362405933 >> 172.21.15.52:0/3362405933 conn(0x7f34c4076210 msgr2=0x7f34c4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 -- 172.21.15.52:0/3362405933 shutdown_connections 2024-04-26T01:53:10.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.590+0000 7f34c8a00700 1 -- 172.21.15.52:0/3362405933 wait complete. 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.591+0000 7f34c8a00700 1 Processor -- start 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.591+0000 7f34c8a00700 1 -- start start 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 0x7f34c419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 0x7f34c419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 0x7f34c41a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34c4119240 con 0x7f34c410e3b0 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41190c0 con 0x7f34c41092d0 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c8a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41193c0 con 0x7f34c4109c80 2024-04-26T01:53:10.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 0x7f34c41a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c1400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 0x7f34c419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 0x7f34c41a3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59536/0 (socket says 172.21.15.52:59536) 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.592+0000 7f34c2800700 1 -- 172.21.15.52:0/3863114963 learned_addr learned my addr 172.21.15.52:0/3863114963 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1e00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 0x7f34c419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 -- 172.21.15.52:0/3863114963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 msgr2=0x7f34c419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 0x7f34c419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 -- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 msgr2=0x7f34c41a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 0x7f34c41a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 -- 172.21.15.52:0/3863114963 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c41a4640 con 0x7f34c4109c80 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.593+0000 7f34c1400700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 0x7f34c419fb70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b580 tx=0x7f34b00055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.594+0000 7f34baa00700 1 -- 172.21.15.52:0/3863114963 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0026070 con 0x7f34c4109c80 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.594+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c41a48d0 con 0x7f34c4109c80 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.594+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c41ac1b0 con 0x7f34c4109c80 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.594+0000 7f34baa00700 1 -- 172.21.15.52:0/3863114963 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b000b790 con 0x7f34c4109c80 2024-04-26T01:53:10.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.594+0000 7f34baa00700 1 -- 172.21.15.52:0/3863114963 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0030060 con 0x7f34c4109c80 2024-04-26T01:53:10.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.599+0000 7f34baa00700 1 -- 172.21.15.52:0/3863114963 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f34b0029070 con 0x7f34c4109c80 2024-04-26T01:53:10.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.600+0000 7f34d3360b80 1 Processor -- start 2024-04-26T01:53:10.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.600+0000 7f34d3360b80 1 -- start start 2024-04-26T01:53:10.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.600+0000 7f34baa00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 0x7f34a80a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.600+0000 7f34baa00700 1 -- 172.21.15.52:0/3863114963 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f34b00d0ef0 con 0x7f34c4109c80 2024-04-26T01:53:10.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cd634f6020 con 0x55cd635a36f0 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cd6349c8c0 con 0x55cd635a2dd0 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34d3360b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cd634f1fe0 con 0x55cd635ae560 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c2800700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 0x7f34a80a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c1400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55936/0 (socket says 172.21.15.52:55936) 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c1e00700 1 -- 172.21.15.52:0/2767105477 learned_addr learned my addr 172.21.15.52:0/2767105477 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:10.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.601+0000 7f34c2800700 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 -- 172.21.15.52:0/2767105477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 msgr2=0x55cd635b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c2800700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 0x7f34a80a6310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009c10 tx=0x7f34bc009920 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 -- 172.21.15.52:0/2767105477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 msgr2=0x55cd635a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 -- 172.21.15.52:0/2767105477 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd63492810 con 0x55cd635a36f0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1e00700 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34c1400700 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635ae020 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f34b00ce5b0 tx=0x7f34b009ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a971d6043b79ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34af400700 1 -- 172.21.15.52:0/2767105477 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b009ce90 con 0x55cd635a36f0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34af400700 1 -- 172.21.15.52:0/2767105477 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00a1240 con 0x55cd635a36f0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.602+0000 7f34af400700 1 -- 172.21.15.52:0/2767105477 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00a5840 con 0x55cd635a36f0 2024-04-26T01:53:10.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 -- 172.21.15.52:0/2767105477 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 msgr2=0x55cd635ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635ae020 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f34b00ce5b0 tx=0x7f34b009ca50 comp rx=0 tx=0).stop 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 -- 172.21.15.52:0/2767105477 shutdown_connections 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2767105477 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635ae020 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.603+0000 7f34d3360b80 1 -- 172.21.15.52:0/2767105477 >> 172.21.15.52:0/2767105477 conn(0x55cd634d20e0 msgr2=0x55cd6359f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.604+0000 7f34d3360b80 1 -- 172.21.15.52:0/2767105477 shutdown_connections 2024-04-26T01:53:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.604+0000 7f34d3360b80 1 -- 172.21.15.52:0/2767105477 wait complete. 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.604+0000 7f34d3360b80 1 -- start start 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.604+0000 7f34d3360b80 1 Processor -- start 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a2dd0 0x55cd635adc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a36f0 0x55cd635aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cd635b0e90 con 0x55cd635ae560 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cd634af9f0 con 0x55cd635a36f0 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cd635b1010 con 0x55cd635a2dd0 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34d3360b80 10 client.0 _ll_register_callbacks cb 0x55cd6348e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59540/0 (socket says 172.21.15.52:59540) 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34c2800700 1 -- 172.21.15.52:0/2591988824 learned_addr learned my addr 172.21.15.52:0/2591988824 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34c1400700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a36f0 0x55cd635aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.605+0000 7f34c1e00700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a2dd0 0x55cd635adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 -- 172.21.15.52:0/2591988824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a2dd0 msgr2=0x55cd635adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a2dd0 0x55cd635adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 -- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a36f0 msgr2=0x55cd635aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a36f0 0x55cd635aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 -- 172.21.15.52:0/2591988824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd63443a60 con 0x55cd635ae560 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34c2800700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f34bc019e10 tx=0x7f34bc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6b4744a87c5cb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.606+0000 7f34a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc026070 con 0x55cd635ae560 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f34bc004db0 con 0x55cd635ae560 2024-04-26T01:53:10.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b00a1240 con 0x55cd635ae560 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f34bc031050 con 0x55cd635ae560 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34d3360b80 10 client.15978 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cd6348cc80 con 0x55cd635ae560 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.607+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00a1240 con 0x55cd635ae560 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f34bc0041f0 con 0x55cd635ae560 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 10 client.15978 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 10 client.15978 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 20 client.15978 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd635bdae0 0x55cd635bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 4 client.15978 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd63503810 con 0x55cd635bdae0 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.608+0000 7f34c1e00700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd635bdae0 0x55cd635bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.609+0000 7f34c1e00700 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd635bdae0 0x55cd635bfed0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b43d7b298fd505d server_cookie=6e6f0d4ebfeabdb6 in_seq=0 out_seq=0 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.609+0000 7f34a3e00700 10 client.15978 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34a3e00700 1 -- 172.21.15.52:0/2591988824 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd63503810 con 0x55cd635bdae0 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34a3e00700 10 client.15978 handle_command_reply: tid=0 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 client.15978 shutdown 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 msgr2=0x55cd635ab530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f34bc019e10 tx=0x7f34bc004b40 comp rx=0 tx=0).stop 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 shutdown_connections 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd635bdae0 0x55cd635bfed0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635ab530 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34a3e00700 0 client.15978 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a2dd0 0x55cd635adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 --2- 172.21.15.52:0/2591988824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a36f0 0x55cd635aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 >> 172.21.15.52:0/2591988824 conn(0x55cd634d20e0 msgr2=0x55cd6359eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 shutdown_connections 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.610+0000 7f34d3360b80 1 -- 172.21.15.52:0/2591988824 wait complete. 2024-04-26T01:53:10.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.611+0000 7f34d3360b80 20 client.15978 trim_cache size 0 max 16384 2024-04-26T01:53:10.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 Processor -- start 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 -- start start 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 0x55cd635ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400fee0 con 0x55cd635ae560 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400d770 con 0x55cd635a2dd0 2024-04-26T01:53:10.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.632+0000 7f34d3360b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400b5e0 con 0x55cd635a36f0 2024-04-26T01:53:10.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59546/0 (socket says 172.21.15.52:59546) 2024-04-26T01:53:10.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c2800700 1 -- 172.21.15.52:0/937004334 learned_addr learned my addr 172.21.15.52:0/937004334 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:10.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 0x55cd635ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 -- 172.21.15.52:0/937004334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 msgr2=0x55cd635b9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 -- 172.21.15.52:0/937004334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 msgr2=0x55cd635a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c1400700 1 -- 172.21.15.52:0/937004334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd635c0530 con 0x55cd635a36f0 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.633+0000 7f34c2800700 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.634+0000 7f34c1400700 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 0x55cd635ad2e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f34b00ce5e0 tx=0x7f34b0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a04aef412e1a1d66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.634+0000 7f34aea00700 1 -- 172.21.15.52:0/937004334 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00a5a70 con 0x55cd635a36f0 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.634+0000 7f34aea00700 1 -- 172.21.15.52:0/937004334 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00a1240 con 0x55cd635a36f0 2024-04-26T01:53:10.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.634+0000 7f34aea00700 1 -- 172.21.15.52:0/937004334 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00a3450 con 0x55cd635a36f0 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- 172.21.15.52:0/937004334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 msgr2=0x55cd635ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 0x55cd635ad2e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f34b00ce5e0 tx=0x7f34b0009310 comp rx=0 tx=0).stop 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- 172.21.15.52:0/937004334 shutdown_connections 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635ae560 0x55cd635a82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 --2- 172.21.15.52:0/937004334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635a36f0 0x55cd635ad2e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- 172.21.15.52:0/937004334 >> 172.21.15.52:0/937004334 conn(0x55cd634d20e0 msgr2=0x55cd635acbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:10.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- 172.21.15.52:0/937004334 shutdown_connections 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- 172.21.15.52:0/937004334 wait complete. 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.635+0000 7f34d3360b80 1 -- start start 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 Processor -- start 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400bc30 con 0x55cd635a36f0 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400d0a0 con 0x55cd635a2dd0 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f34a400c300 con 0x55cd635ae560 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34d3360b80 10 client.0 _ll_register_callbacks cb 0x55cd6348e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.636+0000 7f34c1400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55974/0 (socket says 172.21.15.52:55974) 2024-04-26T01:53:10.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 -- 172.21.15.52:0/1305605411 learned_addr learned my addr 172.21.15.52:0/1305605411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c2800700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 -- 172.21.15.52:0/1305605411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 msgr2=0x55cd635b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 -- 172.21.15.52:0/1305605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 msgr2=0x55cd635b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:10.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 -- 172.21.15.52:0/1305605411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd6359fdb0 con 0x55cd635a2dd0 2024-04-26T01:53:10.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34c1e00700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55cd635c0700 tx=0x7f34b4012a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36434a429e7439ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:10.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.637+0000 7f34a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:10.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4020580 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8565+0+0 (secure 0 0 0) 0x7f34b4020720 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b00a1240 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f34b4030050 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34d3360b80 10 client.26102 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cd63625ad0 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.638+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34b00a1240 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f34b4010430 con 0x55cd635a2dd0 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 10 client.26102 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 10 client.26102 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 20 client.26102 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd63626200 0x55cd63628620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 4 client.26102 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await_for": 42.0}] 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await_for": 42.0}) -- 0x55cd634fd8b0 con 0x55cd63626200 2024-04-26T01:53:10.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34c1400700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd63626200 0x55cd63628620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:10.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34c1400700 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd63626200 0x55cd63628620 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d502a9b5efa09b98 server_cookie=b16d77864dd12391 in_seq=0 out_seq=0 2024-04-26T01:53:10.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:10.639+0000 7f34a3e00700 10 client.26102 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:11.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206093 3.04 MB/sec execute 270 sec latency 2644.668 ms 2024-04-26T01:53:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:11 smithi191 ceph-mon[32745]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:11.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:11 smithi183 ceph-mon[32752]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:11.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:11 smithi052 ceph-mon[27851]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-26T01:53:12.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206093 3.03 MB/sec execute 271 sec latency 1423.522 ms 2024-04-26T01:53:13.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.105+0000 7f34a3e00700 1 -- 172.21.15.52:0/1305605411 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55cd634fd8b0 con 0x55cd63626200 2024-04-26T01:53:13.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.105+0000 7f34a3e00700 10 client.26102 handle_command_reply: tid=0 2024-04-26T01:53:13.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.105+0000 7f34d3360b80 1 client.26102 shutdown 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 msgr2=0x55cd635b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55cd635c0700 tx=0x7f34b4012a70 comp rx=0 tx=0).stop 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 shutdown_connections 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd63626200 0x55cd63628620 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd635ae560 0x55cd635b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd635a2dd0 0x55cd635b61e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 --2- 172.21.15.52:0/1305605411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd635a36f0 0x55cd635b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 >> 172.21.15.52:0/1305605411 conn(0x55cd634d20e0 msgr2=0x55cd635ad2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:13.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34a3e00700 0 client.26102 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:13.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 shutdown_connections 2024-04-26T01:53:13.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.106+0000 7f34d3360b80 1 -- 172.21.15.52:0/1305605411 wait complete. 2024-04-26T01:53:13.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.107+0000 7f34d3360b80 20 client.26102 trim_cache size 0 max 16384 2024-04-26T01:53:13.106 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:53:13.106 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 22, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "af4e799c": { 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 22, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.9, 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:53:13.107 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:13.108 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.107+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 msgr2=0x7f34a80a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.107+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 0x7f34a80a6310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009c10 tx=0x7f34bc009920 comp rx=0 tx=0).stop 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.107+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 msgr2=0x7f34c419fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.107+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 0x7f34c419fb70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b580 tx=0x7f34b00055f0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 shutdown_connections 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34a80a3e50 0x7f34a80a6310 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34c410e3b0 0x7f34c41a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34c41092d0 0x7f34c419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 --2- 172.21.15.52:0/3863114963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34c4109c80 0x7f34c419fb70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:13.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 >> 172.21.15.52:0/3863114963 conn(0x7f34c4076210 msgr2=0x7f34c40766f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:13.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 shutdown_connections 2024-04-26T01:53:13.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:13.108+0000 7f34c8a00700 1 -- 172.21.15.52:0/3863114963 wait complete. 2024-04-26T01:53:13.110 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 3.02 MB/sec execute 272 sec latency 2401.309 ms 2024-04-26T01:53:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:13 smithi052 ceph-mon[27851]: pgmap v253: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T01:53:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:13 smithi052 ceph-mon[27851]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:53:13.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:13 smithi183 ceph-mon[32752]: pgmap v253: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T01:53:13.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:13 smithi183 ceph-mon[32752]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:53:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:13 smithi191 ceph-mon[32745]: pgmap v253: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 180 op/s 2024-04-26T01:53:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:13 smithi191 ceph-mon[32745]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T01:53:14.038 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'af4e799c', quiesce took 11.4 seconds. Will release after: 22.9 2024-04-26T01:53:14.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 3.01 MB/sec execute 273 sec latency 1022.090 ms 2024-04-26T01:53:15.109 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 3.00 MB/sec execute 274 sec latency 2022.238 ms 2024-04-26T01:53:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:15 smithi191 ceph-mon[32745]: pgmap v255: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 916 KiB/s wr, 103 op/s 2024-04-26T01:53:15.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:15 smithi183 ceph-mon[32752]: pgmap v255: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 916 KiB/s wr, 103 op/s 2024-04-26T01:53:15.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:15 smithi052 ceph-mon[27851]: pgmap v255: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 916 KiB/s wr, 103 op/s 2024-04-26T01:53:16.110 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.99 MB/sec execute 275 sec latency 3022.388 ms 2024-04-26T01:53:17.110 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.98 MB/sec execute 276 sec latency 4022.549 ms 2024-04-26T01:53:17.515 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:17 smithi052 ceph-mon[27851]: pgmap v256: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2024-04-26T01:53:17.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:17 smithi191 ceph-mon[32745]: pgmap v256: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2024-04-26T01:53:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:17 smithi183 ceph-mon[32752]: pgmap v256: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 1.3 MiB/s wr, 105 op/s 2024-04-26T01:53:18.110 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.97 MB/sec execute 277 sec latency 5022.642 ms 2024-04-26T01:53:19.110 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.96 MB/sec execute 278 sec latency 6022.766 ms 2024-04-26T01:53:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:19 smithi191 ceph-mon[32745]: pgmap v257: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:19.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:19 smithi183 ceph-mon[32752]: pgmap v257: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:19 smithi052 ceph-mon[27851]: pgmap v257: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:20.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.95 MB/sec execute 279 sec latency 7023.249 ms 2024-04-26T01:53:21.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.94 MB/sec execute 280 sec latency 8023.413 ms 2024-04-26T01:53:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:21 smithi191 ceph-mon[32745]: pgmap v258: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:21 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:21 smithi183 ceph-mon[32752]: pgmap v258: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:21.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:21 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:21 smithi052 ceph-mon[27851]: pgmap v258: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.2 MiB/s wr, 104 op/s 2024-04-26T01:53:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:21 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 -- 172.21.15.52:0/3906951345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc10e840 msgr2=0x7f84bc110c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 --2- 172.21.15.52:0/3906951345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc10e840 0x7f84bc110c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009ca0 tx=0x7f84b402a950 comp rx=0 tx=0).stop 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 -- 172.21.15.52:0/3906951345 shutdown_connections 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 --2- 172.21.15.52:0/3906951345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc10e840 0x7f84bc110c30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 --2- 172.21.15.52:0/3906951345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074b40 0x7f84bc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 --2- 172.21.15.52:0/3906951345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc074220 0x7f84bc074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 -- 172.21.15.52:0/3906951345 >> 172.21.15.52:0/3906951345 conn(0x7f84bc0793e0 msgr2=0x7f84bc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 -- 172.21.15.52:0/3906951345 shutdown_connections 2024-04-26T01:53:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.039+0000 7f84c2400640 1 -- 172.21.15.52:0/3906951345 wait complete. 2024-04-26T01:53:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.040+0000 7f84c2400640 1 Processor -- start 2024-04-26T01:53:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.040+0000 7f84c2400640 1 -- start start 2024-04-26T01:53:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074220 0x7f84bc1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc074b40 0x7f84bc1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc114400 con 0x7f84bc074220 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc114280 con 0x7f84bc074b40 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84c2400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc114580 con 0x7f84bc10e840 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84bbe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84bb400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074220 0x7f84bc1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84bbe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34960/0 (socket says 172.21.15.52:34960) 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84bbe00640 1 -- 172.21.15.52:0/1609234628 learned_addr learned my addr 172.21.15.52:0/1609234628 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.041+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc074b40 0x7f84bc1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 -- 172.21.15.52:0/1609234628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc074b40 msgr2=0x7f84bc1a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc074b40 0x7f84bc1a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 -- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074220 msgr2=0x7f84bc1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074220 0x7f84bc1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 -- 172.21.15.52:0/1609234628 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc19afc0 con 0x7f84bc10e840 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84bbe00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f84b4005e30 tx=0x7f84b4003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84b8c00640 1 -- 172.21.15.52:0/1609234628 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4034040 con 0x7f84bc10e840 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84b8c00640 1 -- 172.21.15.52:0/1609234628 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84b402f870 con 0x7f84bc10e840 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84b8c00640 1 -- 172.21.15.52:0/1609234628 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4004580 con 0x7f84bc10e840 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84bc19b250 con 0x7f84bc10e840 2024-04-26T01:53:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.042+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84bc1a7390 con 0x7f84bc10e840 2024-04-26T01:53:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.047+0000 7f84c53ec740 1 Processor -- start 2024-04-26T01:53:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.047+0000 7f84b8c00640 1 -- 172.21.15.52:0/1609234628 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f84b4037020 con 0x7f84bc10e840 2024-04-26T01:53:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.047+0000 7f84c53ec740 1 -- start start 2024-04-26T01:53:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a237c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a244df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a44a19d900 con 0x55a44a238140 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a44a143ef0 con 0x55a44a242a00 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84c53ec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a44a1998c0 con 0x55a44a237820 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84bb400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84bb400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50680/0 (socket says 172.21.15.52:50680) 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84bb400640 1 -- 172.21.15.52:0/1513910343 learned_addr learned my addr 172.21.15.52:0/1513910343 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a244df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a237c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84bb400640 1 -- 172.21.15.52:0/1513910343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 msgr2=0x55a44a237c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.048+0000 7f84bb400640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a237c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84bb400640 1 -- 172.21.15.52:0/1513910343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 msgr2=0x55a44a244df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84bb400640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a244df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84bb400640 1 -- 172.21.15.52:0/1513910343 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a449eb0210 con 0x55a44a238140 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a237c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84bb400640 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009cf0 tx=0x7f84ac011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207f4c9b89005c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b8c00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 0x7f84a00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b0c00640 1 -- 172.21.15.52:0/1513910343 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac01e070 con 0x55a44a238140 2024-04-26T01:53:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b0c00640 1 -- 172.21.15.52:0/1513910343 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ac019ae0 con 0x55a44a238140 2024-04-26T01:53:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b0c00640 1 -- 172.21.15.52:0/1513910343 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00e970 con 0x55a44a238140 2024-04-26T01:53:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b8c00640 1 -- 172.21.15.52:0/1609234628 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f84b40f13c0 con 0x7f84bc10e840 2024-04-26T01:53:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.049+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 0x7f84a00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 -- 172.21.15.52:0/1513910343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 msgr2=0x55a44a2424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009cf0 tx=0x7f84ac011810 comp rx=0 tx=0).stop 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84b3e00640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 0x7f84a00a6300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55a44a235150 tx=0x7f84a40046c0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 -- 172.21.15.52:0/1513910343 shutdown_connections 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a244df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a2424c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 --2- 172.21.15.52:0/1513910343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a237c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 -- 172.21.15.52:0/1513910343 >> 172.21.15.52:0/1513910343 conn(0x55a44a233d30 msgr2=0x55a44a234110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 -- 172.21.15.52:0/1513910343 shutdown_connections 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.050+0000 7f84c53ec740 1 -- 172.21.15.52:0/1513910343 wait complete. 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.051+0000 7f84c53ec740 1 -- start start 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.051+0000 7f84c53ec740 1 Processor -- start 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.051+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a23f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.051+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a23fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84c53ec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a44a245630 con 0x55a44a238140 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84c53ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a44a157310 con 0x55a44a242a00 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84c53ec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a44a2457b0 con 0x55a44a237820 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84c53ec740 10 client.0 _ll_register_callbacks cb 0x55a44a125cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84b3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a23fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a23f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:55390/0 (socket says 172.21.15.52:55390) 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 -- 172.21.15.52:0/4167954483 learned_addr learned my addr 172.21.15.52:0/4167954483 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 -- 172.21.15.52:0/4167954483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 msgr2=0x55a44a23f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a23f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 -- 172.21.15.52:0/4167954483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 msgr2=0x55a44a23fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a23fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 -- 172.21.15.52:0/4167954483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a449f7c620 con 0x55a44a242a00 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84bbe00640 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84b402af10 tx=0x7f84b40bdbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e8436dbba4b3305 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.052+0000 7f84a9600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40d6060 con 0x55a44a242a00 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f84b40eed30 con 0x55a44a242a00 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84ac019ae0 con 0x55a44a242a00 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f84b40c6080 con 0x55a44a242a00 2024-04-26T01:53:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 10 client.26114 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a44a245630 con 0x55a44a242a00 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ac019ae0 con 0x55a44a242a00 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f84b40bd670 con 0x55a44a242a00 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 10 client.26114 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 20 client.26114 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a252c50 0x55a44a2550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 4 client.26114 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.053+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a44a1b18b0 con 0x55a44a252c50 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.054+0000 7f84bb400640 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a252c50 0x55a44a2550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.054+0000 7f84bb400640 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a252c50 0x55a44a2550b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b79b585f913d1a2 server_cookie=1c2d96277903fc79 in_seq=0 out_seq=0 2024-04-26T01:53:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.054+0000 7f84a9600640 10 client.26114 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84a9600640 1 -- 172.21.15.52:0/4167954483 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a44a1b18b0 con 0x55a44a252c50 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84a9600640 10 client.26114 handle_command_reply: tid=0 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 client.26114 shutdown 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 msgr2=0x55a44a2403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84b402af10 tx=0x7f84b40bdbd0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 shutdown_connections 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a252c50 0x55a44a2550b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a2403e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a23fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 --2- 172.21.15.52:0/4167954483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a23f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 >> 172.21.15.52:0/4167954483 conn(0x55a44a233d30 msgr2=0x55a44a239a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84a9600640 0 client.26114 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.055+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 shutdown_connections 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.056+0000 7f84c53ec740 1 -- 172.21.15.52:0/4167954483 wait complete. 2024-04-26T01:53:22.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.056+0000 7f84c53ec740 20 client.26114 trim_cache size 0 max 16384 2024-04-26T01:53:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 Processor -- start 2024-04-26T01:53:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 -- start start 2024-04-26T01:53:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a238140 0x55a44a25a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a242a00 0x55a44a25af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1b25a0 con 0x55a44a242a00 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1afe70 con 0x55a44a238140 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.075+0000 7f84c53ec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1adce0 con 0x55a44a237820 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bbe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a242a00 0x55a44a25af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34972/0 (socket says 172.21.15.52:34972) 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 -- 172.21.15.52:0/3623595860 learned_addr learned my addr 172.21.15.52:0/3623595860 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84b3e00640 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a238140 0x55a44a25a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 -- 172.21.15.52:0/3623595860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a238140 msgr2=0x55a44a25a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a238140 0x55a44a25a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 -- 172.21.15.52:0/3623595860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a242a00 msgr2=0x55a44a25af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a242a00 0x55a44a25af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 -- 172.21.15.52:0/3623595860 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a44a22f690 con 0x55a44a237820 2024-04-26T01:53:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.076+0000 7f84bb400640 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0097e0 tx=0x7f84ac005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fccd1ae3881005b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.077+0000 7f84abe00640 1 -- 172.21.15.52:0/3623595860 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac032070 con 0x55a44a237820 2024-04-26T01:53:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.077+0000 7f84abe00640 1 -- 172.21.15.52:0/3623595860 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ac0194d0 con 0x55a44a237820 2024-04-26T01:53:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.077+0000 7f84abe00640 1 -- 172.21.15.52:0/3623595860 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac01e040 con 0x55a44a237820 2024-04-26T01:53:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- 172.21.15.52:0/3623595860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 msgr2=0x55a44a24e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0097e0 tx=0x7f84ac005c50 comp rx=0 tx=0).stop 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- 172.21.15.52:0/3623595860 shutdown_connections 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a242a00 0x55a44a25af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a238140 0x55a44a25a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 --2- 172.21.15.52:0/3623595860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a24e4f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- 172.21.15.52:0/3623595860 >> 172.21.15.52:0/3623595860 conn(0x55a44a233d30 msgr2=0x55a44a234110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- 172.21.15.52:0/3623595860 shutdown_connections 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- 172.21.15.52:0/3623595860 wait complete. 2024-04-26T01:53:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.078+0000 7f84c53ec740 1 -- start start 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 Processor -- start 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a248a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a248f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1ae330 con 0x55a44a238140 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1af7a0 con 0x55a44a242a00 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1aea00 con 0x55a44a237820 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84c53ec740 10 client.0 _ll_register_callbacks cb 0x7f84b40bcfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.079+0000 7f84bb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34978/0 (socket says 172.21.15.52:34978) 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84b3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a248a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 -- 172.21.15.52:0/2074058353 learned_addr learned my addr 172.21.15.52:0/2074058353 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bbe00640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a248f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 -- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 msgr2=0x55a44a248f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a248f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 -- 172.21.15.52:0/2074058353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 msgr2=0x55a44a248a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a248a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 -- 172.21.15.52:0/2074058353 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a44a2b8310 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84bb400640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0057c0 tx=0x7f84ac0099a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7df7d79d3cfbb57c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84a9600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.080+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac01e030 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84ac0194d0 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f84ac019980 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f84ac032070 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 10 client.26122 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a44a2b9620 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ac0194d0 con 0x55a44a237820 2024-04-26T01:53:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f84ac003ba0 con 0x55a44a237820 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 10 client.26122 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 10 client.26122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 20 client.26122 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a2b9aa0 0x55a44a2bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 4 client.26122 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.081+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a44a1ab320 con 0x55a44a2b9aa0 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.082+0000 7f84b3e00640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a2b9aa0 0x55a44a2bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.082+0000 7f84b3e00640 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a2b9aa0 0x55a44a2bbe90 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99184ec03155adba server_cookie=af69c7ba55caec88 in_seq=0 out_seq=0 2024-04-26T01:53:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.082+0000 7f84a9600640 10 client.26122 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84a9600640 1 -- 172.21.15.52:0/2074058353 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a44a1ab320 con 0x55a44a2b9aa0 2024-04-26T01:53:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84a9600640 10 client.26122 handle_command_reply: tid=0 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 client.26122 shutdown 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 msgr2=0x55a44a248510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0057c0 tx=0x7f84ac0099a0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 shutdown_connections 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a44a2b9aa0 0x55a44a2bbe90 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a44a242a00 0x55a44a248f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a44a238140 0x55a44a248a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 --2- 172.21.15.52:0/2074058353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a44a237820 0x55a44a248510 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84a9600640 0 client.26122 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 >> 172.21.15.52:0/2074058353 conn(0x55a44a233d30 msgr2=0x55a44a239140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.110+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 shutdown_connections 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c53ec740 1 -- 172.21.15.52:0/2074058353 wait complete. 2024-04-26T01:53:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c53ec740 20 client.26122 trim_cache size 0 max 16384 2024-04-26T01:53:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:53:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:53:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:53:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:53:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 msgr2=0x7f84a00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 0x7f84a00a6300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55a44a235150 tx=0x7f84a40046c0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 msgr2=0x7f84bc19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.111+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f84b4005e30 tx=0x7f84b4003ba0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 shutdown_connections 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f84a00a3e40 0x7f84a00a6300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84bc10e840 0x7f84bc19a700 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84bc074b40 0x7f84bc1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 --2- 172.21.15.52:0/1609234628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84bc074220 0x7f84bc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 >> 172.21.15.52:0/1609234628 conn(0x7f84bc0793e0 msgr2=0x7f84bc0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 shutdown_connections 2024-04-26T01:53:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:22.112+0000 7f84c2400640 1 -- 172.21.15.52:0/1609234628 wait complete. 2024-04-26T01:53:22.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.93 MB/sec execute 281 sec latency 9023.522 ms 2024-04-26T01:53:23.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.92 MB/sec execute 282 sec latency 10023.633 ms 2024-04-26T01:53:23.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:23 smithi191 ceph-mon[32745]: pgmap v259: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.1 MiB/s wr, 41 op/s 2024-04-26T01:53:23.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:23 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:23.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:23 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:23.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:23 smithi183 ceph-mon[32752]: pgmap v259: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.1 MiB/s wr, 41 op/s 2024-04-26T01:53:23.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:23 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:23.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:23 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:23.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:23 smithi052 ceph-mon[27851]: pgmap v259: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.1 MiB/s wr, 41 op/s 2024-04-26T01:53:23.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:23 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:23.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:23 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:24.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.91 MB/sec execute 283 sec latency 11023.727 ms 2024-04-26T01:53:24.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:24 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:24.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:24 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:24.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:24 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:24.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:24 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:24.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:24 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:24.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:24 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:24.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:24 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:53:24.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:24 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:53:24.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:24 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:53:25.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.89 MB/sec execute 284 sec latency 12023.862 ms 2024-04-26T01:53:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:25 smithi191 ceph-mon[32745]: pgmap v260: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-26T01:53:25.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:25 smithi183 ceph-mon[32752]: pgmap v260: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-26T01:53:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:25 smithi052 ceph-mon[27851]: pgmap v260: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.0 MiB/s wr, 37 op/s 2024-04-26T01:53:26.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.88 MB/sec execute 285 sec latency 13024.017 ms 2024-04-26T01:53:27.111 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.87 MB/sec execute 286 sec latency 14024.179 ms 2024-04-26T01:53:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:27 smithi191 ceph-mon[32745]: pgmap v261: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-04-26T01:53:27.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:27 smithi183 ceph-mon[32752]: pgmap v261: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-04-26T01:53:27.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:27 smithi052 ceph-mon[27851]: pgmap v261: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 1.3 MiB/s wr, 56 op/s 2024-04-26T01:53:28.112 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.86 MB/sec execute 287 sec latency 15024.316 ms 2024-04-26T01:53:29.112 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.85 MB/sec execute 288 sec latency 16024.501 ms 2024-04-26T01:53:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:29 smithi191 ceph-mon[32745]: pgmap v262: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:29.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:29 smithi183 ceph-mon[32752]: pgmap v262: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:29.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:29 smithi052 ceph-mon[27851]: pgmap v262: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:30.112 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.84 MB/sec execute 289 sec latency 17024.659 ms 2024-04-26T01:53:31.112 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.83 MB/sec execute 290 sec latency 18024.823 ms 2024-04-26T01:53:31.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:31 smithi191 ceph-mon[32745]: pgmap v263: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:31.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:31 smithi183 ceph-mon[32752]: pgmap v263: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:31 smithi052 ceph-mon[27851]: pgmap v263: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 838 KiB/s wr, 38 op/s 2024-04-26T01:53:32.112 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.83 MB/sec execute 291 sec latency 19024.986 ms 2024-04-26T01:53:33.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.82 MB/sec execute 292 sec latency 20025.158 ms 2024-04-26T01:53:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:33 smithi191 ceph-mon[32745]: pgmap v264: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-26T01:53:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:33 smithi183 ceph-mon[32752]: pgmap v264: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-26T01:53:33.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:33 smithi052 ceph-mon[27851]: pgmap v264: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-26T01:53:34.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.81 MB/sec execute 293 sec latency 21025.321 ms 2024-04-26T01:53:35.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.80 MB/sec execute 294 sec latency 22025.489 ms 2024-04-26T01:53:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.259+0000 7f30a2e00640 1 -- 172.21.15.52:0/700213970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 msgr2=0x7f309c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.259+0000 7f30a2e00640 1 --2- 172.21.15.52:0/700213970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1089e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3084009cd0 tx=0x7f308402a8e0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 -- 172.21.15.52:0/700213970 shutdown_connections 2024-04-26T01:53:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 --2- 172.21.15.52:0/700213970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 --2- 172.21.15.52:0/700213970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1089e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 --2- 172.21.15.52:0/700213970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 -- 172.21.15.52:0/700213970 >> 172.21.15.52:0/700213970 conn(0x7f309c0764e0 msgr2=0x7f309c078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 -- 172.21.15.52:0/700213970 shutdown_connections 2024-04-26T01:53:35.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.260+0000 7f30a2e00640 1 -- 172.21.15.52:0/700213970 wait complete. 2024-04-26T01:53:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 Processor -- start 2024-04-26T01:53:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 -- start start 2024-04-26T01:53:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1187e0 con 0x7f309c108f20 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f309c118660 con 0x7f309c10e620 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.261+0000 7f30a2e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f309c118960 con 0x7f309c108560 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57826/0 (socket says 172.21.15.52:57826) 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 -- 172.21.15.52:0/2009113152 learned_addr learned my addr 172.21.15.52:0/2009113152 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f309be00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 -- 172.21.15.52:0/2009113152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 msgr2=0x7f309c1a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 -- 172.21.15.52:0/2009113152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 msgr2=0x7f309c19ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c19ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f3093400640 1 -- 172.21.15.52:0/2009113152 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c19f200 con 0x7f309c108f20 2024-04-26T01:53:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.262+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f3093400640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3084005e30 tx=0x7f30840039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f3099600640 1 -- 172.21.15.52:0/2009113152 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084004040 con 0x7f309c108f20 2024-04-26T01:53:35.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f3099600640 1 -- 172.21.15.52:0/2009113152 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3084004690 con 0x7f309c108f20 2024-04-26T01:53:35.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309c19f490 con 0x7f309c108f20 2024-04-26T01:53:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f3099600640 1 -- 172.21.15.52:0/2009113152 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084046480 con 0x7f309c108f20 2024-04-26T01:53:35.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.263+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309c1ab790 con 0x7f309c108f20 2024-04-26T01:53:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.266+0000 7f30a5bf1740 1 Processor -- start 2024-04-26T01:53:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 -- start start 2024-04-26T01:53:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7ded4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f3099600640 1 -- 172.21.15.52:0/2009113152 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3084030630 con 0x7f309c108f20 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7defdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55efe7d48900 con 0x55efe7de2820 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55efe7ceeef0 con 0x55efe7de3140 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efe7d448c0 con 0x55efe7deda00 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a0e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a0e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57828/0 (socket says 172.21.15.52:57828) 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f30a0e00640 1 -- 172.21.15.52:0/2832046172 learned_addr learned my addr 172.21.15.52:0/2832046172 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f3093400640 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7defdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.267+0000 7f309be00640 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7ded4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 -- 172.21.15.52:0/2832046172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 msgr2=0x55efe7defdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7defdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 -- 172.21.15.52:0/2832046172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 msgr2=0x55efe7ded4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7ded4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 -- 172.21.15.52:0/2832046172 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efe7a5b210 con 0x55efe7de2820 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3094011540 tx=0x7f3094011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157dd9ee9ce66223 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f3091600640 1 -- 172.21.15.52:0/2832046172 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309400f5c0 con 0x55efe7de2820 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.268+0000 7f3091600640 1 -- 172.21.15.52:0/2832046172 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309400fc10 con 0x55efe7de2820 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f3091600640 1 -- 172.21.15.52:0/2832046172 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094019bf0 con 0x55efe7de2820 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f3099600640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 0x7f30800a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2832046172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 msgr2=0x55efe7de2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3094011540 tx=0x7f3094011a00 comp rx=0 tx=0).stop 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f3099600640 1 -- 172.21.15.52:0/2009113152 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f308402e030 con 0x7f309c108f20 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.269+0000 7f309be00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 0x7f30800a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2832046172 shutdown_connections 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7defdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7ded4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2832046172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7de2c00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2832046172 >> 172.21.15.52:0/2832046172 conn(0x55efe7dded30 msgr2=0x55efe7ddf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2832046172 shutdown_connections 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2832046172 wait complete. 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f309be00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 0x7f30800a6280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55efe7de3dc0 tx=0x7f308c005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.270+0000 7f30a5bf1740 1 -- start start 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 Processor -- start 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7deaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7deb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55efe7df0630 con 0x55efe7de2820 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55efe7d02310 con 0x55efe7de3140 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efe7df07b0 con 0x55efe7deda00 2024-04-26T01:53:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a5bf1740 10 client.0 _ll_register_callbacks cb 0x55efe7cd0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f309be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f309be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57838/0 (socket says 172.21.15.52:57838) 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f309be00640 1 -- 172.21.15.52:0/473365336 learned_addr learned my addr 172.21.15.52:0/473365336 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.271+0000 7f30a0e00640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7deb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f3093400640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7deaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 -- 172.21.15.52:0/473365336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 msgr2=0x55efe7deb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7deb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 -- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 msgr2=0x55efe7deaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7deaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 -- 172.21.15.52:0/473365336 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efe7b27620 con 0x55efe7de2820 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f3093400640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7deaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f309be00640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f308c004620 tx=0x7f308c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c550eeb5738f3bda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f308a000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308c01f640 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f308c01f7e0 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.272+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309400fc10 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f308c01f9e0 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 10 client.16014 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55efe7dfc910 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309400fc10 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f308c027070 con 0x55efe7de2820 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 10 client.16014 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 20 client.16014 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7dfde20 0x55efe7e00250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 4 client.16014 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.273+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efe7d5c8b0 con 0x55efe7dfde20 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.274+0000 7f3093400640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7dfde20 0x55efe7e00250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.274+0000 7f3093400640 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7dfde20 0x55efe7e00250 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bcc176378f9189d server_cookie=4096da635d5d1277 in_seq=0 out_seq=0 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.274+0000 7f308a000640 10 client.16014 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f308a000640 1 -- 172.21.15.52:0/473365336 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efe7d5c8b0 con 0x55efe7dfde20 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f308a000640 10 client.16014 handle_command_reply: tid=0 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 client.16014 shutdown 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 msgr2=0x55efe7dea970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f308c004620 tx=0x7f308c004730 comp rx=0 tx=0).stop 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 shutdown_connections 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7dfde20 0x55efe7e00250 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7deb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de3140 0x55efe7deaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/473365336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de2820 0x55efe7dea970 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 >> 172.21.15.52:0/473365336 conn(0x55efe7dded30 msgr2=0x55efe7de4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.275+0000 7f308a000640 0 client.16014 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.276+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 shutdown_connections 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.276+0000 7f30a5bf1740 1 -- 172.21.15.52:0/473365336 wait complete. 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.276+0000 7f30a5bf1740 20 client.16014 trim_cache size 0 max 16384 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.292+0000 7f30a5bf1740 1 Processor -- start 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.292+0000 7f30a5bf1740 1 -- start start 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.292+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 0x55efe7df94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.292+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7e05b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b69a0 con 0x55efe7de3140 2024-04-26T01:53:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b4270 con 0x55efe7de2820 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b20e0 con 0x55efe7deda00 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57840/0 (socket says 172.21.15.52:57840) 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 -- 172.21.15.52:0/2220170542 learned_addr learned my addr 172.21.15.52:0/2220170542 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f309be00640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 0x55efe7df94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7e05b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 -- 172.21.15.52:0/2220170542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 msgr2=0x55efe7e05b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7e05b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 -- 172.21.15.52:0/2220170542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 msgr2=0x55efe7df94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 0x55efe7df94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f3093400640 1 -- 172.21.15.52:0/2220170542 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efe7deb8d0 con 0x55efe7de3140 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f309be00640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 0x55efe7df94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.293+0000 7f30a0e00640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7e05b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.294+0000 7f3093400640 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3084031e80 tx=0x7f30840a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7438be3421bd567d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.294+0000 7f3090c00640 1 -- 172.21.15.52:0/2220170542 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30840bd790 con 0x55efe7de3140 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.294+0000 7f3090c00640 1 -- 172.21.15.52:0/2220170542 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3084036040 con 0x55efe7de3140 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.294+0000 7f3090c00640 1 -- 172.21.15.52:0/2220170542 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30840bd320 con 0x55efe7de3140 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.294+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2220170542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 msgr2=0x55efe7de5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3084031e80 tx=0x7f30840a8040 comp rx=0 tx=0).stop 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2220170542 shutdown_connections 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7deda00 0x55efe7e05b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7de5230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/2220170542 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7de2820 0x55efe7df94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2220170542 >> 172.21.15.52:0/2220170542 conn(0x55efe7dded30 msgr2=0x55efe7deb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2220170542 shutdown_connections 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 -- 172.21.15.52:0/2220170542 wait complete. 2024-04-26T01:53:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 -- start start 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.295+0000 7f30a5bf1740 1 Processor -- start 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7de2820 0x55efe7df5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7deda00 0x55efe7df35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b2730 con 0x55efe7de3140 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b3ba0 con 0x55efe7deda00 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f309c1b2e00 con 0x55efe7de2820 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a5bf1740 10 client.0 _ll_register_callbacks cb 0x7f308c0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f3093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57846/0 (socket says 172.21.15.52:57846) 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f3093400640 1 -- 172.21.15.52:0/3111737276 learned_addr learned my addr 172.21.15.52:0/3111737276 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f309be00640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7de2820 0x55efe7df5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f30a0e00640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7deda00 0x55efe7df35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.296+0000 7f3093400640 1 -- 172.21.15.52:0/3111737276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7de2820 msgr2=0x55efe7df5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f3093400640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7de2820 0x55efe7df5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f3093400640 1 -- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7deda00 msgr2=0x55efe7df35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f3093400640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7deda00 0x55efe7df35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f3093400640 1 -- 172.21.15.52:0/3111737276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efe7df3dd0 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f3093400640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30840ba390 tx=0x7f30840309d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a33af4345b8491 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f308a000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30840bd500 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f30840ef2e0 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3084036040 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f30840bd790 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f30a5bf1740 10 client.16020 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.297+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55efe7e64380 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3084036040 con 0x55efe7de3140 2024-04-26T01:53:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3084030070 con 0x55efe7de3140 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 10 client.16020 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 20 client.16020 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7e64aa0 0x55efe7e66ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 4 client.16020 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55efe7de6570 con 0x55efe7e64aa0 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.298+0000 7f309be00640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7e64aa0 0x55efe7e66ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.299+0000 7f309be00640 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7e64aa0 0x55efe7e66ec0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc1897f2f4e119f7 server_cookie=899bce49019b54cc in_seq=0 out_seq=0 2024-04-26T01:53:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.299+0000 7f308a000640 10 client.16020 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f308a000640 1 -- 172.21.15.52:0/3111737276 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55efe7de6570 con 0x55efe7e64aa0 2024-04-26T01:53:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f308a000640 10 client.16020 handle_command_reply: tid=0 2024-04-26T01:53:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 client.16020 shutdown 2024-04-26T01:53:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 msgr2=0x55efe7df3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30840ba390 tx=0x7f30840309d0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 shutdown_connections 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55efe7e64aa0 0x55efe7e66ec0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55efe7deda00 0x55efe7df35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55efe7de3140 0x55efe7df3090 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 --2- 172.21.15.52:0/3111737276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efe7de2820 0x55efe7df5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f308a000640 0 client.16020 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 >> 172.21.15.52:0/3111737276 conn(0x55efe7dded30 msgr2=0x55efe7ddd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 shutdown_connections 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.329+0000 7f30a5bf1740 1 -- 172.21.15.52:0/3111737276 wait complete. 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.330+0000 7f30a5bf1740 20 client.16020 trim_cache size 0 max 16384 2024-04-26T01:53:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.330+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 msgr2=0x7f30800a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.330+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 0x7f30800a6280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55efe7de3dc0 tx=0x7f308c005f90 comp rx=0 tx=0).stop 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.330+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 msgr2=0x7f309c1a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:35.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.330+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3084005e30 tx=0x7f30840039f0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 shutdown_connections 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f30800a3dc0 0x7f30800a6280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f309c10e620 0x7f309c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f309c108f20 0x7f309c1a4e10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 --2- 172.21.15.52:0/2009113152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c108560 0x7f309c1a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 >> 172.21.15.52:0/2009113152 conn(0x7f309c0764e0 msgr2=0x7f309c077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 shutdown_connections 2024-04-26T01:53:35.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:35.331+0000 7f30a2e00640 1 -- 172.21.15.52:0/2009113152 wait complete. 2024-04-26T01:53:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:35 smithi191 ceph-mon[32745]: pgmap v265: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 583 KiB/s wr, 37 op/s 2024-04-26T01:53:35.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:35 smithi183 ceph-mon[32752]: pgmap v265: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 583 KiB/s wr, 37 op/s 2024-04-26T01:53:35.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:35 smithi052 ceph-mon[27851]: pgmap v265: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 583 KiB/s wr, 37 op/s 2024-04-26T01:53:36.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.79 MB/sec execute 295 sec latency 23025.641 ms 2024-04-26T01:53:36.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:53:37.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.78 MB/sec execute 296 sec latency 24025.769 ms 2024-04-26T01:53:37.434 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:37 smithi191 ceph-mon[32745]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 656 KiB/s wr, 59 op/s 2024-04-26T01:53:37.668 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:37 smithi052 ceph-mon[27851]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 656 KiB/s wr, 59 op/s 2024-04-26T01:53:37.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:37 smithi183 ceph-mon[32752]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 656 KiB/s wr, 59 op/s 2024-04-26T01:53:38.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.77 MB/sec execute 297 sec latency 25025.929 ms 2024-04-26T01:53:39.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.76 MB/sec execute 298 sec latency 26026.075 ms 2024-04-26T01:53:39.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:39 smithi052 ceph-mon[27851]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:39 smithi191 ceph-mon[32745]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:39.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:39 smithi183 ceph-mon[32752]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:39.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 -- 172.21.15.52:0/914566317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 msgr2=0x7f5370105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 --2- 172.21.15.52:0/914566317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f5370105910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f536800b0a0 tx=0x7f536800b3b0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 -- 172.21.15.52:0/914566317 shutdown_connections 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 --2- 172.21.15.52:0/914566317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f5370105910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 --2- 172.21.15.52:0/914566317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 0x7f537006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 --2- 172.21.15.52:0/914566317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 0x7f537006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 -- 172.21.15.52:0/914566317 >> 172.21.15.52:0/914566317 conn(0x7f53700788f0 msgr2=0x7f5370078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 -- 172.21.15.52:0/914566317 shutdown_connections 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.896+0000 7f5376c00700 1 -- 172.21.15.52:0/914566317 wait complete. 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.897+0000 7f5376c00700 1 Processor -- start 2024-04-26T01:53:39.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 -- start start 2024-04-26T01:53:39.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 0x7f537019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:39.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:39.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 0x7f53701a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:39.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f53701191b0 con 0x7f5370105480 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5370119030 con 0x7f5370107970 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5376c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5370119330 con 0x7f537006ac90 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5367400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5367400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57872/0 (socket says 172.21.15.52:57872) 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f5367400700 1 -- 172.21.15.52:0/625841296 learned_addr learned my addr 172.21.15.52:0/625841296 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.898+0000 7f536fe00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 0x7f537019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5374c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 0x7f53701a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 -- 172.21.15.52:0/625841296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 msgr2=0x7f537019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 0x7f537019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 -- 172.21.15.52:0/625841296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 msgr2=0x7f53701a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 0x7f53701a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 -- 172.21.15.52:0/625841296 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53701a46a0 con 0x7f5370105480 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.899+0000 7f5367400700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f535800e9a0 tx=0x7f535800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:39.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.900+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535800ca10 con 0x7f5370105480 2024-04-26T01:53:39.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.900+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535800cbb0 con 0x7f5370105480 2024-04-26T01:53:39.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.900+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53701a4990 con 0x7f5370105480 2024-04-26T01:53:39.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.900+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358018680 con 0x7f5370105480 2024-04-26T01:53:39.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.900+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701ac210 con 0x7f5370105480 2024-04-26T01:53:39.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.901+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5370069730 con 0x7f5370105480 2024-04-26T01:53:39.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.903+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5358010430 con 0x7f5370105480 2024-04-26T01:53:39.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.905+0000 7f536d600700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 0x7f53540a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:39.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.905+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f53580d0970 con 0x7f5370105480 2024-04-26T01:53:39.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.905+0000 7f536fe00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 0x7f53540a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:39.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.906+0000 7f536fe00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 0x7f53540a6270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5360009680 tx=0x7f5360009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:39.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:39.910+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f535809cfb0 con 0x7f5370105480 2024-04-26T01:53:40.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.75 MB/sec execute 299 sec latency 27026.200 ms 2024-04-26T01:53:40.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.192+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53701ac410 con 0x7f5370105480 2024-04-26T01:53:40.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.195+0000 7f536d600700 1 -- 172.21.15.52:0/625841296 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f535809c8b0 con 0x7f5370105480 2024-04-26T01:53:40.194 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:53:40.194 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:51:02.817858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25664":{"gid":25664,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2373306392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2373306392},{"type":"v1","addr":"172.21.15.183:6837","nonce":2373306392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:53:40.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 msgr2=0x7f53540a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 0x7f53540a6270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5360009680 tx=0x7f5360009260 comp rx=0 tx=0).stop 2024-04-26T01:53:40.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 msgr2=0x7f537019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:40.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f535800e9a0 tx=0x7f535800ecb0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 shutdown_connections 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53540a3db0 0x7f53540a6270 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5370107970 0x7f53701a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537006ac90 0x7f537019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 --2- 172.21.15.52:0/625841296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5370105480 0x7f537019fc20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 >> 172.21.15.52:0/625841296 conn(0x7f53700788f0 msgr2=0x7f5370104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 shutdown_connections 2024-04-26T01:53:40.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:40.198+0000 7f5376c00700 1 -- 172.21.15.52:0/625841296 wait complete. 2024-04-26T01:53:40.198 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 38 2024-04-26T01:53:40.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:40 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/625841296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:40.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:40 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/625841296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:40.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:40 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/625841296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:53:41.081 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id af4e799c --release --await' 2024-04-26T01:53:41.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id af4e799c --release --await 2024-04-26T01:53:41.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.74 MB/sec execute 300 sec latency 28026.297 ms 2024-04-26T01:53:41.545 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:53:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:41 smithi191 ceph-mon[32745]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:41.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:41 smithi183 ceph-mon[32752]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:41.696 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:41 smithi052 ceph-mon[27851]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 334 KiB/s wr, 37 op/s 2024-04-26T01:53:42.113 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.73 MB/sec execute 301 sec latency 29026.361 ms 2024-04-26T01:53:43.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206096 2.72 MB/sec execute 302 sec latency 30026.498 ms 2024-04-26T01:53:43.577 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:43 smithi052 ceph-mon[27851]: pgmap v269: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 772 KiB/s wr, 49 op/s 2024-04-26T01:53:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:43 smithi191 ceph-mon[32745]: pgmap v269: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 772 KiB/s wr, 49 op/s 2024-04-26T01:53:43.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:43 smithi183 ceph-mon[32752]: pgmap v269: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 772 KiB/s wr, 49 op/s 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 -- 172.21.15.52:0/4101969286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18074e70 msgr2=0x7f3a18075250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4101969286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18074e70 0x7f3a18075250 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009990 tx=0x7f3a08009ca0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 -- 172.21.15.52:0/4101969286 shutdown_connections 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4101969286 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a18111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4101969286 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18074e70 0x7f3a18075250 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4101969286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18075790 0x7f3a1810ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 -- 172.21.15.52:0/4101969286 >> 172.21.15.52:0/4101969286 conn(0x7f3a180fe450 msgr2=0x7f3a18100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 -- 172.21.15.52:0/4101969286 shutdown_connections 2024-04-26T01:53:43.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.890+0000 7f3a1d000700 1 -- 172.21.15.52:0/4101969286 wait complete. 2024-04-26T01:53:43.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.891+0000 7f3a1d000700 1 Processor -- start 2024-04-26T01:53:43.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.891+0000 7f3a1d000700 1 -- start start 2024-04-26T01:53:43.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 0x7f3a1819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 0x7f3a1819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a1819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3a18114d30 con 0x7f3a18075790 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a18114bb0 con 0x7f3a18074e70 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a1d000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3a18114eb0 con 0x7f3a1810f1a0 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a0ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 0x7f3a1819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a0ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 0x7f3a1819b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57884/0 (socket says 172.21.15.52:57884) 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a17e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a1819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a0ea00700 1 -- 172.21.15.52:0/4112208994 learned_addr learned my addr 172.21.15.52:0/4112208994 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:43.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.892+0000 7f3a17400700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 0x7f3a1819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 -- 172.21.15.52:0/4112208994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 msgr2=0x7f3a1819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 0x7f3a1819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 -- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 msgr2=0x7f3a1819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 0x7f3a1819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 -- 172.21.15.52:0/4112208994 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a181a0240 con 0x7f3a1810f1a0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17400700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 0x7f3a1819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a17e00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a1819fb00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000e970 tx=0x7f3a1000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a14c00700 1 -- 172.21.15.52:0/4112208994 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10004220 con 0x7f3a1810f1a0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a181a0530 con 0x7f3a1810f1a0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.893+0000 7f3a14c00700 1 -- 172.21.15.52:0/4112208994 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a100043c0 con 0x7f3a1810f1a0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.894+0000 7f3a14c00700 1 -- 172.21.15.52:0/4112208994 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10010700 con 0x7f3a1810f1a0 2024-04-26T01:53:43.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.894+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a181a7db0 con 0x7f3a1810f1a0 2024-04-26T01:53:43.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.898+0000 7f3a14c00700 1 -- 172.21.15.52:0/4112208994 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3a10004750 con 0x7f3a1810f1a0 2024-04-26T01:53:43.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 Processor -- start 2024-04-26T01:53:43.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 -- start start 2024-04-26T01:53:43.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a14c00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 0x7f39fc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c67c0 0x5606703d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703d1670 0x5606703d3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560670318f90 con 0x5606703c67c0 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5606702bf830 con 0x5606703d1670 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560670314f50 con 0x5606703c5ea0 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.900+0000 7f3a14c00700 1 -- 172.21.15.52:0/4112208994 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f3a100d05d0 con 0x7f3a1810f1a0 2024-04-26T01:53:43.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 0x7f39fc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52726/0 (socket says 172.21.15.52:52726) 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a0ea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c67c0 0x5606703d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 -- 172.21.15.52:0/3916837327 learned_addr learned my addr 172.21.15.52:0/3916837327 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17e00700 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703d1670 0x5606703d3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 -- 172.21.15.52:0/3916837327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703d1670 msgr2=0x5606703d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703d1670 0x5606703d3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 -- 172.21.15.52:0/3916837327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c67c0 msgr2=0x5606703d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c67c0 0x5606703d1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.901+0000 7f3a17400700 1 -- 172.21.15.52:0/3916837327 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606702b5780 con 0x5606703c5ea0 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.902+0000 7f3a17400700 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009990 tx=0x7f3a08005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a18b7115107de6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.902+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 0x7f39fc0a6270 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009da0 tx=0x7f3a00009920 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.902+0000 7f3a0cc00700 1 -- 172.21.15.52:0/3916837327 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08027070 con 0x5606703c5ea0 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.902+0000 7f3a0cc00700 1 -- 172.21.15.52:0/3916837327 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a08003680 con 0x5606703c5ea0 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.902+0000 7f3a0cc00700 1 -- 172.21.15.52:0/3916837327 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0800f6a0 con 0x5606703c5ea0 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/3916837327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 msgr2=0x5606703c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009990 tx=0x7f3a08005960 comp rx=0 tx=0).stop 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/3916837327 shutdown_connections 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703d1670 0x5606703d3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703c6280 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/3916837327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c67c0 0x5606703d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/3916837327 >> 172.21.15.52:0/3916837327 conn(0x5606702f5050 msgr2=0x5606703c23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/3916837327 shutdown_connections 2024-04-26T01:53:43.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.903+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/3916837327 wait complete. 2024-04-26T01:53:43.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.904+0000 7f3a28f7ab80 1 -- start start 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.904+0000 7f3a28f7ab80 1 Processor -- start 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.904+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 0x5606703d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.904+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.904+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 0x5606703ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5606703d40a0 con 0x5606703c5ea0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5606702d2960 con 0x5606703c67c0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5606703d4220 con 0x5606703d1670 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a28f7ab80 10 client.0 _ll_register_callbacks cb 0x5606702b1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a17400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 0x5606703d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a17400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 0x5606703d0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57904/0 (socket says 172.21.15.52:57904) 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a17400700 1 -- 172.21.15.52:0/555668498 learned_addr learned my addr 172.21.15.52:0/555668498 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a17e00700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 0x5606703ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 -- 172.21.15.52:0/555668498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 msgr2=0x5606703ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 0x5606703ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 -- 172.21.15.52:0/555668498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 msgr2=0x5606703d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 0x5606703d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a0ea00700 1 -- 172.21.15.52:0/555668498 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606702669d0 con 0x5606703c67c0 2024-04-26T01:53:43.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.905+0000 7f3a17e00700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 0x5606703ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:43.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.906+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703ce0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009690 tx=0x7f3a00004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8007ea771237b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.906+0000 7f3a05600700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.906+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00029020 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.906+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f3a0001a7c0 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.906+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a08003680 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3a00034050 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 10 client.26156 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5606702afbf0 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a08003680 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3a00024bf0 con 0x5606703c67c0 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 10 client.26156 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 10 client.26156 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 20 client.26156 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606703e0e60 0x5606703e3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 4 client.26156 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:43.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.907+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560670326780 con 0x5606703e0e60 2024-04-26T01:53:43.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.908+0000 7f3a17400700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606703e0e60 0x5606703e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.908+0000 7f3a17400700 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606703e0e60 0x5606703e3250 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=173969c597b8fd10 server_cookie=e457161f1181fe51 in_seq=0 out_seq=0 2024-04-26T01:53:43.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.908+0000 7f3a05600700 10 client.26156 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.909+0000 7f3a05600700 1 -- 172.21.15.52:0/555668498 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560670326780 con 0x5606703e0e60 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.909+0000 7f3a05600700 10 client.26156 handle_command_reply: tid=0 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 client.26156 shutdown 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 msgr2=0x5606703ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703ce0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009690 tx=0x7f3a00004b40 comp rx=0 tx=0).stop 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 shutdown_connections 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606703e0e60 0x5606703e3250 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703d1670 0x5606703ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703c5ea0 0x5606703d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/555668498 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703ce0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a05600700 0 client.26156 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 >> 172.21.15.52:0/555668498 conn(0x5606702f5050 msgr2=0x5606703c1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 shutdown_connections 2024-04-26T01:53:43.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/555668498 wait complete. 2024-04-26T01:53:43.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.910+0000 7f3a28f7ab80 20 client.26156 trim_cache size 0 max 16384 2024-04-26T01:53:43.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 Processor -- start 2024-04-26T01:53:43.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 -- start start 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 0x5606703dcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c67c0 0x5606703c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181b1bb0 con 0x5606703d1670 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181af440 con 0x5606703c5ea0 2024-04-26T01:53:43.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.932+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181ad2b0 con 0x5606703c67c0 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57918/0 (socket says 172.21.15.52:57918) 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 -- 172.21.15.52:0/1903627581 learned_addr learned my addr 172.21.15.52:0/1903627581 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17400700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 0x5606703dcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c67c0 0x5606703c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 -- 172.21.15.52:0/1903627581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c67c0 msgr2=0x5606703c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c67c0 0x5606703c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 -- 172.21.15.52:0/1903627581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 msgr2=0x5606703dcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 0x5606703dcf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17e00700 1 -- 172.21.15.52:0/1903627581 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606703c7cf0 con 0x5606703d1670 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.933+0000 7f3a17400700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 0x5606703dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.934+0000 7f3a17e00700 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a100cdd20 tx=0x7f3a10009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e291d4221db8b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.934+0000 7f3a07e00700 1 -- 172.21.15.52:0/1903627581 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1001c040 con 0x5606703d1670 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.934+0000 7f3a07e00700 1 -- 172.21.15.52:0/1903627581 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a1009cc10 con 0x5606703d1670 2024-04-26T01:53:43.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.934+0000 7f3a07e00700 1 -- 172.21.15.52:0/1903627581 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1009c9d0 con 0x5606703d1670 2024-04-26T01:53:43.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/1903627581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 msgr2=0x5606703c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a100cdd20 tx=0x7f3a10009770 comp rx=0 tx=0).stop 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/1903627581 shutdown_connections 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x5606703c21b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c5ea0 0x5606703dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/1903627581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c67c0 0x5606703c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/1903627581 >> 172.21.15.52:0/1903627581 conn(0x5606702f5050 msgr2=0x5606703c1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/1903627581 shutdown_connections 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/1903627581 wait complete. 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 -- start start 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.935+0000 7f3a28f7ab80 1 Processor -- start 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703d6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x56067044fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181ad900 con 0x5606703d1670 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181aed70 con 0x5606703c67c0 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181adfd0 con 0x5606703c5ea0 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a28f7ab80 10 client.0 _ll_register_callbacks cb 0x5606702b1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a17400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a17400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52752/0 (socket says 172.21.15.52:52752) 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a17400700 1 -- 172.21.15.52:0/2415497105 learned_addr learned my addr 172.21.15.52:0/2415497105 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703d6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.936+0000 7f3a17e00700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x56067044fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 -- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 msgr2=0x5606703d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703d6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 -- 172.21.15.52:0/2415497105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 msgr2=0x56067044fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x56067044fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 -- 172.21.15.52:0/2415497105 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606703de830 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a17400700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009ce0 tx=0x7f3a08009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d0719ae2639145 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a05600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08039060 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f3a080050b0 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.937+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a1009cc10 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3a08027040 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a28f7ab80 10 client.26164 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560670451250 con 0x5606703c5ea0 2024-04-26T01:53:43.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a1009cc10 con 0x5606703c5ea0 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f3a08033780 con 0x5606703c5ea0 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a28f7ab80 10 client.26164 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a28f7ab80 10 client.26164 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.938+0000 7f3a28f7ab80 20 client.26164 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606704514e0 0x5606704538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a28f7ab80 4 client.26164 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "af4e799c", "release": true, "await": true}] 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "af4e799c", "release": true, "await": true}) -- 0x560670320820 con 0x5606704514e0 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606704514e0 0x5606704538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a0ea00700 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606704514e0 0x5606704538d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=201e71445cc32de6 server_cookie=4152c464bc6d7c27 in_seq=0 out_seq=0 2024-04-26T01:53:43.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.939+0000 7f3a05600700 10 client.26164 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:43.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a05600700 1 -- 172.21.15.52:0/2415497105 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560670320820 con 0x5606704514e0 2024-04-26T01:53:43.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a05600700 10 client.26164 handle_command_reply: tid=0 2024-04-26T01:53:43.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 client.26164 shutdown 2024-04-26T01:53:43.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 msgr2=0x5606703d6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009ce0 tx=0x7f3a08009300 comp rx=0 tx=0).stop 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 shutdown_connections 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5606704514e0 0x5606704538d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5606703d1670 0x56067044fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5606703c5ea0 0x5606703d6620 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a05600700 0 client.26164 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 --2- 172.21.15.52:0/2415497105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5606703c67c0 0x5606703d6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 >> 172.21.15.52:0/2415497105 conn(0x5606702f5050 msgr2=0x5606703c1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 shutdown_connections 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 1 -- 172.21.15.52:0/2415497105 wait complete. 2024-04-26T01:53:43.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.947+0000 7f3a28f7ab80 20 client.26164 trim_cache size 0 max 16384 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 38, 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 24, 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "af4e799c": { 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 24, 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:53:43.947 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.9, 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:53:43.948 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.948+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 msgr2=0x7f39fc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.948+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 0x7f39fc0a6270 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009da0 tx=0x7f3a00009920 comp rx=0 tx=0).stop 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.948+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 msgr2=0x7f3a1819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.948+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a1819fb00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000e970 tx=0x7f3a1000ec80 comp rx=0 tx=0).stop 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 shutdown_connections 2024-04-26T01:53:43.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39fc0a3db0 0x7f39fc0a6270 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a1810f1a0 0x7f3a1819fb00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a18074e70 0x7f3a1819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 --2- 172.21.15.52:0/4112208994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3a18075790 0x7f3a1819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 >> 172.21.15.52:0/4112208994 conn(0x7f3a180fe450 msgr2=0x7f3a180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 shutdown_connections 2024-04-26T01:53:43.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:53:43.949+0000 7f3a1d000700 1 -- 172.21.15.52:0/4112208994 wait complete. 2024-04-26T01:53:44.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206144 2.71 MB/sec execute 303 sec latency 30862.636 ms 2024-04-26T01:53:44.878 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'af4e799c', seconds elapsed: 42.2 2024-04-26T01:53:45.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 206837 2.71 MB/sec execute 304 sec latency 24.289 ms 2024-04-26T01:53:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:45 smithi191 ceph-mon[32745]: pgmap v270: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 511 KiB/s wr, 34 op/s 2024-04-26T01:53:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:45 smithi183 ceph-mon[32752]: pgmap v270: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 511 KiB/s wr, 34 op/s 2024-04-26T01:53:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:45 smithi052 ceph-mon[27851]: pgmap v270: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 511 KiB/s wr, 34 op/s 2024-04-26T01:53:46.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 207770 2.71 MB/sec execute 305 sec latency 26.704 ms 2024-04-26T01:53:47.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 209238 2.74 MB/sec execute 306 sec latency 15.902 ms 2024-04-26T01:53:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:47 smithi191 ceph-mon[32745]: pgmap v271: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 704 KiB/s wr, 63 op/s 2024-04-26T01:53:47.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:47 smithi183 ceph-mon[32752]: pgmap v271: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 704 KiB/s wr, 63 op/s 2024-04-26T01:53:47.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:47 smithi052 ceph-mon[27851]: pgmap v271: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 704 KiB/s wr, 63 op/s 2024-04-26T01:53:48.114 INFO:tasks.workunit.client.0.smithi052.stdout: 1 209660 2.74 MB/sec execute 307 sec latency 26.087 ms 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 -- 172.21.15.52:0/3488786482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a8104d60 msgr2=0x7f87a8108da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 --2- 172.21.15.52:0/3488786482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a8104d60 0x7f87a8108da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8794009860 tx=0x7f879402a340 comp rx=0 tx=0).stop 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 -- 172.21.15.52:0/3488786482 shutdown_connections 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 --2- 172.21.15.52:0/3488786482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 0x7f87a8111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 --2- 172.21.15.52:0/3488786482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a8104d60 0x7f87a8108da0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 --2- 172.21.15.52:0/3488786482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a8104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 -- 172.21.15.52:0/3488786482 >> 172.21.15.52:0/3488786482 conn(0x7f87a80ffc50 msgr2=0x7f87a8102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.487+0000 7f87ae000640 1 -- 172.21.15.52:0/3488786482 shutdown_connections 2024-04-26T01:53:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.488+0000 7f87ae000640 1 -- 172.21.15.52:0/3488786482 wait complete. 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 Processor -- start 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 -- start start 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 0x7f87a81a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 0x7f87a819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81143a0 con 0x7f87a8109480 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8114220 con 0x7f87a81043b0 2024-04-26T01:53:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.489+0000 7f87ae000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8114520 con 0x7f87a819a720 2024-04-26T01:53:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49530/0 (socket says 172.21.15.52:49530) 2024-04-26T01:53:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 -- 172.21.15.52:0/2028410148 learned_addr learned my addr 172.21.15.52:0/2028410148 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7e00640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 0x7f87a819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a6a00640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 0x7f87a81a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 -- 172.21.15.52:0/2028410148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 msgr2=0x7f87a819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 0x7f87a819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 -- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 msgr2=0x7f87a81a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 0x7f87a81a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7400640 1 -- 172.21.15.52:0/2028410148 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a819b320 con 0x7f87a81043b0 2024-04-26T01:53:48.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.490+0000 7f87a7e00640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 0x7f87a819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f87a7400640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9c0 tx=0x7f879800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f879fe00640 1 -- 172.21.15.52:0/2028410148 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879800ccc0 con 0x7f87a81043b0 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a81a7400 con 0x7f87a81043b0 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f879fe00640 1 -- 172.21.15.52:0/2028410148 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87980054d0 con 0x7f87a81043b0 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f879fe00640 1 -- 172.21.15.52:0/2028410148 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798010430 con 0x7f87a81043b0 2024-04-26T01:53:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.491+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a81a7800 con 0x7f87a81043b0 2024-04-26T01:53:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.495+0000 7f87b0e5d740 1 Processor -- start 2024-04-26T01:53:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.495+0000 7f87b0e5d740 1 -- start start 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556cc0b0d900 con 0x556cc0bb2a00 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556cc0ab3ef0 con 0x556cc0ba8140 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556cc0b098c0 con 0x556cc0ba7820 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f879fe00640 1 -- 172.21.15.52:0/2028410148 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f87980046c0 con 0x7f87a81043b0 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40784/0 (socket says 172.21.15.52:40784) 2024-04-26T01:53:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 -- 172.21.15.52:0/1644035575 learned_addr learned my addr 172.21.15.52:0/1644035575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7400640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 -- 172.21.15.52:0/1644035575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 msgr2=0x556cc0bb24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a6a00640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.496+0000 7f87a7e00640 1 -- 172.21.15.52:0/1644035575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 msgr2=0x556cc0bb4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f87a7e00640 1 -- 172.21.15.52:0/1644035575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc0820210 con 0x556cc0ba7820 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f87a7400640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f87a6a00640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:53:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87a0011790 tx=0x7f87a0011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33056075ffeafbbe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f8793e00640 1 -- 172.21.15.52:0/1644035575 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0003c30 con 0x556cc0ba7820 2024-04-26T01:53:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f8793e00640 1 -- 172.21.15.52:0/1644035575 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a000fe50 con 0x556cc0ba7820 2024-04-26T01:53:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.497+0000 7f8793e00640 1 -- 172.21.15.52:0/1644035575 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0004840 con 0x556cc0ba7820 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f879fe00640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 0x7f878c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f879fe00640 1 -- 172.21.15.52:0/2028410148 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f87980d0ab0 con 0x7f87a81043b0 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1644035575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 msgr2=0x556cc0ba7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87a0011790 tx=0x7f87a0011c50 comp rx=0 tx=0).stop 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87a7400640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 0x7f878c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1644035575 shutdown_connections 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1644035575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0ba7c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.498+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1644035575 >> 172.21.15.52:0/1644035575 conn(0x556cc0ba3d30 msgr2=0x556cc0ba4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.499+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1644035575 shutdown_connections 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.499+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1644035575 wait complete. 2024-04-26T01:53:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.499+0000 7f87b0e5d740 1 -- start start 2024-04-26T01:53:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.499+0000 7f87a7400640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 0x7f878c0a6300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x556cc0ba8dc0 tx=0x7f8798009f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.499+0000 7f87b0e5d740 1 Processor -- start 2024-04-26T01:53:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.500+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.500+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bafe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556cc0bb5630 con 0x556cc0bb2a00 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556cc0ac7310 con 0x556cc0ba8140 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556cc0bb57b0 con 0x556cc0ba7820 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40800/0 (socket says 172.21.15.52:40800) 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a7e00640 1 -- 172.21.15.52:0/3799966315 learned_addr learned my addr 172.21.15.52:0/3799966315 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87b0e5d740 10 client.0 _ll_register_callbacks cb 0x556cc0a95cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bafe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a7e00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 -- 172.21.15.52:0/3799966315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 msgr2=0x556cc0baf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 -- 172.21.15.52:0/3799966315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 msgr2=0x556cc0bb0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.501+0000 7f87a6a00640 1 -- 172.21.15.52:0/3799966315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc08ec620 con 0x556cc0ba8140 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f87a7e00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f87a6a00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bafe30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f879402a860 tx=0x7f8794005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a6a15b74bd7e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f8790c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8794037040 con 0x556cc0ba8140 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a000fe50 con 0x556cc0ba8140 2024-04-26T01:53:48.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.502+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f8794030450 con 0x556cc0ba8140 2024-04-26T01:53:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f879404e050 con 0x556cc0ba8140 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 10 client.26171 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556cc0bc1790 con 0x556cc0ba8140 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a000fe50 con 0x556cc0ba8140 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f8794032440 con 0x556cc0ba8140 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 10 client.26171 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 10 client.26171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 20 client.26171 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0bc2c50 0x556cc0bc5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 4 client.26171 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.503+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cc0b218b0 con 0x556cc0bc2c50 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.504+0000 7f87a7e00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0bc2c50 0x556cc0bc5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.504+0000 7f87a7e00640 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0bc2c50 0x556cc0bc5060 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c24e9b81644f4e16 server_cookie=81bea8b61174373d in_seq=0 out_seq=0 2024-04-26T01:53:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.504+0000 7f8790c00640 10 client.26171 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.505+0000 7f8790c00640 1 -- 172.21.15.52:0/3799966315 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cc0b218b0 con 0x556cc0bc2c50 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.505+0000 7f8790c00640 10 client.26171 handle_command_reply: tid=0 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.505+0000 7f87b0e5d740 1 client.26171 shutdown 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.505+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 msgr2=0x556cc0bafe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.505+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bafe30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f879402a860 tx=0x7f8794005cf0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 shutdown_connections 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0bc2c50 0x556cc0bc5060 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0bb2a00 0x556cc0bb0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bafe30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f8790c00640 0 client.26171 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/3799966315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0baf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 >> 172.21.15.52:0/3799966315 conn(0x556cc0ba3d30 msgr2=0x556cc0ba9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 shutdown_connections 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 1 -- 172.21.15.52:0/3799966315 wait complete. 2024-04-26T01:53:48.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.506+0000 7f87b0e5d740 20 client.26171 trim_cache size 0 max 16384 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.527+0000 7f87b0e5d740 1 Processor -- start 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.527+0000 7f87b0e5d740 1 -- start start 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0bbe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba8140 0x556cc0baa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81b2630 con 0x556cc0ba8140 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81aff00 con 0x556cc0bb2a00 2024-04-26T01:53:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81add70 con 0x556cc0ba7820 2024-04-26T01:53:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49550/0 (socket says 172.21.15.52:49550) 2024-04-26T01:53:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 -- 172.21.15.52:0/1569822458 learned_addr learned my addr 172.21.15.52:0/1569822458 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba8140 0x556cc0baa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0bbe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 -- 172.21.15.52:0/1569822458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 msgr2=0x556cc0bbe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0bbe4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 -- 172.21.15.52:0/1569822458 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba8140 msgr2=0x556cc0baa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba8140 0x556cc0baa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.528+0000 7f87a7e00640 1 -- 172.21.15.52:0/1569822458 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cc0bae8a0 con 0x556cc0bb2a00 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.529+0000 7f87a7e00640 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f87a0005f10 tx=0x7f87a0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27898774ce127107 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.529+0000 7f8793400640 1 -- 172.21.15.52:0/1569822458 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a001f040 con 0x556cc0bb2a00 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.529+0000 7f8793400640 1 -- 172.21.15.52:0/1569822458 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a0011310 con 0x556cc0bb2a00 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.529+0000 7f8793400640 1 -- 172.21.15.52:0/1569822458 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a000fc80 con 0x556cc0bb2a00 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1569822458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 msgr2=0x556cc0ba2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f87a0005f10 tx=0x7f87a0011c50 comp rx=0 tx=0).stop 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1569822458 shutdown_connections 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0bb2a00 0x556cc0ba2e90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba8140 0x556cc0baa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/1569822458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0ba7820 0x556cc0bbe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1569822458 >> 172.21.15.52:0/1569822458 conn(0x556cc0ba3d30 msgr2=0x556cc0ba4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1569822458 shutdown_connections 2024-04-26T01:53:48.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.530+0000 7f87b0e5d740 1 -- 172.21.15.52:0/1569822458 wait complete. 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 -- start start 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 Processor -- start 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 0x556cc0bbaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0bb2a00 0x556cc0bb8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81ae3c0 con 0x556cc0ba7820 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81af830 con 0x556cc0ba8140 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81aea90 con 0x556cc0bb2a00 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.531+0000 7f87b0e5d740 10 client.0 _ll_register_callbacks cb 0x7f87940096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49560/0 (socket says 172.21.15.52:49560) 2024-04-26T01:53:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 -- 172.21.15.52:0/237740554 learned_addr learned my addr 172.21.15.52:0/237740554 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 0x556cc0bbaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a7e00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0bb2a00 0x556cc0bb8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 -- 172.21.15.52:0/237740554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0bb2a00 msgr2=0x556cc0bb8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0bb2a00 0x556cc0bb8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 -- 172.21.15.52:0/237740554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 msgr2=0x556cc0bbaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 0x556cc0bbaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 -- 172.21.15.52:0/237740554 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cc0bb9060 con 0x556cc0ba8140 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a7e00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 0x556cc0bbaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f87a6a00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556cc0bb0ec0 tx=0x7f8794032670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186e16f665932177 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.532+0000 7f8790c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8794056070 con 0x556cc0ba8140 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a0011310 con 0x556cc0ba8140 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7f8794030450 con 0x556cc0ba8140 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8794061050 con 0x556cc0ba8140 2024-04-26T01:53:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 10 client.26177 fetch_fsmap learned FSMap version 38 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556cc0c29300 con 0x556cc0ba8140 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a0011310 con 0x556cc0ba8140 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f8794003aa0 con 0x556cc0ba8140 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 10 client.26177 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 20 client.26177 populate_metadata read hostname 'smithi052' 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0c299c0 0x556cc0c2bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 4 client.26177 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.533+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556cc0b1b320 con 0x556cc0c299c0 2024-04-26T01:53:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.534+0000 7f87a7e00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0c299c0 0x556cc0c2bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:53:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.534+0000 7f87a7e00640 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0c299c0 0x556cc0c2bde0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c39b3712f835f1a server_cookie=229d5391833e435d in_seq=0 out_seq=0 2024-04-26T01:53:48.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.534+0000 7f8790c00640 10 client.26177 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f8790c00640 1 -- 172.21.15.52:0/237740554 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556cc0b1b320 con 0x556cc0c299c0 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f8790c00640 10 client.26177 handle_command_reply: tid=0 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 client.26177 shutdown 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 msgr2=0x556cc0bb8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556cc0bb0ec0 tx=0x7f8794032670 comp rx=0 tx=0).stop 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 shutdown_connections 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556cc0c299c0 0x556cc0c2bde0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556cc0bb2a00 0x556cc0bb8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556cc0ba8140 0x556cc0bb8250 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 --2- 172.21.15.52:0/237740554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556cc0ba7820 0x556cc0bbaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f8790c00640 0 client.26177 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 >> 172.21.15.52:0/237740554 conn(0x556cc0ba3d30 msgr2=0x556cc0ba2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 shutdown_connections 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.575+0000 7f87b0e5d740 1 -- 172.21.15.52:0/237740554 wait complete. 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.576+0000 7f87b0e5d740 20 client.26177 trim_cache size 0 max 16384 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:53:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.576+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 msgr2=0x7f878c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.576+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 0x7f878c0a6300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x556cc0ba8dc0 tx=0x7f8798009f90 comp rx=0 tx=0).stop 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.576+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 msgr2=0x7f87a81a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.576+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9c0 tx=0x7f879800ee80 comp rx=0 tx=0).stop 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 shutdown_connections 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f878c0a3e40 0x7f878c0a6300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87a819a720 0x7f87a819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87a8109480 0x7f87a81a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 --2- 172.21.15.52:0/2028410148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87a81043b0 0x7f87a81a0660 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:53:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 >> 172.21.15.52:0/2028410148 conn(0x7f87a80ffc50 msgr2=0x7f87a8101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:53:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 shutdown_connections 2024-04-26T01:53:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:53:48.577+0000 7f87ae000640 1 -- 172.21.15.52:0/2028410148 wait complete. 2024-04-26T01:53:49.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 210402 2.74 MB/sec execute 308 sec latency 44.224 ms 2024-04-26T01:53:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:49 smithi191 ceph-mon[32745]: pgmap v272: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:49.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:49 smithi183 ceph-mon[32752]: pgmap v272: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:49 smithi052 ceph-mon[27851]: pgmap v272: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:50.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 211686 2.75 MB/sec execute 309 sec latency 20.872 ms 2024-04-26T01:53:51.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 212960 2.77 MB/sec execute 310 sec latency 24.173 ms 2024-04-26T01:53:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:51 smithi191 ceph-mon[32745]: pgmap v273: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:51.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:51 smithi183 ceph-mon[32752]: pgmap v273: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:51 smithi052 ceph-mon[27851]: pgmap v273: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 631 KiB/s wr, 40 op/s 2024-04-26T01:53:52.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 213268 2.77 MB/sec execute 311 sec latency 32.975 ms 2024-04-26T01:53:53.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 213989 2.76 MB/sec execute 312 sec latency 23.964 ms 2024-04-26T01:53:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:53 smithi191 ceph-mon[32745]: pgmap v274: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-26T01:53:53.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:53 smithi183 ceph-mon[32752]: pgmap v274: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-26T01:53:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:53 smithi052 ceph-mon[27851]: pgmap v274: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-26T01:53:54.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 215252 2.77 MB/sec execute 313 sec latency 22.899 ms 2024-04-26T01:53:55.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 216535 2.80 MB/sec execute 314 sec latency 31.343 ms 2024-04-26T01:53:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:55 smithi191 ceph-mon[32745]: pgmap v275: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 634 KiB/s wr, 95 op/s 2024-04-26T01:53:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:55 smithi183 ceph-mon[32752]: pgmap v275: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 634 KiB/s wr, 95 op/s 2024-04-26T01:53:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:55 smithi052 ceph-mon[27851]: pgmap v275: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 634 KiB/s wr, 95 op/s 2024-04-26T01:53:56.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 216858 2.79 MB/sec execute 315 sec latency 23.921 ms 2024-04-26T01:53:57.115 INFO:tasks.workunit.client.0.smithi052.stdout: 1 217564 2.79 MB/sec execute 316 sec latency 23.014 ms 2024-04-26T01:53:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:57 smithi191 ceph-mon[32745]: pgmap v276: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 996 KiB/s wr, 144 op/s 2024-04-26T01:53:57.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:57 smithi183 ceph-mon[32752]: pgmap v276: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 996 KiB/s wr, 144 op/s 2024-04-26T01:53:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:57 smithi052 ceph-mon[27851]: pgmap v276: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 996 KiB/s wr, 144 op/s 2024-04-26T01:53:58.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 218248 2.79 MB/sec execute 317 sec latency 24.543 ms 2024-04-26T01:53:59.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 219777 2.81 MB/sec execute 318 sec latency 14.985 ms 2024-04-26T01:53:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:53:59 smithi191 ceph-mon[32745]: pgmap v277: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:53:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:53:59 smithi183 ceph-mon[32752]: pgmap v277: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:53:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:53:59 smithi052 ceph-mon[27851]: pgmap v277: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:54:00.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 220343 2.82 MB/sec execute 319 sec latency 57.133 ms 2024-04-26T01:54:01.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 220343 2.81 MB/sec execute 320 sec latency 1057.232 ms 2024-04-26T01:54:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:01 smithi183 ceph-mon[32752]: pgmap v278: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:54:01.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:01 smithi052 ceph-mon[27851]: pgmap v278: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 -- 172.21.15.52:0/1254886751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100bd0 msgr2=0x7fbc08104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/1254886751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100bd0 0x7fbc08104c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009900 tx=0x7fbbf402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 -- 172.21.15.52:0/1254886751 shutdown_connections 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/1254886751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc081052f0 0x7fbc0810cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/1254886751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100bd0 0x7fbc08104c10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/1254886751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc08100220 0x7fbc08100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.723+0000 7fbc0ea00640 1 -- 172.21.15.52:0/1254886751 >> 172.21.15.52:0/1254886751 conn(0x7fbc08076db0 msgr2=0x7fbc080fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.724+0000 7fbc0ea00640 1 -- 172.21.15.52:0/1254886751 shutdown_connections 2024-04-26T01:54:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.724+0000 7fbc0ea00640 1 -- 172.21.15.52:0/1254886751 wait complete. 2024-04-26T01:54:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 Processor -- start 2024-04-26T01:54:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 -- start start 2024-04-26T01:54:01.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc081052f0 0x7fbc081a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc0819a660 0x7fbc0819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810fe20 con 0x7fbc081052f0 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810fca0 con 0x7fbc0819a660 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0ea00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810ffa0 con 0x7fbc08100220 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.725+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40034/0 (socket says 172.21.15.52:40034) 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 -- 172.21.15.52:0/588413754 learned_addr learned my addr 172.21.15.52:0/588413754 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc081052f0 0x7fbc081a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0e000640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc0819a660 0x7fbc0819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 -- 172.21.15.52:0/588413754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc0819a660 msgr2=0x7fbc0819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc0819a660 0x7fbc0819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 -- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc081052f0 msgr2=0x7fbc081a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc081052f0 0x7fbc081a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 -- 172.21.15.52:0/588413754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0819b180 con 0x7fbc08100220 2024-04-26T01:54:01.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.726+0000 7fbc0d600640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009800 tx=0x7fbbfc009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.727+0000 7fbc06000640 1 -- 172.21.15.52:0/588413754 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc018020 con 0x7fbc08100220 2024-04-26T01:54:01.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.727+0000 7fbc06000640 1 -- 172.21.15.52:0/588413754 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc014410 con 0x7fbc08100220 2024-04-26T01:54:01.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.727+0000 7fbc06000640 1 -- 172.21.15.52:0/588413754 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0135d0 con 0x7fbc08100220 2024-04-26T01:54:01.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.727+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0819b470 con 0x7fbc08100220 2024-04-26T01:54:01.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.727+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081a76c0 con 0x7fbc08100220 2024-04-26T01:54:01.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.730+0000 7fbc12fc9740 1 Processor -- start 2024-04-26T01:54:01.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.730+0000 7fbc06000640 1 -- 172.21.15.52:0/588413754 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbbfc01b020 con 0x7fbc08100220 2024-04-26T01:54:01.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.730+0000 7fbc12fc9740 1 -- start start 2024-04-26T01:54:01.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd860cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 0x564dd8619f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564dd8572900 con 0x564dd860c9b0 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564dd8518ef0 con 0x564dd8617b90 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564dd856e8c0 con 0x564dd860d2d0 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40042/0 (socket says 172.21.15.52:40042) 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc0d600640 1 -- 172.21.15.52:0/3304045796 learned_addr learned my addr 172.21.15.52:0/3304045796 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc0e000640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd860cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.731+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 0x564dd8619f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 -- 172.21.15.52:0/3304045796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 msgr2=0x564dd8619f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 0x564dd8619f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 -- 172.21.15.52:0/3304045796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 msgr2=0x564dd860cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd860cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 -- 172.21.15.52:0/3304045796 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dd8285210 con 0x564dd860d2d0 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 0x564dd8619f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc06000640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 0x7fbbf00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0d600640 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cdb20 tx=0x7fbbfc0d6ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c15d6783b0fb8d58 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc06000640 1 -- 172.21.15.52:0/588413754 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7fbbfc0d04f0 con 0x7fbc08100220 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbc0e000640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 0x7fbbf00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbbfa600640 1 -- 172.21.15.52:0/3304045796 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0986d0 con 0x564dd860d2d0 2024-04-26T01:54:01.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbbfa600640 1 -- 172.21.15.52:0/3304045796 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc09cb30 con 0x564dd860d2d0 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.732+0000 7fbbfa600640 1 -- 172.21.15.52:0/3304045796 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc09bd90 con 0x564dd860d2d0 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 -- 172.21.15.52:0/3304045796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 msgr2=0x564dd8617650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cdb20 tx=0x7fbbfc0d6ac0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc0e000640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 0x7fbbf00a6250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005fd0 tx=0x7fbc00009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 -- 172.21.15.52:0/3304045796 shutdown_connections 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd8617b90 0x564dd8619f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860d2d0 0x564dd8617650 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/3304045796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd860cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 -- 172.21.15.52:0/3304045796 >> 172.21.15.52:0/3304045796 conn(0x564dd8608ec0 msgr2=0x564dd86092a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 -- 172.21.15.52:0/3304045796 shutdown_connections 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.733+0000 7fbc12fc9740 1 -- 172.21.15.52:0/3304045796 wait complete. 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.734+0000 7fbc12fc9740 1 -- start start 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.734+0000 7fbc12fc9740 1 Processor -- start 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd8615020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564dd861a760 con 0x564dd860c9b0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564dd852c310 con 0x564dd860d2d0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564dd861a8e0 con 0x564dd8617b90 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc12fc9740 10 client.0 _ll_register_callbacks cb 0x564dd84facb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:40052/0 (socket says 172.21.15.52:40052) 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49152/0 (socket says 172.21.15.52:49152) 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0e000640 1 -- 172.21.15.52:0/1407139773 learned_addr learned my addr 172.21.15.52:0/1407139773 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd8615020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 -- 172.21.15.52:0/1407139773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 msgr2=0x564dd8615560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 -- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 msgr2=0x564dd8615020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd8615020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0d600640 1 -- 172.21.15.52:0/1407139773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dd8351620 con 0x564dd860c9b0 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.735+0000 7fbc0e000640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbc0d600640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc014800 tx=0x7fbbfc00a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ace9256d4d0c4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbbeea00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc09bf70 con 0x564dd860c9b0 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7fbbfc0d6150 con 0x564dd860c9b0 2024-04-26T01:54:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc09cb30 con 0x564dd860c9b0 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbbfc09c200 con 0x564dd860c9b0 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbc12fc9740 10 client.16071 fetch_fsmap learned FSMap version 38 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564dd861a760 con 0x564dd860c9b0 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.736+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc09cb30 con 0x564dd860c9b0 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fbbfc0cde60 con 0x564dd860c9b0 2024-04-26T01:54:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 10 client.16071 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 20 client.16071 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8627af0 0x564dd8629ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 4 client.16071 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dd85868b0 con 0x564dd8627af0 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.737+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8627af0 0x564dd8629ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.738+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8627af0 0x564dd8629ee0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dbe8973f96c7478 server_cookie=13636f877cd0e935 in_seq=0 out_seq=0 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.738+0000 7fbbeea00640 10 client.16071 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbbeea00640 1 -- 172.21.15.52:0/1407139773 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564dd85868b0 con 0x564dd8627af0 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbbeea00640 10 client.16071 handle_command_reply: tid=0 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 client.16071 shutdown 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 msgr2=0x564dd8614ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc014800 tx=0x7fbbfc00a300 comp rx=0 tx=0).stop 2024-04-26T01:54:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 shutdown_connections 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8627af0 0x564dd8629ee0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8617b90 0x564dd8615560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd8615020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/1407139773 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8614ae0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbbeea00640 0 client.16071 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 >> 172.21.15.52:0/1407139773 conn(0x564dd8608ec0 msgr2=0x564dd860ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.739+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 shutdown_connections 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.740+0000 7fbc12fc9740 1 -- 172.21.15.52:0/1407139773 wait complete. 2024-04-26T01:54:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.740+0000 7fbc12fc9740 20 client.16071 trim_cache size 0 max 16384 2024-04-26T01:54:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.756+0000 7fbc12fc9740 1 Processor -- start 2024-04-26T01:54:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.756+0000 7fbc12fc9740 1 -- start start 2024-04-26T01:54:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860c9b0 0x564dd860f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd86092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081b2580 con 0x564dd8608ec0 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081afe50 con 0x564dd860d2d0 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081adcc0 con 0x564dd860c9b0 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49162/0 (socket says 172.21.15.52:49162) 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 -- 172.21.15.52:0/2185019489 learned_addr learned my addr 172.21.15.52:0/2185019489 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860c9b0 0x564dd860f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0e000640 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd86092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 -- 172.21.15.52:0/2185019489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860c9b0 msgr2=0x564dd860f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860c9b0 0x564dd860f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 -- 172.21.15.52:0/2185019489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 msgr2=0x564dd86092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd86092a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.757+0000 7fbc0d600640 1 -- 172.21.15.52:0/2185019489 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dd860f8b0 con 0x564dd8608ec0 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.758+0000 7fbc0d600640 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc09c130 tx=0x7fbbfc014bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d5355355ec813a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.758+0000 7fbbf9c00640 1 -- 172.21.15.52:0/2185019489 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc09bd90 con 0x564dd8608ec0 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.758+0000 7fbbf9c00640 1 -- 172.21.15.52:0/2185019489 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc0d3d70 con 0x564dd8608ec0 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.758+0000 7fbbf9c00640 1 -- 172.21.15.52:0/2185019489 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0d6150 con 0x564dd8608ec0 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- 172.21.15.52:0/2185019489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 msgr2=0x564dd8623620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc09c130 tx=0x7fbbfc014bf0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- 172.21.15.52:0/2185019489 shutdown_connections 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd86092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd860c9b0 0x564dd860f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/2185019489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd8608ec0 0x564dd8623620 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- 172.21.15.52:0/2185019489 >> 172.21.15.52:0/2185019489 conn(0x564dd86104b0 msgr2=0x564dd8616ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- 172.21.15.52:0/2185019489 shutdown_connections 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- 172.21.15.52:0/2185019489 wait complete. 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.759+0000 7fbc12fc9740 1 -- start start 2024-04-26T01:54:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 Processor -- start 2024-04-26T01:54:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 0x564dd861faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd861d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081ae310 con 0x564dd860c9b0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081af780 con 0x564dd860d2d0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081ae9e0 con 0x564dd8608ec0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc12fc9740 10 client.0 _ll_register_callbacks cb 0x7fbbfc00af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc0cc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc0cc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49168/0 (socket says 172.21.15.52:49168) 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc0cc00640 1 -- 172.21.15.52:0/928182467 learned_addr learned my addr 172.21.15.52:0/928182467 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.760+0000 7fbc0d600640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 0x564dd861faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0e000640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd861d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 -- 172.21.15.52:0/928182467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 msgr2=0x564dd861faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 0x564dd861faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 -- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 msgr2=0x564dd861d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd861d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 -- 172.21.15.52:0/928182467 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dd861d8d0 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0d600640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 0x564dd861faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbc0cc00640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402a9b0 tx=0x7fbbf402aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d2bb3024b50a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbbeea00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4035040 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.761+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8521+0+0 (secure 0 0 0) 0x7fbbf4005140 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc0d3d70 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbbf4005340 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 10 client.16077 fetch_fsmap learned FSMap version 38 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564dd868da10 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc0d3d70 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7fbbf4004980 con 0x564dd860c9b0 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 10 client.16077 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:54:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 20 client.16077 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8617b90 0x564dd8690180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 4 client.16077 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.762+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564dd8580320 con 0x564dd8617b90 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.763+0000 7fbc0d600640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8617b90 0x564dd8690180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.763+0000 7fbc0d600640 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8617b90 0x564dd8690180 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a589abbb4ccb416 server_cookie=71750457ad10d84 in_seq=0 out_seq=0 2024-04-26T01:54:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.763+0000 7fbbeea00640 10 client.16077 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbbeea00640 1 -- 172.21.15.52:0/928182467 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564dd8580320 con 0x564dd8617b90 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbbeea00640 10 client.16077 handle_command_reply: tid=0 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 client.16077 shutdown 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 msgr2=0x564dd8620030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402a9b0 tx=0x7fbbf402aa70 comp rx=0 tx=0).stop 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 shutdown_connections 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564dd8617b90 0x564dd8690180 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564dd860d2d0 0x564dd861d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564dd860c9b0 0x564dd8620030 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 --2- 172.21.15.52:0/928182467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564dd8608ec0 0x564dd861faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbbeea00640 0 client.16077 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 >> 172.21.15.52:0/928182467 conn(0x564dd86104b0 msgr2=0x564dd8613b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 shutdown_connections 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.803+0000 7fbc12fc9740 1 -- 172.21.15.52:0/928182467 wait complete. 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.804+0000 7fbc12fc9740 20 client.16077 trim_cache size 0 max 16384 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:54:01.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.804+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 msgr2=0x7fbbf00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.804+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 0x7fbbf00a6250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005fd0 tx=0x7fbc00009500 comp rx=0 tx=0).stop 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.804+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 msgr2=0x7fbc081a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.804+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009800 tx=0x7fbbfc009cc0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 shutdown_connections 2024-04-26T01:54:01.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbbf00a3d90 0x7fbbf00a6250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc0819a660 0x7fbc0819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc081052f0 0x7fbc081a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 --2- 172.21.15.52:0/588413754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc08100220 0x7fbc081a0460 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 >> 172.21.15.52:0/588413754 conn(0x7fbc08076db0 msgr2=0x7fbc081032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 shutdown_connections 2024-04-26T01:54:01.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:01.805+0000 7fbc0ea00640 1 -- 172.21.15.52:0/588413754 wait complete. 2024-04-26T01:54:01.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:01 smithi191 ceph-mon[32745]: pgmap v278: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 KiB/s wr, 115 op/s 2024-04-26T01:54:02.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 220343 2.80 MB/sec execute 321 sec latency 2057.378 ms 2024-04-26T01:54:03.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 220355 2.79 MB/sec execute 322 sec latency 3003.384 ms 2024-04-26T01:54:03.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:03 smithi183 ceph-mon[32752]: pgmap v279: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-04-26T01:54:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:03 smithi052 ceph-mon[27851]: pgmap v279: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-04-26T01:54:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:03 smithi191 ceph-mon[32745]: pgmap v279: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-04-26T01:54:04.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 220973 2.79 MB/sec execute 323 sec latency 22.380 ms 2024-04-26T01:54:05.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 221390 2.79 MB/sec execute 324 sec latency 57.466 ms 2024-04-26T01:54:05.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:05 smithi183 ceph-mon[32752]: pgmap v280: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 673 KiB/s wr, 89 op/s 2024-04-26T01:54:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:05 smithi052 ceph-mon[27851]: pgmap v280: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 673 KiB/s wr, 89 op/s 2024-04-26T01:54:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:05 smithi191 ceph-mon[32745]: pgmap v280: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 673 KiB/s wr, 89 op/s 2024-04-26T01:54:06.116 INFO:tasks.workunit.client.0.smithi052.stdout: 1 222946 2.80 MB/sec execute 325 sec latency 19.138 ms 2024-04-26T01:54:07.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 223864 2.82 MB/sec execute 326 sec latency 22.010 ms 2024-04-26T01:54:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:07 smithi052 ceph-mon[27851]: pgmap v281: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-04-26T01:54:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:07 smithi052 ceph-mon[27851]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:54:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:07 smithi191 ceph-mon[32745]: pgmap v281: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-04-26T01:54:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:07 smithi191 ceph-mon[32745]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:54:07.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:07 smithi183 ceph-mon[32752]: pgmap v281: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 172 op/s 2024-04-26T01:54:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:07 smithi183 ceph-mon[32752]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T01:54:08.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 224387 2.81 MB/sec execute 327 sec latency 21.062 ms 2024-04-26T01:54:09.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 225132 2.81 MB/sec execute 328 sec latency 19.868 ms 2024-04-26T01:54:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:09 smithi052 ceph-mon[27851]: pgmap v283: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:09.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:09 smithi191 ceph-mon[32745]: pgmap v283: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:09.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:09 smithi183 ceph-mon[32752]: pgmap v283: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:10.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 226282 2.82 MB/sec execute 329 sec latency 43.337 ms 2024-04-26T01:54:11.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 227321 2.84 MB/sec execute 330 sec latency 29.949 ms 2024-04-26T01:54:11.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:11 smithi052 ceph-mon[27851]: pgmap v284: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:11.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:11 smithi191 ceph-mon[32745]: pgmap v284: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:11.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:11 smithi183 ceph-mon[32752]: pgmap v284: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 847 KiB/s wr, 148 op/s 2024-04-26T01:54:12.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 227733 2.83 MB/sec execute 331 sec latency 19.305 ms 2024-04-26T01:54:13.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 228565 2.84 MB/sec execute 332 sec latency 24.550 ms 2024-04-26T01:54:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:13 smithi052 ceph-mon[27851]: pgmap v285: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 966 KiB/s wr, 154 op/s 2024-04-26T01:54:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:13 smithi052 ceph-mon[27851]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:54:13.725 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:13 smithi183 ceph-mon[32752]: pgmap v285: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 966 KiB/s wr, 154 op/s 2024-04-26T01:54:13.725 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:13 smithi183 ceph-mon[32752]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:54:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:13 smithi191 ceph-mon[32745]: pgmap v285: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 966 KiB/s wr, 154 op/s 2024-04-26T01:54:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:13 smithi191 ceph-mon[32745]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T01:54:14.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 229527 2.84 MB/sec execute 333 sec latency 21.339 ms 2024-04-26T01:54:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 -- 172.21.15.52:0/429726911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 msgr2=0x7f87bc110e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 --2- 172.21.15.52:0/429726911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc110e80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009ca0 tx=0x7f87b402a8c0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 -- 172.21.15.52:0/429726911 shutdown_connections 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 --2- 172.21.15.52:0/429726911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc110e80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 --2- 172.21.15.52:0/429726911 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc104b90 0x7f87bc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 --2- 172.21.15.52:0/429726911 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 -- 172.21.15.52:0/429726911 >> 172.21.15.52:0/429726911 conn(0x7f87bc0ffa60 msgr2=0x7f87bc101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 -- 172.21.15.52:0/429726911 shutdown_connections 2024-04-26T01:54:14.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.965+0000 7f87c2a00640 1 -- 172.21.15.52:0/429726911 wait complete. 2024-04-26T01:54:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.966+0000 7f87c2a00640 1 Processor -- start 2024-04-26T01:54:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 -- start start 2024-04-26T01:54:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc19a5b0 0x7f87bc19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc114250 con 0x7f87bc19a5b0 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1140d0 con 0x7f87bc1041e0 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1143d0 con 0x7f87bc1092b0 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c1600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c1600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38704/0 (socket says 172.21.15.52:38704) 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c1600640 1 -- 172.21.15.52:0/2768452234 learned_addr learned my addr 172.21.15.52:0/2768452234 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c0c00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc1a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.967+0000 7f87c2000640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc19a5b0 0x7f87bc19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 -- 172.21.15.52:0/2768452234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 msgr2=0x7f87bc1a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc1a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 -- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc19a5b0 msgr2=0x7f87bc19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc19a5b0 0x7f87bc19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 -- 172.21.15.52:0/2768452234 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc19b0d0 con 0x7f87bc1041e0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87c1600640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f87b000e9f0 tx=0x7f87b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.968+0000 7f87ba000640 1 -- 172.21.15.52:0/2768452234 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b000cd80 con 0x7f87bc1041e0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.969+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc19b3c0 con 0x7f87bc1041e0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.969+0000 7f87ba000640 1 -- 172.21.15.52:0/2768452234 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b00108e0 con 0x7f87bc1041e0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.969+0000 7f87ba000640 1 -- 172.21.15.52:0/2768452234 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0018980 con 0x7f87bc1041e0 2024-04-26T01:54:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.969+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc1a7660 con 0x7f87bc1041e0 2024-04-26T01:54:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.972+0000 7f87ba000640 1 -- 172.21.15.52:0/2768452234 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f87b0010430 con 0x7f87bc1041e0 2024-04-26T01:54:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.972+0000 7f87c6f7f740 1 Processor -- start 2024-04-26T01:54:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 -- start start 2024-04-26T01:54:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8980f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ca88d9900 con 0x556ca89742d0 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556ca887fef0 con 0x556ca89739b0 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556ca88d58c0 con 0x556ca897eb90 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38714/0 (socket says 172.21.15.52:38714) 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.973+0000 7f87c2000640 1 -- 172.21.15.52:0/545774137 learned_addr learned my addr 172.21.15.52:0/545774137 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c0c00640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8980f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c1600640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 -- 172.21.15.52:0/545774137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 msgr2=0x556ca8980f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8980f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 -- 172.21.15.52:0/545774137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 msgr2=0x556ca897e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 -- 172.21.15.52:0/545774137 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca85ec210 con 0x556ca89739b0 2024-04-26T01:54:14.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c1600640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:14.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.974+0000 7f87c2000640 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009ca0 tx=0x7f87b40093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc4bf7b34defe1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:14.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87ba000640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 0x7f87a40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87ae600640 1 -- 172.21.15.52:0/545774137 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4031cf0 con 0x556ca89739b0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87ae600640 1 -- 172.21.15.52:0/545774137 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b4036030 con 0x556ca89739b0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87ae600640 1 -- 172.21.15.52:0/545774137 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b40042a0 con 0x556ca89739b0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87c1600640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 0x7f87a40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.975+0000 7f87ba000640 1 -- 172.21.15.52:0/2768452234 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8521+0+0 (secure 0 0 0) 0x7f87b00d0b40 con 0x7f87bc1041e0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 -- 172.21.15.52:0/545774137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 msgr2=0x556ca8973d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009ca0 tx=0x7f87b40093e0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c1600640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 0x7f87a40a61d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x556ca8974f50 tx=0x7f87b0000bc0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 -- 172.21.15.52:0/545774137 shutdown_connections 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8980f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/545774137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8973d90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 -- 172.21.15.52:0/545774137 >> 172.21.15.52:0/545774137 conn(0x556ca896fec0 msgr2=0x556ca89702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 -- 172.21.15.52:0/545774137 shutdown_connections 2024-04-26T01:54:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.976+0000 7f87c6f7f740 1 -- 172.21.15.52:0/545774137 wait complete. 2024-04-26T01:54:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.977+0000 7f87c6f7f740 1 -- start start 2024-04-26T01:54:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.977+0000 7f87c6f7f740 1 Processor -- start 2024-04-26T01:54:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca897c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ca89815e0 con 0x556ca89742d0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556ca8893310 con 0x556ca89739b0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556ca8981760 con 0x556ca897eb90 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c6f7f740 10 client.0 _ll_register_callbacks cb 0x556ca8861cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38718/0 (socket says 172.21.15.52:38718) 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c2000640 1 -- 172.21.15.52:0/1789183102 learned_addr learned my addr 172.21.15.52:0/1789183102 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c2000640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca897c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 -- 172.21.15.52:0/1789183102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 msgr2=0x556ca897c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca897c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 -- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 msgr2=0x556ca897bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:14.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c0c00640 1 -- 172.21.15.52:0/1789183102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca86b8620 con 0x556ca89742d0 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.978+0000 7f87c2000640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.979+0000 7f87c0c00640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897c0b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x556ca8971050 tx=0x7f87a8012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1705a4fc0b0efe1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.979+0000 7f87a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.979+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8004030 con 0x556ca89742d0 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.979+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8521+0+0 (secure 0 0 0) 0x7f87a80041d0 con 0x556ca89742d0 2024-04-26T01:54:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.979+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b4036030 con 0x556ca89742d0 2024-04-26T01:54:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.980+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f87a80043d0 con 0x556ca89742d0 2024-04-26T01:54:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.980+0000 7f87c6f7f740 10 client.16089 fetch_fsmap learned FSMap version 38 2024-04-26T01:54:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.980+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556ca898d8f0 con 0x556ca89742d0 2024-04-26T01:54:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.980+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b4036030 con 0x556ca89742d0 2024-04-26T01:54:14.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.980+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f87a8003a30 con 0x556ca89742d0 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 10 client.16089 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 20 client.16089 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca898ed60 0x556ca8991150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 4 client.16089 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ca88ed8b0 con 0x556ca898ed60 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c2000640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca898ed60 0x556ca8991150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87c2000640 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca898ed60 0x556ca8991150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0591d6613d07390 server_cookie=d417bc6bf44b2543 in_seq=0 out_seq=0 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.981+0000 7f87a2a00640 10 client.16089 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.982+0000 7f87a2a00640 1 -- 172.21.15.52:0/1789183102 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ca88ed8b0 con 0x556ca898ed60 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.982+0000 7f87a2a00640 10 client.16089 handle_command_reply: tid=0 2024-04-26T01:54:14.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.982+0000 7f87c6f7f740 1 client.16089 shutdown 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 msgr2=0x556ca897c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897c0b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x556ca8971050 tx=0x7f87a8012b60 comp rx=0 tx=0).stop 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 shutdown_connections 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca898ed60 0x556ca8991150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca897c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca89742d0 0x556ca897c0b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/1789183102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca897bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:14.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87a2a00640 0 client.16089 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:14.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 >> 172.21.15.52:0/1789183102 conn(0x556ca896fec0 msgr2=0x556ca8975c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:14.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 shutdown_connections 2024-04-26T01:54:14.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 1 -- 172.21.15.52:0/1789183102 wait complete. 2024-04-26T01:54:14.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:14.983+0000 7f87c6f7f740 20 client.16089 trim_cache size 0 max 16384 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 Processor -- start 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 -- start start 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca898a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 0x556ca8996f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1b2510 con 0x556ca896f7a0 2024-04-26T01:54:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1afde0 con 0x556ca89739b0 2024-04-26T01:54:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1adc50 con 0x556ca897eb90 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca898a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60642/0 (socket says 172.21.15.52:60642) 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c0c00640 1 -- 172.21.15.52:0/2116995905 learned_addr learned my addr 172.21.15.52:0/2116995905 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.002+0000 7f87c2000640 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 0x556ca8996f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 -- 172.21.15.52:0/2116995905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 msgr2=0x556ca898a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca898a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 -- 172.21.15.52:0/2116995905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 msgr2=0x556ca8996f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 0x556ca8996f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 -- 172.21.15.52:0/2116995905 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca896b820 con 0x556ca897eb90 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c2000640 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 0x556ca8996f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87c0c00640 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x556ca8971050 tx=0x7f87a80123a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbdd61f23720878e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87adc00640 1 -- 172.21.15.52:0/2116995905 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a80255a0 con 0x556ca897eb90 2024-04-26T01:54:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87adc00640 1 -- 172.21.15.52:0/2116995905 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a800fda0 con 0x556ca897eb90 2024-04-26T01:54:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.003+0000 7f87adc00640 1 -- 172.21.15.52:0/2116995905 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8025d80 con 0x556ca897eb90 2024-04-26T01:54:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 -- 172.21.15.52:0/2116995905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 msgr2=0x556ca8996a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x556ca8971050 tx=0x7f87a80123a0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 -- 172.21.15.52:0/2116995905 shutdown_connections 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca896f7a0 0x556ca8996f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca897eb90 0x556ca8996a20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/2116995905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca898a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 -- 172.21.15.52:0/2116995905 >> 172.21.15.52:0/2116995905 conn(0x556ca896fec0 msgr2=0x556ca897dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 -- 172.21.15.52:0/2116995905 shutdown_connections 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.004+0000 7f87c6f7f740 1 -- 172.21.15.52:0/2116995905 wait complete. 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.005+0000 7f87c6f7f740 1 -- start start 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.005+0000 7f87c6f7f740 1 Processor -- start 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8986dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca89846a0 0x556ca8984b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1ae2a0 con 0x556ca897eb90 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1af710 con 0x556ca89739b0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1ae970 con 0x556ca89846a0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c6f7f740 10 client.0 _ll_register_callbacks cb 0x7f87a8009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8986dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43104/0 (socket says 172.21.15.52:43104) 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 -- 172.21.15.52:0/3326476054 learned_addr learned my addr 172.21.15.52:0/3326476054 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c2000640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca89846a0 0x556ca8984b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 -- 172.21.15.52:0/3326476054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca89846a0 msgr2=0x556ca8984b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca89846a0 0x556ca8984b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 -- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 msgr2=0x556ca8986dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8986dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c0c00640 1 -- 172.21.15.52:0/3326476054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca89f4450 con 0x556ca897eb90 2024-04-26T01:54:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.006+0000 7f87c2000640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8986dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87c0c00640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87a8000c00 tx=0x7f87a8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ce3737fddeed6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8009e90 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8521+0+0 (secure 0 0 0) 0x7f87a800fb00 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a800fda0 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f87a8025a00 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87c6f7f740 10 client.16095 fetch_fsmap learned FSMap version 38 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.007+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556ca89f56e0 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a800fda0 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 38) ==== 8342+0+0 (secure 0 0 0) 0x7f87a80050d0 con 0x556ca897eb90 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 10 client.16095 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T01:54:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 10 client.16095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 20 client.16095 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca89f5d90 0x556ca89f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 4 client.16095 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ca88e7320 con 0x556ca89f5d90 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.008+0000 7f87c2000640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca89f5d90 0x556ca89f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.009+0000 7f87c2000640 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca89f5d90 0x556ca89f81b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a6689ba1a41ffb server_cookie=b26275a0f2aa27b5 in_seq=0 out_seq=0 2024-04-26T01:54:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.009+0000 7f87a2a00640 10 client.16095 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87a2a00640 1 -- 172.21.15.52:0/3326476054 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ca88e7320 con 0x556ca89f5d90 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87a2a00640 10 client.16095 handle_command_reply: tid=0 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 client.16095 shutdown 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 msgr2=0x556ca8984160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87a8000c00 tx=0x7f87a8004360 comp rx=0 tx=0).stop 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 shutdown_connections 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556ca89f5d90 0x556ca89f81b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ca89846a0 0x556ca8984b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ca897eb90 0x556ca8984160 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 --2- 172.21.15.52:0/3326476054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556ca89739b0 0x556ca8986dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87a2a00640 0 client.16095 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:15.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 >> 172.21.15.52:0/3326476054 conn(0x556ca896fec0 msgr2=0x556ca897bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 shutdown_connections 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.059+0000 7f87c6f7f740 1 -- 172.21.15.52:0/3326476054 wait complete. 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.060+0000 7f87c6f7f740 20 client.16095 trim_cache size 0 max 16384 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:54:15.059 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.060+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 msgr2=0x7f87a40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.060+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 0x7f87a40a61d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x556ca8974f50 tx=0x7f87b0000bc0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.060+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 msgr2=0x7f87bc1a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.060+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f87b000e9f0 tx=0x7f87b000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 shutdown_connections 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a40a3d10 0x7f87a40a61d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87bc19a5b0 0x7f87bc19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87bc1092b0 0x7f87bc1a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 --2- 172.21.15.52:0/2768452234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87bc1041e0 0x7f87bc1a0400 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 >> 172.21.15.52:0/2768452234 conn(0x7f87bc0ffa60 msgr2=0x7f87bc101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 shutdown_connections 2024-04-26T01:54:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:15.061+0000 7f87c2a00640 1 -- 172.21.15.52:0/2768452234 wait complete. 2024-04-26T01:54:15.117 INFO:tasks.workunit.client.0.smithi052.stdout: 1 230824 2.86 MB/sec execute 334 sec latency 42.652 ms 2024-04-26T01:54:15.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:15 smithi052 ceph-mon[27851]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 616 KiB/s wr, 69 op/s 2024-04-26T01:54:15.883 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:15 smithi183 ceph-mon[32752]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 616 KiB/s wr, 69 op/s 2024-04-26T01:54:15.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:15 smithi191 ceph-mon[32745]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 616 KiB/s wr, 69 op/s 2024-04-26T01:54:16.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231147 2.85 MB/sec execute 335 sec latency 21.845 ms 2024-04-26T01:54:16.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T01:54:16.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T01:54:16.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:15 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/3236117500 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555adba89a40 con 0x555adba5d000 2024-04-26T01:54:16.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T01:54:16.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:16 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T01:54:16.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:16 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T01:54:16.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:16 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T01:54:17.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.85 MB/sec execute 336 sec latency 883.177 ms 2024-04-26T01:54:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: pgmap v288: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-26T01:54:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:54:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/2011820441,v1:172.21.15.183:6837/2011820441] up:boot 2024-04-26T01:54:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:17.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:17 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: pgmap v288: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-26T01:54:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:54:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/2011820441,v1:172.21.15.183:6837/2011820441] up:boot 2024-04-26T01:54:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:17 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: pgmap v288: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/2011820441,v1:172.21.15.183:6837/2011820441] up:boot 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:17 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:18.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.84 MB/sec execute 337 sec latency 1883.289 ms 2024-04-26T01:54:19.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.83 MB/sec execute 338 sec latency 2883.397 ms 2024-04-26T01:54:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:19 smithi052 ceph-mon[27851]: pgmap v290: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-26T01:54:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:19 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:19 smithi191 ceph-mon[32745]: pgmap v290: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-26T01:54:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:19 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:19 smithi183 ceph-mon[32752]: pgmap v290: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-26T01:54:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:19 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:20.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.82 MB/sec execute 339 sec latency 3883.480 ms 2024-04-26T01:54:21.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.81 MB/sec execute 340 sec latency 4883.612 ms 2024-04-26T01:54:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:21 smithi191 ceph-mon[32745]: pgmap v291: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 771 KiB/s wr, 113 op/s 2024-04-26T01:54:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:21 smithi183 ceph-mon[32752]: pgmap v291: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 771 KiB/s wr, 113 op/s 2024-04-26T01:54:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:21 smithi052 ceph-mon[27851]: pgmap v291: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 771 KiB/s wr, 113 op/s 2024-04-26T01:54:22.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231200 2.80 MB/sec execute 341 sec latency 5883.751 ms 2024-04-26T01:54:23.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 231427 2.80 MB/sec execute 342 sec latency 6618.852 ms 2024-04-26T01:54:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:23 smithi191 ceph-mon[32745]: pgmap v292: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.6 MiB/s wr, 128 op/s 2024-04-26T01:54:23.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:23 smithi183 ceph-mon[32752]: pgmap v292: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.6 MiB/s wr, 128 op/s 2024-04-26T01:54:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:23 smithi052 ceph-mon[27851]: pgmap v292: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.6 MiB/s wr, 128 op/s 2024-04-26T01:54:24.118 INFO:tasks.workunit.client.0.smithi052.stdout: 1 232412 2.80 MB/sec execute 343 sec latency 22.703 ms 2024-04-26T01:54:24.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:24 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:24.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:24 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:24.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:24 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:54:25.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 234113 2.83 MB/sec execute 344 sec latency 20.475 ms 2024-04-26T01:54:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:25 smithi191 ceph-mon[32745]: pgmap v293: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:54:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:25 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:25 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:25 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:54:25.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:25 smithi183 ceph-mon[32752]: pgmap v293: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:54:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:25 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:25 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:25 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:54:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:25 smithi052 ceph-mon[27851]: pgmap v293: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-26T01:54:25.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:25 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:54:25.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:25 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:54:25.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:25 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:54:26.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 234542 2.83 MB/sec execute 345 sec latency 32.939 ms 2024-04-26T01:54:27.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 235094 2.82 MB/sec execute 346 sec latency 24.652 ms 2024-04-26T01:54:27.408 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:54:27.408 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.1 2024-04-26T01:54:27.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:54:27.897 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:27.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:27 smithi191 ceph-mon[32745]: pgmap v294: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:27.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:27 smithi183 ceph-mon[32752]: pgmap v294: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:27.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:27 smithi052 ceph-mon[27851]: pgmap v294: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:28.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 236126 2.83 MB/sec execute 347 sec latency 32.226 ms 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 -- 172.21.15.52:0/1923879764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c941014e0 msgr2=0x7f6c9410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 --2- 172.21.15.52:0/1923879764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c941014e0 0x7f6c9410e1a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009900 tx=0x7f6c7c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 -- 172.21.15.52:0/1923879764 shutdown_connections 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 --2- 172.21.15.52:0/1923879764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 0x7f6c94110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 --2- 172.21.15.52:0/1923879764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c941014e0 0x7f6c9410e1a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 --2- 172.21.15.52:0/1923879764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c94100b30 0x7f6c94100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 -- 172.21.15.52:0/1923879764 >> 172.21.15.52:0/1923879764 conn(0x7f6c940fc790 msgr2=0x7f6c940febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 -- 172.21.15.52:0/1923879764 shutdown_connections 2024-04-26T01:54:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.216+0000 7f6c93200640 1 -- 172.21.15.52:0/1923879764 wait complete. 2024-04-26T01:54:28.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.217+0000 7f6c93200640 1 Processor -- start 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.217+0000 7f6c93200640 1 -- start start 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 0x7f6c941a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 0x7f6c9419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6c94114270 con 0x7f6c9410e6e0 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941140f0 con 0x7f6c94100b30 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c93200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941143f0 con 0x7f6c941014e0 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c91e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c89400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 0x7f6c941a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c91e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60298/0 (socket says 172.21.15.52:60298) 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c91e00640 1 -- 172.21.15.52:0/1401205674 learned_addr learned my addr 172.21.15.52:0/1401205674 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.218+0000 7f6c92800640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 0x7f6c9419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 -- 172.21.15.52:0/1401205674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 msgr2=0x7f6c941a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 0x7f6c941a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 -- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 msgr2=0x7f6c9419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 0x7f6c9419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 -- 172.21.15.52:0/1401205674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9419ad70 con 0x7f6c94100b30 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c89400640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 0x7f6c941a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c91e00640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400e9f0 tx=0x7f6c8400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c8b400640 1 -- 172.21.15.52:0/1401205674 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8400cd80 con 0x7f6c94100b30 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c8b400640 1 -- 172.21.15.52:0/1401205674 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c840108e0 con 0x7f6c94100b30 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c8b400640 1 -- 172.21.15.52:0/1401205674 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84018980 con 0x7f6c94100b30 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9419b060 con 0x7f6c94100b30 2024-04-26T01:54:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.219+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c941a7300 con 0x7f6c94100b30 2024-04-26T01:54:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.223+0000 7f6c8b400640 1 -- 172.21.15.52:0/1401205674 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6c84010430 con 0x7f6c94100b30 2024-04-26T01:54:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.223+0000 7f6c9c16f740 1 Processor -- start 2024-04-26T01:54:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.223+0000 7f6c9c16f740 1 -- start start 2024-04-26T01:54:28.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1bbad2900 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f1bba78ef0 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f1bbace8c0 con 0x55f1bbb77b90 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c92800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c92800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60310/0 (socket says 172.21.15.52:60310) 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c92800640 1 -- 172.21.15.52:0/3678284559 learned_addr learned my addr 172.21.15.52:0/3678284559 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c91e00640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.224+0000 7f6c89400640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 -- 172.21.15.52:0/3678284559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 msgr2=0x55f1bbb79f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb79f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 -- 172.21.15.52:0/3678284559 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 msgr2=0x55f1bbb77650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb77650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 -- 172.21.15.52:0/3678284559 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1bb7e5210 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c91e00640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c92800640 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009ec0 tx=0x7f6c8c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e737b170574d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c83400640 1 -- 172.21.15.52:0/3678284559 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c004d20 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c83400640 1 -- 172.21.15.52:0/3678284559 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c003e20 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c83400640 1 -- 172.21.15.52:0/3678284559 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c005020 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.225+0000 7f6c8b400640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 0x7f6c780a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c8b400640 1 -- 172.21.15.52:0/1401205674 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8477+0+0 (secure 0 0 0) 0x7f6c840d0bb0 con 0x7f6c94100b30 2024-04-26T01:54:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c91e00640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 0x7f6c780a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3678284559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 msgr2=0x55f1bbb6cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009ec0 tx=0x7f6c8c0107a0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3678284559 shutdown_connections 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3678284559 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb6cd90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3678284559 >> 172.21.15.52:0/3678284559 conn(0x55f1bbb68ec0 msgr2=0x55f1bbb692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c91e00640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 0x7f6c780a6250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55f1bbb6df50 tx=0x7f6c84000bc0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.226+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3678284559 shutdown_connections 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.227+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3678284559 wait complete. 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.227+0000 7f6c9c16f740 1 -- start start 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.227+0000 7f6c9c16f740 1 Processor -- start 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb74b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb75040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb75580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1bbb7a7c0 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f1bba8c310 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f1bbb7a940 con 0x55f1bbb77b90 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c9c16f740 10 client.0 _ll_register_callbacks cb 0x55f1bba5acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c92800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb74b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c89400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb75040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c92800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb74b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39538/0 (socket says 172.21.15.52:39538) 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c92800640 1 -- 172.21.15.52:0/3829393305 learned_addr learned my addr 172.21.15.52:0/3829393305 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.228+0000 7f6c92800640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb75580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 -- 172.21.15.52:0/3829393305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 msgr2=0x55f1bbb75580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb75580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 -- 172.21.15.52:0/3829393305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 msgr2=0x55f1bbb74b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb74b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 -- 172.21.15.52:0/3829393305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1bb8b1620 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c89400640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb75040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c02a910 tx=0x7f6c7c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bfd9c265e383b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c77e00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:54:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c035040 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8477+0+0 (secure 0 0 0) 0x7f6c7c004770 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.229+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c003e20 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6c7c004970 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c9c16f740 10 client.26231 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f1bbb86a00 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c003e20 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f6c7c0314a0 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c9c16f740 10 client.26231 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.230+0000 7f6c9c16f740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.231+0000 7f6c9c16f740 20 client.26231 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.231+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbb87e70 0x55f1bbb8a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.231+0000 7f6c9c16f740 4 client.26231 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.231+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1bbae68b0 con 0x55f1bbb87e70 2024-04-26T01:54:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.231+0000 7f6c92800640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbb87e70 0x55f1bbb8a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.232+0000 7f6c92800640 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbb87e70 0x55f1bbb8a260 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e89c813753603fa2 server_cookie=2fb748b81531e16e in_seq=0 out_seq=0 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.232+0000 7f6c77e00640 10 client.26231 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.232+0000 7f6c77e00640 1 -- 172.21.15.52:0/3829393305 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1bbae68b0 con 0x55f1bbb87e70 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.232+0000 7f6c77e00640 10 client.26231 handle_command_reply: tid=0 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.232+0000 7f6c9c16f740 1 client.26231 shutdown 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 msgr2=0x55f1bbb75040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb75040 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c02a910 tx=0x7f6c7c005270 comp rx=0 tx=0).stop 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 shutdown_connections 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbb87e70 0x55f1bbb8a260 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb77b90 0x55f1bbb75580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb75040 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/3829393305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb74b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c77e00640 0 client.26231 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 >> 172.21.15.52:0/3829393305 conn(0x55f1bbb68ec0 msgr2=0x55f1bbb6ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 shutdown_connections 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 1 -- 172.21.15.52:0/3829393305 wait complete. 2024-04-26T01:54:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.233+0000 7f6c9c16f740 20 client.26231 trim_cache size 0 max 16384 2024-04-26T01:54:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.251+0000 7f6c9c16f740 1 Processor -- start 2024-04-26T01:54:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.251+0000 7f6c9c16f740 1 -- start start 2024-04-26T01:54:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.251+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.251+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb8fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.251+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb90090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941b2490 con 0x55f1bbb77b90 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941afd60 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941adbb0 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60332/0 (socket says 172.21.15.52:60332) 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 -- 172.21.15.52:0/2128554806 learned_addr learned my addr 172.21.15.52:0/2128554806 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb90090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c89400640 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb8fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 -- 172.21.15.52:0/2128554806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 msgr2=0x55f1bbb8fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb8fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 -- 172.21.15.52:0/2128554806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 msgr2=0x55f1bbb90090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb90090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.252+0000 7f6c92800640 1 -- 172.21.15.52:0/2128554806 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1bbb79f80 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.253+0000 7f6c92800640 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55f1bbb905d0 tx=0x7f6c8c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a2084287108b2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.253+0000 7f6c82a00640 1 -- 172.21.15.52:0/2128554806 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c025030 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.253+0000 7f6c82a00640 1 -- 172.21.15.52:0/2128554806 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c026040 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.253+0000 7f6c82a00640 1 -- 172.21.15.52:0/2128554806 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c004010 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- 172.21.15.52:0/2128554806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 msgr2=0x55f1bbb83680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55f1bbb905d0 tx=0x7f6c8c005bd0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- 172.21.15.52:0/2128554806 shutdown_connections 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb90090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb8fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/2128554806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb83680 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- 172.21.15.52:0/2128554806 >> 172.21.15.52:0/2128554806 conn(0x55f1bbb68ec0 msgr2=0x55f1bbb6feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- 172.21.15.52:0/2128554806 shutdown_connections 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- 172.21.15.52:0/2128554806 wait complete. 2024-04-26T01:54:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.254+0000 7f6c9c16f740 1 -- start start 2024-04-26T01:54:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 Processor -- start 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb7fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb7d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941ae220 con 0x55f1bbb77b90 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941af690 con 0x55f1bbb6c9b0 2024-04-26T01:54:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6c941ae8f0 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c9c16f740 10 client.0 _ll_register_callbacks cb 0x7f6c7c004480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c89400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c89400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60232/0 (socket says 172.21.15.52:60232) 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c89400640 1 -- 172.21.15.52:0/947820307 learned_addr learned my addr 172.21.15.52:0/947820307 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c92800640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb7fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.255+0000 7f6c92800640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb7d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 -- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 msgr2=0x55f1bbb7fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb7fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 -- 172.21.15.52:0/947820307 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 msgr2=0x55f1bbb7d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb7d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 -- 172.21.15.52:0/947820307 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1bbb7d890 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c89400640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c031840 tx=0x7f6c7c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428f1712762c6db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c77e00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c035070 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8477+0+0 (secure 0 0 0) 0x7f6c7c0314a0 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c026040 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.256+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6c7c005850 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 10 client.26236 fetch_fsmap learned FSMap version 41 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f1bbbf6820 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8c026040 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 41) ==== 8346+0+0 (secure 0 0 0) 0x7f6c7c003710 con 0x55f1bbb6d2d0 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 10 client.26236 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 10 client.26236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 20 client.26236 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbbf6fd0 0x55f1bbbf93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 4 client.26236 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1bbae0320 con 0x55f1bbbf6fd0 2024-04-26T01:54:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.257+0000 7f6c92800640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbbf6fd0 0x55f1bbbf93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.258+0000 7f6c92800640 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbbf6fd0 0x55f1bbbf93f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55618ef91a5e1d27 server_cookie=5785dfd24e3ccab9 in_seq=0 out_seq=0 2024-04-26T01:54:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.258+0000 7f6c77e00640 10 client.26236 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.301+0000 7f6c77e00640 1 -- 172.21.15.52:0/947820307 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1bbae0320 con 0x55f1bbbf6fd0 2024-04-26T01:54:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.301+0000 7f6c77e00640 10 client.26236 handle_command_reply: tid=0 2024-04-26T01:54:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 client.26236 shutdown 2024-04-26T01:54:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 msgr2=0x55f1bbb80050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c031840 tx=0x7f6c7c005fd0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 shutdown_connections 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f1bbbf6fd0 0x55f1bbbf93f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1bbb77b90 0x55f1bbb7d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f1bbb6d2d0 0x55f1bbb80050 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c77e00640 0 client.26236 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 --2- 172.21.15.52:0/947820307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f1bbb6c9b0 0x55f1bbb7fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 >> 172.21.15.52:0/947820307 conn(0x55f1bbb68ec0 msgr2=0x55f1bbb6e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 shutdown_connections 2024-04-26T01:54:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.302+0000 7f6c9c16f740 1 -- 172.21.15.52:0/947820307 wait complete. 2024-04-26T01:54:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c9c16f740 20 client.26236 trim_cache size 0 max 16384 2024-04-26T01:54:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:54:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:54:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:54:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 msgr2=0x7f6c780a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 0x7f6c780a6250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55f1bbb6df50 tx=0x7f6c84000bc0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 msgr2=0x7f6c941a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400e9f0 tx=0x7f6c8400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 shutdown_connections 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6c780a3d90 0x7f6c780a6250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9410e6e0 0x7f6c9419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c941014e0 0x7f6c941a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.303+0000 7f6c93200640 1 --2- 172.21.15.52:0/1401205674 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c94100b30 0x7f6c941a0440 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.304+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 >> 172.21.15.52:0/1401205674 conn(0x7f6c940fc790 msgr2=0x7f6c940feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.304+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 shutdown_connections 2024-04-26T01:54:28.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:28.304+0000 7f6c93200640 1 -- 172.21.15.52:0/1401205674 wait complete. 2024-04-26T01:54:28.925 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T01:54:28.925 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T01:54:28.925 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:28 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/2415042021 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595c512fa40 con 0x5595c5102800 2024-04-26T01:54:28.925 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:54:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T01:54:29.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 237814 2.85 MB/sec execute 348 sec latency 28.511 ms 2024-04-26T01:54:29.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:29 smithi052 ceph-mon[27851]: pgmap v295: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-04-26T01:54:29.742 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:29 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T01:54:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:29 smithi191 ceph-mon[32745]: pgmap v295: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-04-26T01:54:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:29 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T01:54:29.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:29 smithi183 ceph-mon[32752]: pgmap v295: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 73 op/s 2024-04-26T01:54:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:29 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T01:54:30.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 238171 2.85 MB/sec execute 349 sec latency 23.960 ms 2024-04-26T01:54:30.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 -- 172.21.15.52:0/2644330280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 msgr2=0x7f3dd8107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:30.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/2644330280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8107d60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009990 tx=0x7f3dc8009ca0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 -- 172.21.15.52:0/2644330280 shutdown_connections 2024-04-26T01:54:30.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/2644330280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 0x7f3dd810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/2644330280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8107d60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/2644330280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 -- 172.21.15.52:0/2644330280 >> 172.21.15.52:0/2644330280 conn(0x7f3dd80788b0 msgr2=0x7f3dd8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:30.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 -- 172.21.15.52:0/2644330280 shutdown_connections 2024-04-26T01:54:30.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.599+0000 7f3ddcc00700 1 -- 172.21.15.52:0/2644330280 wait complete. 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.600+0000 7f3ddcc00700 1 Processor -- start 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 -- start start 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd81034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 0x7f3dd8103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd8111ad0 con 0x7f3dd81022f0 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd8111950 con 0x7f3dd8107980 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3ddcc00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd8111c50 con 0x7f3dd8101930 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3dcd400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 0x7f3dd8103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:30.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3dd6800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:30.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3dd5e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd81034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:30.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3dd6800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60354/0 (socket says 172.21.15.52:60354) 2024-04-26T01:54:30.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.601+0000 7f3dd6800700 1 -- 172.21.15.52:0/4075545768 learned_addr learned my addr 172.21.15.52:0/4075545768 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:30.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 -- 172.21.15.52:0/4075545768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 msgr2=0x7f3dd81034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd81034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 -- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 msgr2=0x7f3dd8103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 0x7f3dd8103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 -- 172.21.15.52:0/4075545768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd81a7670 con 0x7f3dd8107980 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd5e00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd81034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.602+0000 7f3dd6800700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000ef30 tx=0x7f3dd000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.603+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00092e0 con 0x7f3dd8107980 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.603+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dd0004bb0 con 0x7f3dd8107980 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.603+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0004020 con 0x7f3dd8107980 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.603+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd81a7900 con 0x7f3dd8107980 2024-04-26T01:54:30.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.603+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd81a7df0 con 0x7f3dd8107980 2024-04-26T01:54:30.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.604+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd8069730 con 0x7f3dd8107980 2024-04-26T01:54:30.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.606+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3dd001b030 con 0x7f3dd8107980 2024-04-26T01:54:30.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.608+0000 7f3dcf400700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 0x7f3dbc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:30.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.608+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f3dd00d0190 con 0x7f3dd8107980 2024-04-26T01:54:30.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.608+0000 7f3dd5e00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 0x7f3dbc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:30.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.609+0000 7f3dd5e00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 0x7f3dbc0a62f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800b2e0 tx=0x7f3dc801a040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:30.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.613+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3dd0022020 con 0x7f3dd8107980 2024-04-26T01:54:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:30 smithi052 ceph-mon[27851]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:54:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:30 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/4039499547,v1:172.21.15.183:6837/4039499547] up:boot 2024-04-26T01:54:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:30 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:30 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:30.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:30 smithi191 ceph-mon[32745]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:54:30.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:30.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:30 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/4039499547,v1:172.21.15.183:6837/4039499547] up:boot 2024-04-26T01:54:30.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:30 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:30 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:30.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:30 smithi183 ceph-mon[32752]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T01:54:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:54:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:30 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/4039499547,v1:172.21.15.183:6837/4039499547] up:boot 2024-04-26T01:54:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:30 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:54:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:30 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:30.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.943+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dd81a8150 con 0x7f3dd8107980 2024-04-26T01:54:30.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.944+0000 7f3dcf400700 1 -- 172.21.15.52:0/4075545768 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11885 (secure 0 0 0) 0x7f3dd0010040 con 0x7f3dd8107980 2024-04-26T01:54:30.943 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:54:30.943 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":43,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:54:29.535587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.948+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 msgr2=0x7f3dbc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.948+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 0x7f3dbc0a62f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800b2e0 tx=0x7f3dc801a040 comp rx=0 tx=0).stop 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.948+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 msgr2=0x7f3dd8103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.948+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000ef30 tx=0x7f3dd000c3a0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 shutdown_connections 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3dbc0a3e30 0x7f3dbc0a62f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd8107980 0x7f3dd8103f60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dd8101930 0x7f3dd81034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 --2- 172.21.15.52:0/4075545768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3dd81022f0 0x7f3dd8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 >> 172.21.15.52:0/4075545768 conn(0x7f3dd80788b0 msgr2=0x7f3dd806b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 shutdown_connections 2024-04-26T01:54:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:30.949+0000 7f3ddcc00700 1 -- 172.21.15.52:0/4075545768 wait complete. 2024-04-26T01:54:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 43 2024-04-26T01:54:31.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 239215 2.86 MB/sec execute 350 sec latency 22.739 ms 2024-04-26T01:54:31.803 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:31 smithi191 ceph-mon[32745]: pgmap v297: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:31.803 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:31 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4075545768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:31 smithi183 ceph-mon[32752]: pgmap v297: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:31 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4075545768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:31 smithi052 ceph-mon[27851]: pgmap v297: 97 pgs: 97 active+clean; 94 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-04-26T01:54:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:31 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4075545768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:54:32.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 240689 2.87 MB/sec execute 351 sec latency 56.887 ms 2024-04-26T01:54:32.660 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0' 2024-04-26T01:54:32.660 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0 2024-04-26T01:54:33.119 INFO:tasks.workunit.client.0.smithi052.stdout: 1 241620 2.88 MB/sec execute 352 sec latency 14.646 ms 2024-04-26T01:54:33.151 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:33.568 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:33 smithi052 ceph-mon[27851]: pgmap v298: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:33.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:33 smithi191 ceph-mon[32745]: pgmap v298: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:33.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:33 smithi183 ceph-mon[32752]: pgmap v298: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:34.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 242374 2.88 MB/sec execute 353 sec latency 21.321 ms 2024-04-26T01:54:35.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 243690 2.88 MB/sec execute 354 sec latency 22.432 ms 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 -- 172.21.15.52:0/940340351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc108a60 msgr2=0x7f41dc108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 --2- 172.21.15.52:0/940340351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc108a60 0x7f41dc108e40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009990 tx=0x7f41cc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 -- 172.21.15.52:0/940340351 shutdown_connections 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 --2- 172.21.15.52:0/940340351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc10f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 --2- 172.21.15.52:0/940340351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc108a60 0x7f41dc108e40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 --2- 172.21.15.52:0/940340351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc06a740 0x7f41dc06abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 -- 172.21.15.52:0/940340351 >> 172.21.15.52:0/940340351 conn(0x7f41dc0788e0 msgr2=0x7f41dc078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 -- 172.21.15.52:0/940340351 shutdown_connections 2024-04-26T01:54:35.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.741+0000 7f41e2c00700 1 -- 172.21.15.52:0/940340351 wait complete. 2024-04-26T01:54:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 Processor -- start 2024-04-26T01:54:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 -- start start 2024-04-26T01:54:35.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 0x7f41dc19b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 0x7f41dc19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc112bf0 con 0x7f41dc06b100 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc112a70 con 0x7f41dc108a60 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.743+0000 7f41e2c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc112d70 con 0x7f41dc06a740 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 0x7f41dc19b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41db400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41e0c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 0x7f41dc19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41db400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc19b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56976/0 (socket says 172.21.15.52:56976) 2024-04-26T01:54:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41e0c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 0x7f41dc19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54138/0 (socket says 172.21.15.52:54138) 2024-04-26T01:54:35.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41db400700 1 -- 172.21.15.52:0/2070013317 learned_addr learned my addr 172.21.15.52:0/2070013317 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 -- 172.21.15.52:0/2070013317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 msgr2=0x7f41dc19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 0x7f41dc19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 -- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 msgr2=0x7f41dc19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc19b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.744+0000 7f41dbe00700 1 -- 172.21.15.52:0/2070013317 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc1a0260 con 0x7f41dc06a740 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 0x7f41dc19b250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006010 tx=0x7f41cc0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41d8c00700 1 -- 172.21.15.52:0/2070013317 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc027070 con 0x7f41dc06a740 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41d8c00700 1 -- 172.21.15.52:0/2070013317 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41cc022470 con 0x7f41dc06a740 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc1a04f0 con 0x7f41dc06a740 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41d8c00700 1 -- 172.21.15.52:0/2070013317 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc017650 con 0x7f41dc06a740 2024-04-26T01:54:35.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.745+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1a7dd0 con 0x7f41dc06a740 2024-04-26T01:54:35.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.749+0000 7f41d8c00700 1 -- 172.21.15.52:0/2070013317 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f41cc02a070 con 0x7f41dc06a740 2024-04-26T01:54:35.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.750+0000 7f41d8c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 0x7f41c00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.750+0000 7f41ed616b80 1 Processor -- start 2024-04-26T01:54:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.750+0000 7f41d8c00700 1 -- 172.21.15.52:0/2070013317 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f41cc0d1390 con 0x7f41dc06a740 2024-04-26T01:54:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.751+0000 7f41db400700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 0x7f41c00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.751+0000 7f41ed616b80 1 -- start start 2024-04-26T01:54:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63d02d0 0x5584b63dac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63dd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5584b6323020 con 0x5584b63d02d0 2024-04-26T01:54:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5584b62c98c0 con 0x5584b63db180 2024-04-26T01:54:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41ed616b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5584b631efe0 con 0x5584b63cf9b0 2024-04-26T01:54:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41db400700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 0x7f41c00a6390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41dc10fc70 tx=0x7f41c8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41dbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41dbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37650/0 (socket says 172.21.15.52:37650) 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41dbe00700 1 -- 172.21.15.52:0/2286887391 learned_addr learned my addr 172.21.15.52:0/2286887391 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41e0c00700 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63dd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.752+0000 7f41db400700 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63d02d0 0x5584b63dac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 -- 172.21.15.52:0/2286887391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 msgr2=0x5584b63dd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63dd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 -- 172.21.15.52:0/2286887391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63d02d0 msgr2=0x5584b63dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63d02d0 0x5584b63dac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 -- 172.21.15.52:0/2286887391 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b62bf810 con 0x5584b63cf9b0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41dbe00700 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0a5a40 tx=0x7f41cc09cfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2c67b0c3e5ac247 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41d1600700 1 -- 172.21.15.52:0/2286887391 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc09d390 con 0x5584b63cf9b0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41d1600700 1 -- 172.21.15.52:0/2286887391 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41cc0a1740 con 0x5584b63cf9b0 2024-04-26T01:54:35.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.753+0000 7f41d1600700 1 -- 172.21.15.52:0/2286887391 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc09d5f0 con 0x5584b63cf9b0 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 -- 172.21.15.52:0/2286887391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 msgr2=0x5584b63cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0a5a40 tx=0x7f41cc09cfa0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 -- 172.21.15.52:0/2286887391 shutdown_connections 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63dd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63cfd90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 --2- 172.21.15.52:0/2286887391 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63d02d0 0x5584b63dac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.754+0000 7f41ed616b80 1 -- 172.21.15.52:0/2286887391 >> 172.21.15.52:0/2286887391 conn(0x5584b62fe730 msgr2=0x5584b63cbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.755+0000 7f41ed616b80 1 -- 172.21.15.52:0/2286887391 shutdown_connections 2024-04-26T01:54:35.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.755+0000 7f41ed616b80 1 -- 172.21.15.52:0/2286887391 wait complete. 2024-04-26T01:54:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.755+0000 7f41ed616b80 1 -- start start 2024-04-26T01:54:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.756+0000 7f41ed616b80 1 Processor -- start 2024-04-26T01:54:35.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.756+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.756+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63d8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41db400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41db400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37662/0 (socket says 172.21.15.52:37662) 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41db400700 1 -- 172.21.15.52:0/1312277168 learned_addr learned my addr 172.21.15.52:0/1312277168 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41dbe00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5584b63ddab0 con 0x5584b63cf9b0 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5584b62dc9f0 con 0x5584b63db180 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5584b63ddc30 con 0x5584b63d02d0 2024-04-26T01:54:35.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.757+0000 7f41ed616b80 10 client.0 _ll_register_callbacks cb 0x5584b62bb8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41e0c00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41db400700 1 -- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 msgr2=0x5584b63d8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41db400700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63d8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41db400700 1 -- 172.21.15.52:0/1312277168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 msgr2=0x5584b63da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41db400700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63da8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41db400700 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b6270a60 con 0x5584b63d02d0 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41dbe00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63da8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.758+0000 7f41e0c00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63d8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41db400700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f41c801dc50 tx=0x7f41c80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372a091bf4d0b1fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41c6000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:54:35.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c800ea80 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41cc0a1740 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f41c8009040 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f41c800ec20 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41ed616b80 10 client.26260 fetch_fsmap learned FSMap version 43 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.759+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5584b62b9c80 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41cc0a1740 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f41c802f3d0 con 0x5584b63d02d0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 10 client.26260 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 10 client.26260 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 20 client.26260 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b63ea6e0 0x5584b63ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 4 client.26260 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.760+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584b6330810 con 0x5584b63ea6e0 2024-04-26T01:54:35.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.761+0000 7f41dbe00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b63ea6e0 0x5584b63ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.761+0000 7f41dbe00700 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b63ea6e0 0x5584b63ecad0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f71e53d9f3bc24d server_cookie=d5e700900a134586 in_seq=0 out_seq=0 2024-04-26T01:54:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.761+0000 7f41c6000700 10 client.26260 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41c6000700 1 -- 172.21.15.52:0/1312277168 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584b6330810 con 0x5584b63ea6e0 2024-04-26T01:54:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41c6000700 10 client.26260 handle_command_reply: tid=0 2024-04-26T01:54:35.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 client.26260 shutdown 2024-04-26T01:54:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 msgr2=0x5584b63d7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f41c801dc50 tx=0x7f41c80047c0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 shutdown_connections 2024-04-26T01:54:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b63ea6e0 0x5584b63ecad0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63db180 0x5584b63d8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63da8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 --2- 172.21.15.52:0/1312277168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63d02d0 0x5584b63d7c40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 >> 172.21.15.52:0/1312277168 conn(0x5584b62fe730 msgr2=0x5584b63cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41c6000700 0 client.26260 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.762+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 shutdown_connections 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.763+0000 7f41ed616b80 1 -- 172.21.15.52:0/1312277168 wait complete. 2024-04-26T01:54:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.763+0000 7f41ed616b80 20 client.26260 trim_cache size 0 max 16384 2024-04-26T01:54:35.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 Processor -- start 2024-04-26T01:54:35.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 -- start start 2024-04-26T01:54:35.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:35 smithi052 ceph-mon[27851]: pgmap v299: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:35.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63e6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 0x5584b63f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1b1c80 con 0x5584b63db180 2024-04-26T01:54:35.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1af510 con 0x5584b63d02d0 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.790+0000 7f41ed616b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1ad360 con 0x5584b63cf9b0 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54168/0 (socket says 172.21.15.52:54168) 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 -- 172.21.15.52:0/963985227 learned_addr learned my addr 172.21.15.52:0/963985227 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41dbe00700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 -- 172.21.15.52:0/963985227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 msgr2=0x5584b63e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41e0c00700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 0x5584b63f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 -- 172.21.15.52:0/963985227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 msgr2=0x5584b63f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 0x5584b63f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41db400700 1 -- 172.21.15.52:0/963985227 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b63d28c0 con 0x5584b63d02d0 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41e0c00700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 0x5584b63f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.791+0000 7f41dbe00700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.792+0000 7f41db400700 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41c8016040 tx=0x7f41c8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935bf6e427ee6c9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.792+0000 7f41d0c00700 1 -- 172.21.15.52:0/963985227 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8034070 con 0x5584b63d02d0 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.792+0000 7f41d0c00700 1 -- 172.21.15.52:0/963985227 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c8011ab0 con 0x5584b63d02d0 2024-04-26T01:54:35.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.792+0000 7f41d0c00700 1 -- 172.21.15.52:0/963985227 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c802f3d0 con 0x5584b63d02d0 2024-04-26T01:54:35.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 -- 172.21.15.52:0/963985227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 msgr2=0x5584b63d9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41c8016040 tx=0x7f41c8000bc0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 -- 172.21.15.52:0/963985227 shutdown_connections 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63db180 0x5584b63f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63cf9b0 0x5584b63e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 --2- 172.21.15.52:0/963985227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63d9a00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 -- 172.21.15.52:0/963985227 >> 172.21.15.52:0/963985227 conn(0x5584b62fe730 msgr2=0x5584b63d92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 -- 172.21.15.52:0/963985227 shutdown_connections 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.793+0000 7f41ed616b80 1 -- 172.21.15.52:0/963985227 wait complete. 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.794+0000 7f41ed616b80 1 -- start start 2024-04-26T01:54:35.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.794+0000 7f41ed616b80 1 Processor -- start 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63dfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63e0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1ad9d0 con 0x5584b63cf9b0 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1aee40 con 0x5584b63d02d0 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1ae0a0 con 0x5584b63db180 2024-04-26T01:54:35.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41ed616b80 10 client.0 _ll_register_callbacks cb 0x5584b62bb8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41e0c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41dbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63dfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41e0c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37672/0 (socket says 172.21.15.52:37672) 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.795+0000 7f41e0c00700 1 -- 172.21.15.52:0/3756700509 learned_addr learned my addr 172.21.15.52:0/3756700509 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41db400700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63e0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41e0c00700 1 -- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 msgr2=0x5584b63e0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41e0c00700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63e0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41e0c00700 1 -- 172.21.15.52:0/3756700509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 msgr2=0x5584b63dfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41e0c00700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63dfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.796+0000 7f41e0c00700 1 -- 172.21.15.52:0/3756700509 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b6451180 con 0x5584b63db180 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.797+0000 7f41e0c00700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f41d4007f00 tx=0x7f41d4017690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=947d385c800c0418 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.797+0000 7f41c6000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.797+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4018420 con 0x5584b63db180 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.797+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f41d40185c0 con 0x5584b63db180 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.797+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c8011ab0 con 0x5584b63db180 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f41d40187c0 con 0x5584b63db180 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 10 client.26266 fetch_fsmap learned FSMap version 43 2024-04-26T01:54:35.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5584b6452290 con 0x5584b63db180 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c8011ab0 con 0x5584b63db180 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f41d4020a70 con 0x5584b63db180 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 10 client.26266 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 10 client.26266 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 20 client.26266 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b6452a70 0x5584b6454e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:35.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.798+0000 7f41ed616b80 4 client.26266 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}] 2024-04-26T01:54:35.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.799+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}) -- 0x5584b632a8b0 con 0x5584b6452a70 2024-04-26T01:54:35.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.799+0000 7f41dbe00700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b6452a70 0x5584b6454e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:35.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.799+0000 7f41dbe00700 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b6452a70 0x5584b6454e90 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3cba024abd1bdaa server_cookie=b156cbd4ad676c05 in_seq=0 out_seq=0 2024-04-26T01:54:35.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.799+0000 7f41c6000700 10 client.26266 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:35.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:35 smithi191 ceph-mon[32745]: pgmap v299: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:35.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.907+0000 7f41c6000700 1 -- 172.21.15.52:0/3756700509 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5584b632a8b0 con 0x5584b6452a70 2024-04-26T01:54:35.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.907+0000 7f41c6000700 10 client.26266 handle_command_reply: tid=0 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.907+0000 7f41ed616b80 1 client.26266 shutdown 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 msgr2=0x5584b6450c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f41d4007f00 tx=0x7f41d4017690 comp rx=0 tx=0).stop 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 shutdown_connections 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5584b6452a70 0x5584b6454e90 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5584b63db180 0x5584b6450c40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584b63cf9b0 0x5584b63dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41c6000700 0 client.26266 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 --2- 172.21.15.52:0/3756700509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5584b63d02d0 0x5584b63e0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 >> 172.21.15.52:0/3756700509 conn(0x5584b62fe730 msgr2=0x5584b63d2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 shutdown_connections 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.908+0000 7f41ed616b80 1 -- 172.21.15.52:0/3756700509 wait complete. 2024-04-26T01:54:35.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41ed616b80 20 client.26266 trim_cache size 0 max 16384 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 43, 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 26, 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "76bc1138": { 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 26, 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:54:35.908 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.1, 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:54:35.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 msgr2=0x7f41c00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 0x7f41c00a6390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41dc10fc70 tx=0x7f41c8009260 comp rx=0 tx=0).stop 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 msgr2=0x7f41dc19b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:35.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 0x7f41dc19b250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006010 tx=0x7f41cc0057c0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.909+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 shutdown_connections 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41c00a3ed0 0x7f41c00a6390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41dc108a60 0x7f41dc19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41dc06a740 0x7f41dc19b250 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 --2- 172.21.15.52:0/2070013317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41dc06b100 0x7f41dc19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 >> 172.21.15.52:0/2070013317 conn(0x7f41dc0788e0 msgr2=0x7f41dc1002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 shutdown_connections 2024-04-26T01:54:35.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:54:35.910+0000 7f41e2c00700 1 -- 172.21.15.52:0/2070013317 wait complete. 2024-04-26T01:54:35.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:35 smithi183 ceph-mon[32752]: pgmap v299: 97 pgs: 97 active+clean; 92 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-26T01:54:36.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.90 MB/sec execute 355 sec latency 227.785 ms 2024-04-26T01:54:36.776 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '76bc1138', quiesce took 9.4 seconds. Will release after: 22.1 2024-04-26T01:54:37.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.90 MB/sec execute 356 sec latency 1227.926 ms 2024-04-26T01:54:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:37 smithi191 ceph-mon[32745]: pgmap v300: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:37 smithi183 ceph-mon[32752]: pgmap v300: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:37 smithi052 ceph-mon[27851]: pgmap v300: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:38.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.89 MB/sec execute 357 sec latency 2228.062 ms 2024-04-26T01:54:38.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:38 smithi191 ceph-mon[32745]: pgmap v301: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:38.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:38 smithi183 ceph-mon[32752]: pgmap v301: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:38.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:38 smithi052 ceph-mon[27851]: pgmap v301: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 162 op/s 2024-04-26T01:54:39.120 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.88 MB/sec execute 358 sec latency 3228.198 ms 2024-04-26T01:54:40.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.87 MB/sec execute 359 sec latency 4228.364 ms 2024-04-26T01:54:41.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.86 MB/sec execute 360 sec latency 5228.501 ms 2024-04-26T01:54:41.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:41 smithi183 ceph-mon[32752]: pgmap v302: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 -- 172.21.15.52:0/3171021058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd81070a0 msgr2=0x7f1fd810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/3171021058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd81070a0 0x7f1fd810ec70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009ca0 tx=0x7f1fd002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 -- 172.21.15.52:0/3171021058 shutdown_connections 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/3171021058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd81070a0 0x7f1fd810ec70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/3171021058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd8102980 0x7f1fd81069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.453+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/3171021058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.454+0000 7f1fdcc00640 1 -- 172.21.15.52:0/3171021058 >> 172.21.15.52:0/3171021058 conn(0x7f1fd80fd890 msgr2=0x7f1fd80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.454+0000 7f1fdcc00640 1 -- 172.21.15.52:0/3171021058 shutdown_connections 2024-04-26T01:54:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.454+0000 7f1fdcc00640 1 -- 172.21.15.52:0/3171021058 wait complete. 2024-04-26T01:54:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.455+0000 7f1fdcc00640 1 Processor -- start 2024-04-26T01:54:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.455+0000 7f1fdcc00640 1 -- start start 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.455+0000 7f1fdcc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fdcc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 0x7f1fd8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fdcc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd819ce30 0x7f1fd819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fdcc00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd8112060 con 0x7f1fd81070a0 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fdcc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd8111ee0 con 0x7f1fd8101fd0 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fdcc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81121e0 con 0x7f1fd819ce30 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 0x7f1fd8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54202/0 (socket says 172.21.15.52:54202) 2024-04-26T01:54:41.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 -- 172.21.15.52:0/418552857 learned_addr learned my addr 172.21.15.52:0/418552857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:41.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd819ce30 0x7f1fd819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 -- 172.21.15.52:0/418552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd819ce30 msgr2=0x7f1fd819d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd819ce30 0x7f1fd819d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 -- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 msgr2=0x7f1fd8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 0x7f1fd8198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd7400640 1 -- 172.21.15.52:0/418552857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd819d920 con 0x7f1fd8101fd0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.456+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 0x7f1fd8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fd7400640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800e9f0 tx=0x7f1fc800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fcfe00640 1 -- 172.21.15.52:0/418552857 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800cd80 con 0x7f1fd8101fd0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd819dc10 con 0x7f1fd8101fd0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fcfe00640 1 -- 172.21.15.52:0/418552857 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc80054d0 con 0x7f1fd8101fd0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fcfe00640 1 -- 172.21.15.52:0/418552857 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8010690 con 0x7f1fd8101fd0 2024-04-26T01:54:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.457+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd81a5400 con 0x7f1fd8101fd0 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.460+0000 7f1fe1065740 1 Processor -- start 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 -- start start 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fcfe00640 1 -- 172.21.15.52:0/418552857 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1fc8004e90 con 0x7f1fd8101fd0 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607422d0 0x55bb6074c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6074cb90 0x55bb6074ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb606a7900 con 0x55bb6074cb90 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bb6064def0 con 0x55bb607419b0 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fe1065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bb606a38c0 con 0x55bb607422d0 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54204/0 (socket says 172.21.15.52:54204) 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fd7e00640 1 -- 172.21.15.52:0/577389503 learned_addr learned my addr 172.21.15.52:0/577389503 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fd7400640 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607422d0 0x55bb6074c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.461+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6074cb90 0x55bb6074ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 -- 172.21.15.52:0/577389503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607422d0 msgr2=0x55bb6074c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607422d0 0x55bb6074c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 -- 172.21.15.52:0/577389503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6074cb90 msgr2=0x55bb6074ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6074cb90 0x55bb6074ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 -- 172.21.15.52:0/577389503 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb603ba210 con 0x55bb607419b0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009ca0 tx=0x7f1fd002f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b304c48b300dca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fcfe00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 0x7f1fbc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fc3e00640 1 -- 172.21.15.52:0/577389503 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0031db0 con 0x55bb607419b0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fc3e00640 1 -- 172.21.15.52:0/577389503 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd0036030 con 0x55bb607419b0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.462+0000 7f1fc3e00640 1 -- 172.21.15.52:0/577389503 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00042a0 con 0x55bb607419b0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fcfe00640 1 -- 172.21.15.52:0/418552857 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f1fc80d0c30 con 0x7f1fd8101fd0 2024-04-26T01:54:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fd7400640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 0x7f1fbc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:41 smithi052 ceph-mon[27851]: pgmap v302: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 -- 172.21.15.52:0/577389503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 msgr2=0x55bb60741d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009ca0 tx=0x7f1fd002f3c0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 -- 172.21.15.52:0/577389503 shutdown_connections 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6074cb90 0x55bb6074ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607422d0 0x55bb6074c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 --2- 172.21.15.52:0/577389503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb60741d90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 -- 172.21.15.52:0/577389503 >> 172.21.15.52:0/577389503 conn(0x55bb6073dec0 msgr2=0x55bb6073e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.463+0000 7f1fe1065740 1 -- 172.21.15.52:0/577389503 shutdown_connections 2024-04-26T01:54:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.464+0000 7f1fd7400640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 0x7f1fbc0a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55bb60742f50 tx=0x7f1fc80d46e0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.464+0000 7f1fe1065740 1 -- 172.21.15.52:0/577389503 wait complete. 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.464+0000 7f1fe1065740 1 -- start start 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.464+0000 7f1fe1065740 1 Processor -- start 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60749b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb6074a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb6074f5e0 con 0x55bb607422d0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bb60661310 con 0x55bb6074cb90 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bb6074f760 con 0x55bb607419b0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fe1065740 10 client.0 _ll_register_callbacks cb 0x55bb6062fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb6074a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb6074a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54208/0 (socket says 172.21.15.52:54208) 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fd7e00640 1 -- 172.21.15.52:0/2371813493 learned_addr learned my addr 172.21.15.52:0/2371813493 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.465+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60749b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 -- 172.21.15.52:0/2371813493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 msgr2=0x55bb60749b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60749b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 -- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 msgr2=0x55bb6074a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb6074a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 -- 172.21.15.52:0/2371813493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb60486620 con 0x55bb607422d0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a060 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80fddb0 tx=0x7f1fc4011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b26916075c6f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4010c30 con 0x55bb607422d0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f1fc4010dd0 con 0x55bb607422d0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.466+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd0036030 con 0x55bb607422d0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1fc4004020 con 0x55bb607422d0 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 10 client.16146 fetch_fsmap learned FSMap version 43 2024-04-26T01:54:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55bb6075b840 con 0x55bb607422d0 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd0036030 con 0x55bb607422d0 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f1fc4023430 con 0x55bb607422d0 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 10 client.16146 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 20 client.16146 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb6075cd50 0x55bb6075f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 4 client.16146 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.467+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb606bb8b0 con 0x55bb6075cd50 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.468+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb6075cd50 0x55bb6075f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.468+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb6075cd50 0x55bb6075f140 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e217eecb1a20a1bb server_cookie=914b52f40b3b371c in_seq=0 out_seq=0 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.468+0000 7f1fc0c00640 10 client.16146 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.469+0000 7f1fc0c00640 1 -- 172.21.15.52:0/2371813493 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb606bb8b0 con 0x55bb6075cd50 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.469+0000 7f1fc0c00640 10 client.16146 handle_command_reply: tid=0 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.469+0000 7f1fe1065740 1 client.16146 shutdown 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 msgr2=0x55bb6074a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a060 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80fddb0 tx=0x7f1fc4011bc0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 shutdown_connections 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb6075cd50 0x55bb6075f140 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb6074a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a060 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 --2- 172.21.15.52:0/2371813493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60749b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fc0c00640 0 client.16146 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 >> 172.21.15.52:0/2371813493 conn(0x55bb6073dec0 msgr2=0x55bb60743bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 shutdown_connections 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 1 -- 172.21.15.52:0/2371813493 wait complete. 2024-04-26T01:54:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.470+0000 7f1fe1065740 20 client.16146 trim_cache size 0 max 16384 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 Processor -- start 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 -- start start 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb6074cb90 0x55bb60764a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81b01c0 con 0x55bb607422d0 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81ada90 con 0x55bb607419b0 2024-04-26T01:54:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fe1065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81ab900 con 0x55bb6074cb90 2024-04-26T01:54:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54218/0 (socket says 172.21.15.52:54218) 2024-04-26T01:54:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fd7e00640 1 -- 172.21.15.52:0/639899060 learned_addr learned my addr 172.21.15.52:0/639899060 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.487+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb6074cb90 0x55bb60764a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 -- 172.21.15.52:0/639899060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb6074cb90 msgr2=0x55bb60764a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb6074cb90 0x55bb60764a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 -- 172.21.15.52:0/639899060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 msgr2=0x55bb607584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 -- 172.21.15.52:0/639899060 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb60738490 con 0x55bb607422d0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a3c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4023a40 tx=0x7f1fc400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4ad67c3765d25e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fc3400640 1 -- 172.21.15.52:0/639899060 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4031070 con 0x55bb607422d0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fc3400640 1 -- 172.21.15.52:0/639899060 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc40054f0 con 0x55bb607422d0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.488+0000 7f1fc3400640 1 -- 172.21.15.52:0/639899060 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4010e90 con 0x55bb607422d0 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 -- 172.21.15.52:0/639899060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 msgr2=0x55bb6074a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a3c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4023a40 tx=0x7f1fc400b540 comp rx=0 tx=0).stop 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 -- 172.21.15.52:0/639899060 shutdown_connections 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb6074cb90 0x55bb60764a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb6074a3c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 --2- 172.21.15.52:0/639899060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb607419b0 0x55bb607584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 -- 172.21.15.52:0/639899060 >> 172.21.15.52:0/639899060 conn(0x55bb6073dec0 msgr2=0x55bb6073d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 -- 172.21.15.52:0/639899060 shutdown_connections 2024-04-26T01:54:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.489+0000 7f1fe1065740 1 -- 172.21.15.52:0/639899060 wait complete. 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 -- start start 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 Processor -- start 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb60753630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb60753b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81abf50 con 0x55bb607422d0 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81ad3c0 con 0x55bb6074cb90 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd81ac620 con 0x55bb607419b0 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.490+0000 7f1fe1065740 10 client.0 _ll_register_callbacks cb 0x7f1fc400bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb60753630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37724/0 (socket says 172.21.15.52:37724) 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 -- 172.21.15.52:0/4063872837 learned_addr learned my addr 172.21.15.52:0/4063872837 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb60753b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 -- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 msgr2=0x55bb60753b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb60753b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 -- 172.21.15.52:0/4063872837 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 msgr2=0x55bb60753630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb60753630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 -- 172.21.15.52:0/4063872837 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb607540b0 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb60753630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fd7e00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009620 tx=0x7f1fd004a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a384f6064e56e4a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.491+0000 7f1fc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0031860 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f1fd0004040 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fc40054f0 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1fd0032ea0 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fe1065740 10 client.26284 fetch_fsmap learned FSMap version 43 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55bb607c3010 con 0x55bb607419b0 2024-04-26T01:54:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc40054f0 con 0x55bb607419b0 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 43) ==== 8346+0+0 (secure 0 0 0) 0x7f1fd0043070 con 0x55bb607419b0 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.492+0000 7f1fe1065740 10 client.26284 fetch_fsmap finished waiting for FSMap version 43 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fe1065740 10 client.26284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fe1065740 20 client.26284 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb607c3760 0x55bb607c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fe1065740 4 client.26284 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb606b5320 con 0x55bb607c3760 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb607c3760 0x55bb607c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.493+0000 7f1fd6a00640 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb607c3760 0x55bb607c5b50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41f7d2d8e2b2d31 server_cookie=8d57016160dda735 in_seq=0 out_seq=0 2024-04-26T01:54:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.494+0000 7f1fc0c00640 10 client.26284 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fc0c00640 1 -- 172.21.15.52:0/4063872837 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb606b5320 con 0x55bb607c3760 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fc0c00640 10 client.26284 handle_command_reply: tid=0 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 client.26284 shutdown 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 msgr2=0x55bb60754bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009620 tx=0x7f1fd004a040 comp rx=0 tx=0).stop 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 shutdown_connections 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bb607c3760 0x55bb607c5b50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bb6074cb90 0x55bb60753b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb607422d0 0x55bb60753630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fc0c00640 0 client.26284 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.541+0000 7f1fe1065740 1 --2- 172.21.15.52:0/4063872837 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bb607419b0 0x55bb60754bb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.542+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 >> 172.21.15.52:0/4063872837 conn(0x55bb6073dec0 msgr2=0x55bb60743360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.542+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 shutdown_connections 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.542+0000 7f1fe1065740 1 -- 172.21.15.52:0/4063872837 wait complete. 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.542+0000 7f1fe1065740 20 client.26284 trim_cache size 0 max 16384 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:54:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.542+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 msgr2=0x7f1fbc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 0x7f1fbc0a61d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55bb60742f50 tx=0x7f1fc80d46e0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 msgr2=0x7f1fd81985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800e9f0 tx=0x7f1fc800edf0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 shutdown_connections 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fbc0a3d10 0x7f1fbc0a61d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fd819ce30 0x7f1fd819d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1fd81070a0 0x7f1fd8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 --2- 172.21.15.52:0/418552857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fd8101fd0 0x7f1fd81985c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 >> 172.21.15.52:0/418552857 conn(0x7f1fd80fd890 msgr2=0x7f1fd80ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 shutdown_connections 2024-04-26T01:54:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:41.543+0000 7f1fdcc00640 1 -- 172.21.15.52:0/418552857 wait complete. 2024-04-26T01:54:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:41 smithi191 ceph-mon[32745]: pgmap v302: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-26T01:54:42.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.86 MB/sec execute 361 sec latency 6228.713 ms 2024-04-26T01:54:43.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.85 MB/sec execute 362 sec latency 7228.850 ms 2024-04-26T01:54:43.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:43 smithi183 ceph-mon[32752]: pgmap v303: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:54:43.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:43 smithi052 ceph-mon[27851]: pgmap v303: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:54:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:43 smithi191 ceph-mon[32745]: pgmap v303: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 146 op/s 2024-04-26T01:54:44.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.84 MB/sec execute 363 sec latency 8229.014 ms 2024-04-26T01:54:45.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.83 MB/sec execute 364 sec latency 9229.148 ms 2024-04-26T01:54:45.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:45 smithi183 ceph-mon[32752]: pgmap v304: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 549 KiB/s wr, 69 op/s 2024-04-26T01:54:45.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:45 smithi052 ceph-mon[27851]: pgmap v304: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 549 KiB/s wr, 69 op/s 2024-04-26T01:54:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:45 smithi191 ceph-mon[32745]: pgmap v304: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 549 KiB/s wr, 69 op/s 2024-04-26T01:54:46.121 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.82 MB/sec execute 365 sec latency 10229.238 ms 2024-04-26T01:54:47.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.82 MB/sec execute 366 sec latency 11229.383 ms 2024-04-26T01:54:47.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:47 smithi183 ceph-mon[32752]: pgmap v305: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 988 KiB/s wr, 107 op/s 2024-04-26T01:54:47.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:47 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:47.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:47 smithi052 ceph-mon[27851]: pgmap v305: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 988 KiB/s wr, 107 op/s 2024-04-26T01:54:47.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:47 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:47 smithi191 ceph-mon[32745]: pgmap v305: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 988 KiB/s wr, 107 op/s 2024-04-26T01:54:47.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:47 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:54:48.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.81 MB/sec execute 367 sec latency 12229.560 ms 2024-04-26T01:54:49.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.80 MB/sec execute 368 sec latency 13229.693 ms 2024-04-26T01:54:49.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:49 smithi052 ceph-mon[27851]: pgmap v306: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:49 smithi191 ceph-mon[32745]: pgmap v306: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:49.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:49 smithi183 ceph-mon[32752]: pgmap v306: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:50.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.79 MB/sec execute 369 sec latency 14229.858 ms 2024-04-26T01:54:51.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.79 MB/sec execute 370 sec latency 15229.984 ms 2024-04-26T01:54:51.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:51 smithi052 ceph-mon[27851]: pgmap v307: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:51 smithi191 ceph-mon[32745]: pgmap v307: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:51 smithi183 ceph-mon[32752]: pgmap v307: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 460 KiB/s wr, 48 op/s 2024-04-26T01:54:52.122 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.78 MB/sec execute 371 sec latency 16230.120 ms 2024-04-26T01:54:53.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.77 MB/sec execute 372 sec latency 17230.267 ms 2024-04-26T01:54:53.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:53 smithi052 ceph-mon[27851]: pgmap v308: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 728 KiB/s wr, 59 op/s 2024-04-26T01:54:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:53 smithi191 ceph-mon[32745]: pgmap v308: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 728 KiB/s wr, 59 op/s 2024-04-26T01:54:53.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:53 smithi183 ceph-mon[32752]: pgmap v308: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 728 KiB/s wr, 59 op/s 2024-04-26T01:54:54.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.76 MB/sec execute 373 sec latency 18230.404 ms 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 -- 172.21.15.52:0/1257536643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 msgr2=0x7f7258111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 --2- 172.21.15.52:0/1257536643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f7258111040 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7250009ca0 tx=0x7f725002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 -- 172.21.15.52:0/1257536643 shutdown_connections 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 --2- 172.21.15.52:0/1257536643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f7258111040 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 --2- 172.21.15.52:0/1257536643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7258104d50 0x7f7258108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 --2- 172.21.15.52:0/1257536643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 0x7f7258104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 -- 172.21.15.52:0/1257536643 >> 172.21.15.52:0/1257536643 conn(0x7f72580ffc40 msgr2=0x7f7258102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 -- 172.21.15.52:0/1257536643 shutdown_connections 2024-04-26T01:54:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.691+0000 7f725ee00640 1 -- 172.21.15.52:0/1257536643 wait complete. 2024-04-26T01:54:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.692+0000 7f725ee00640 1 Processor -- start 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.692+0000 7f725ee00640 1 -- start start 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 0x7f72581a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f725819a690 0x7f725819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7258114410 con 0x7f7258109470 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7258114290 con 0x7f72581043a0 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7258114590 con 0x7f725819a690 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60164/0 (socket says 172.21.15.52:60164) 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f7257400640 1 -- 172.21.15.52:0/3758144180 learned_addr learned my addr 172.21.15.52:0/3758144180 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f7257e00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 0x7f72581a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.693+0000 7f725ce00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f725819a690 0x7f725819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 -- 172.21.15.52:0/3758144180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f725819a690 msgr2=0x7f725819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f725819a690 0x7f725819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 -- 172.21.15.52:0/3758144180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 msgr2=0x7f72581a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 0x7f72581a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 -- 172.21.15.52:0/3758144180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725819b290 con 0x7f7258109470 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7257400640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9c0 tx=0x7f724000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7254c00640 1 -- 172.21.15.52:0/3758144180 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724000cd80 con 0x7f7258109470 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7254c00640 1 -- 172.21.15.52:0/3758144180 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72400054d0 con 0x7f7258109470 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f7254c00640 1 -- 172.21.15.52:0/3758144180 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7240010430 con 0x7f7258109470 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72581a7320 con 0x7f7258109470 2024-04-26T01:54:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.694+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72581a7780 con 0x7f7258109470 2024-04-26T01:54:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 Processor -- start 2024-04-26T01:54:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 -- start start 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7254c00640 1 -- 172.21.15.52:0/3758144180 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7240004fd0 con 0x7f7258109470 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d6140 0x55a0703e04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a07033b900 con 0x55a0703d5820 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a0702e1ef0 con 0x55a0703e0a00 2024-04-26T01:54:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.698+0000 7f7261de8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a0703378c0 con 0x55a0703d6140 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60166/0 (socket says 172.21.15.52:60166) 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 -- 172.21.15.52:0/2619997716 learned_addr learned my addr 172.21.15.52:0/2619997716 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f7257400640 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f7257e00640 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d6140 0x55a0703e04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 -- 172.21.15.52:0/2619997716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d6140 msgr2=0x55a0703e04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d6140 0x55a0703e04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 -- 172.21.15.52:0/2619997716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 msgr2=0x55a0703e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 -- 172.21.15.52:0/2619997716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a07004e210 con 0x55a0703d5820 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.699+0000 7f725ce00640 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7250009ca0 tx=0x7f72500093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e42e98b9ce1842 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f724d600640 1 -- 172.21.15.52:0/2619997716 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250031cf0 con 0x55a0703d5820 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f724d600640 1 -- 172.21.15.52:0/2619997716 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7250036030 con 0x55a0703d5820 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f724d600640 1 -- 172.21.15.52:0/2619997716 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72500042a0 con 0x55a0703d5820 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f7254c00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 0x7f723c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f7254c00640 1 -- 172.21.15.52:0/3758144180 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f72400d0af0 con 0x7f7258109470 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.700+0000 7f7257e00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 0x7f723c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- 172.21.15.52:0/2619997716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 msgr2=0x55a0703d5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7250009ca0 tx=0x7f72500093e0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- 172.21.15.52:0/2619997716 shutdown_connections 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d6140 0x55a0703e04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 --2- 172.21.15.52:0/2619997716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d5820 0x55a0703d5c00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- 172.21.15.52:0/2619997716 >> 172.21.15.52:0/2619997716 conn(0x55a0703d1d30 msgr2=0x55a0703d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- 172.21.15.52:0/2619997716 shutdown_connections 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- 172.21.15.52:0/2619997716 wait complete. 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7257e00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 0x7f723c0a6300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x55a0703d6dc0 tx=0x7f7248005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.701+0000 7f7261de8740 1 -- start start 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 Processor -- start 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703ddec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703de400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a0703e3450 con 0x55a0703d6140 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a0702f5310 con 0x55a0703e0a00 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a0703e35d0 con 0x55a0703d5820 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.702+0000 7f7261de8740 10 client.0 _ll_register_callbacks cb 0x55a0702c3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703ddec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36684/0 (socket says 172.21.15.52:36684) 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257e00640 1 -- 172.21.15.52:0/1701618682 learned_addr learned my addr 172.21.15.52:0/1701618682 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f725ce00640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703de400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 -- 172.21.15.52:0/1701618682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 msgr2=0x55a0703dd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 -- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 msgr2=0x55a0703de400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703de400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 -- 172.21.15.52:0/1701618682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a07011a620 con 0x55a0703d6140 2024-04-26T01:54:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257e00640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7257400640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703ddec0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f72400ce210 tx=0x7f72400d40d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b86e75e5d9ae7d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.703+0000 7f7246000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724009c830 con 0x55a0703d6140 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f724009ad60 con 0x55a0703d6140 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7250036030 con 0x55a0703d6140 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f724009c3c0 con 0x55a0703d6140 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7261de8740 10 client.16164 fetch_fsmap learned FSMap version 44 2024-04-26T01:54:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a0703ef740 con 0x55a0703d6140 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7250036030 con 0x55a0703d6140 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 44) ==== 8350+0+0 (secure 0 0 0) 0x7f72400ce490 con 0x55a0703d6140 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.704+0000 7f7261de8740 10 client.16164 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7261de8740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7261de8740 20 client.16164 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a0703f0bb0 0x55a0703f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7261de8740 4 client.16164 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a07034f8b0 con 0x55a0703f0bb0 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.705+0000 7f7257e00640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a0703f0bb0 0x55a0703f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.706+0000 7f7257e00640 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a0703f0bb0 0x55a0703f2fa0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197a936c16df3d01 server_cookie=4daf1ce7906a27b3 in_seq=0 out_seq=0 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.706+0000 7f7246000640 10 client.16164 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.706+0000 7f7246000640 1 -- 172.21.15.52:0/1701618682 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a07034f8b0 con 0x55a0703f0bb0 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.706+0000 7f7246000640 10 client.16164 handle_command_reply: tid=0 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 client.16164 shutdown 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 msgr2=0x55a0703ddec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703ddec0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f72400ce210 tx=0x7f72400d40d0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 shutdown_connections 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a0703f0bb0 0x55a0703f2fa0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703de400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703ddec0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 --2- 172.21.15.52:0/1701618682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703dd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 >> 172.21.15.52:0/1701618682 conn(0x55a0703d1d30 msgr2=0x55a0703d7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7246000640 0 client.16164 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 shutdown_connections 2024-04-26T01:54:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 1 -- 172.21.15.52:0/1701618682 wait complete. 2024-04-26T01:54:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.707+0000 7f7261de8740 20 client.16164 trim_cache size 0 max 16384 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.723+0000 7f7261de8740 1 Processor -- start 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 -- start start 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703d81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703f8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72581b24e0 con 0x55a0703d6140 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72581afd80 con 0x55a0703e0a00 2024-04-26T01:54:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7261de8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72581adbf0 con 0x55a0703d5820 2024-04-26T01:54:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703d81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7257e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7257e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36692/0 (socket says 172.21.15.52:36692) 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f7257e00640 1 -- 172.21.15.52:0/4035841519 learned_addr learned my addr 172.21.15.52:0/4035841519 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.724+0000 7f725ce00640 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703f8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 -- 172.21.15.52:0/4035841519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 msgr2=0x55a0703f8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703f8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 -- 172.21.15.52:0/4035841519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 msgr2=0x55a0703d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703d81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 -- 172.21.15.52:0/4035841519 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0703f3600 con 0x55a0703d5820 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f7257e00640 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55a0703f3800 tx=0x7f724801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e16cbf484f54dc2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.725+0000 7f724cc00640 1 -- 172.21.15.52:0/4035841519 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248003d40 con 0x55a0703d5820 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f724cc00640 1 -- 172.21.15.52:0/4035841519 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f724802b920 con 0x55a0703d5820 2024-04-26T01:54:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f724cc00640 1 -- 172.21.15.52:0/4035841519 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248005020 con 0x55a0703d5820 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 -- 172.21.15.52:0/4035841519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 msgr2=0x55a0703d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55a0703f3800 tx=0x7f724801b000 comp rx=0 tx=0).stop 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 -- 172.21.15.52:0/4035841519 shutdown_connections 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703f8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703d81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 --2- 172.21.15.52:0/4035841519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703d10d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 -- 172.21.15.52:0/4035841519 >> 172.21.15.52:0/4035841519 conn(0x55a0703d1d30 msgr2=0x55a0703de1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 -- 172.21.15.52:0/4035841519 shutdown_connections 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.726+0000 7f7261de8740 1 -- 172.21.15.52:0/4035841519 wait complete. 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 -- start start 2024-04-26T01:54:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 Processor -- start 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f72581ae240 con 0x55a0703d6140 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72581af6b0 con 0x55a0703e0a00 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f72581ae910 con 0x55a0703d5820 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.727+0000 7f7261de8740 10 client.0 _ll_register_callbacks cb 0x7f72400048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f725ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60182/0 (socket says 172.21.15.52:60182) 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 -- 172.21.15.52:0/579103380 learned_addr learned my addr 172.21.15.52:0/579103380 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257e00640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 -- 172.21.15.52:0/579103380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 msgr2=0x55a0703e8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703e8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 -- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 msgr2=0x55a0703e6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 -- 172.21.15.52:0/579103380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a070456110 con 0x55a0703d6140 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257e00640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703e8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7257400640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f724000c6a0 tx=0x7f72400a52d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da7c1a86a989adf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.728+0000 7f7246000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724009c5a0 con 0x55a0703d6140 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7f724009ad60 con 0x55a0703d6140 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a070349320 con 0x55a0703d6140 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f724009c830 con 0x55a0703d6140 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 10 client.16170 fetch_fsmap learned FSMap version 44 2024-04-26T01:54:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a0704572e0 con 0x55a0703d6140 2024-04-26T01:54:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a070349320 con 0x55a0703d6140 2024-04-26T01:54:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 44) ==== 8350+0+0 (secure 0 0 0) 0x7f72400987b0 con 0x55a0703d6140 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 10 client.16170 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 10 client.16170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 20 client.16170 populate_metadata read hostname 'smithi052' 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a070457a00 0x55a070459e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 4 client.16170 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.729+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a070343190 con 0x55a070457a00 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.730+0000 7f7257e00640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a070457a00 0x55a070459e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.730+0000 7f7246000640 10 client.16170 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.730+0000 7f7257e00640 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a070457a00 0x55a070459e20 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e86049b6731e6b server_cookie=b679b2da05b60961 in_seq=0 out_seq=0 2024-04-26T01:54:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.754+0000 7f7246000640 1 -- 172.21.15.52:0/579103380 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a070343190 con 0x55a070457a00 2024-04-26T01:54:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.754+0000 7f7246000640 10 client.16170 handle_command_reply: tid=0 2024-04-26T01:54:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.754+0000 7f7261de8740 1 client.16170 shutdown 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 msgr2=0x55a0703e6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f724000c6a0 tx=0x7f72400a52d0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 shutdown_connections 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a070457a00 0x55a070459e20 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a0703e0a00 0x55a0703e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a0703d6140 0x55a0703e6140 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 --2- 172.21.15.52:0/579103380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a0703d5820 0x55a0703e8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7246000640 0 client.16170 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 >> 172.21.15.52:0/579103380 conn(0x55a0703d1d30 msgr2=0x55a0703df1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 shutdown_connections 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 1 -- 172.21.15.52:0/579103380 wait complete. 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.755+0000 7f7261de8740 20 client.16170 trim_cache size 0 max 16384 2024-04-26T01:54:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:54:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:54:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:54:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:54:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 msgr2=0x7f723c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 0x7f723c0a6300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x55a0703d6dc0 tx=0x7f7248005f90 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 msgr2=0x7f72581a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9c0 tx=0x7f724000ee80 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 shutdown_connections 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f723c0a3e40 0x7f723c0a6300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f725819a690 0x7f725819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7258109470 0x7f72581a0ac0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 --2- 172.21.15.52:0/3758144180 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72581043a0 0x7f72581a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 >> 172.21.15.52:0/3758144180 conn(0x7f72580ffc40 msgr2=0x7f7258101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 shutdown_connections 2024-04-26T01:54:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:54:54.756+0000 7f725ee00640 1 -- 172.21.15.52:0/3758144180 wait complete. 2024-04-26T01:54:55.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.76 MB/sec execute 374 sec latency 19230.597 ms 2024-04-26T01:54:55.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:55 smithi052 ceph-mon[27851]: pgmap v309: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 707 KiB/s wr, 48 op/s 2024-04-26T01:54:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:55 smithi191 ceph-mon[32745]: pgmap v309: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 707 KiB/s wr, 48 op/s 2024-04-26T01:54:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:55 smithi183 ceph-mon[32752]: pgmap v309: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 707 KiB/s wr, 48 op/s 2024-04-26T01:54:56.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.75 MB/sec execute 375 sec latency 20230.741 ms 2024-04-26T01:54:57.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.74 MB/sec execute 376 sec latency 21230.894 ms 2024-04-26T01:54:57.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:57 smithi052 ceph-mon[27851]: pgmap v310: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 734 KiB/s wr, 60 op/s 2024-04-26T01:54:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:57 smithi191 ceph-mon[32745]: pgmap v310: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 734 KiB/s wr, 60 op/s 2024-04-26T01:54:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:57 smithi183 ceph-mon[32752]: pgmap v310: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 734 KiB/s wr, 60 op/s 2024-04-26T01:54:58.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.73 MB/sec execute 377 sec latency 22231.029 ms 2024-04-26T01:54:58.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:54:59.123 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.73 MB/sec execute 378 sec latency 23231.159 ms 2024-04-26T01:54:59.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:54:59 smithi052 ceph-mon[27851]: pgmap v311: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:54:59.361 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:54:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:54:59 smithi191 ceph-mon[32745]: pgmap v311: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:54:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:54:59 smithi183 ceph-mon[32752]: pgmap v311: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:55:00.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.72 MB/sec execute 379 sec latency 24231.277 ms 2024-04-26T01:55:01.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.71 MB/sec execute 380 sec latency 25231.425 ms 2024-04-26T01:55:01.427 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:01 smithi052 ceph-mon[27851]: pgmap v312: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:55:01.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:01 smithi191 ceph-mon[32745]: pgmap v312: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:55:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:01 smithi183 ceph-mon[32752]: pgmap v312: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 296 KiB/s wr, 22 op/s 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.700+0000 7fbbc5200700 1 -- 172.21.15.52:0/1077795750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0108a50 msgr2=0x7fbbc0108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.700+0000 7fbbc5200700 1 --2- 172.21.15.52:0/1077795750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0108a50 0x7fbbc0108e30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009990 tx=0x7fbbb0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 -- 172.21.15.52:0/1077795750 shutdown_connections 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 --2- 172.21.15.52:0/1077795750 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0103450 0x7fbbc010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 --2- 172.21.15.52:0/1077795750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0108a50 0x7fbbc0108e30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 --2- 172.21.15.52:0/1077795750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0102a90 0x7fbbc0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 -- 172.21.15.52:0/1077795750 >> 172.21.15.52:0/1077795750 conn(0x7fbbc0079ba0 msgr2=0x7fbbc0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 -- 172.21.15.52:0/1077795750 shutdown_connections 2024-04-26T01:55:01.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.701+0000 7fbbc5200700 1 -- 172.21.15.52:0/1077795750 wait complete. 2024-04-26T01:55:01.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.702+0000 7fbbc5200700 1 Processor -- start 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.702+0000 7fbbc5200700 1 -- start start 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 0x7fbbc010ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 0x7fbbc01a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc0112c30 con 0x7fbbc0108a50 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc0112ab0 con 0x7fbbc0102a90 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbc5200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc0112db0 con 0x7fbbc0103450 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbbd400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbbd400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36728/0 (socket says 172.21.15.52:36728) 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbbd400700 1 -- 172.21.15.52:0/346090345 learned_addr learned my addr 172.21.15.52:0/346090345 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:01.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbbde00700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 0x7fbbc010ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.703+0000 7fbbbe800700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 0x7fbbc01a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 -- 172.21.15.52:0/346090345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 msgr2=0x7fbbc010f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 -- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 msgr2=0x7fbbc01a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 0x7fbbc01a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 -- 172.21.15.52:0/346090345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc01a4720 con 0x7fbbc0102a90 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbd400700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbe800700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 0x7fbbc01a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbbde00700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 0x7fbbc010ec40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0000c00 tx=0x7fbbb00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.704+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0026070 con 0x7fbbc0102a90 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.705+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc01a49b0 con 0x7fbbc0102a90 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.705+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbb0021410 con 0x7fbbc0102a90 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.705+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0017650 con 0x7fbbc0102a90 2024-04-26T01:55:01.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.705+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc01ac290 con 0x7fbbc0102a90 2024-04-26T01:55:01.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.706+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc0069730 con 0x7fbbc0102a90 2024-04-26T01:55:01.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.708+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbbb0029070 con 0x7fbbc0102a90 2024-04-26T01:55:01.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.710+0000 7fbbb6a00700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 0x7fbba40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:01.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.710+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7fbbb00d1c70 con 0x7fbbc0102a90 2024-04-26T01:55:01.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.710+0000 7fbbbd400700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 0x7fbba40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:01.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.711+0000 7fbbbd400700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 0x7fbba40a6270 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0076d60 tx=0x7fbbac009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:01.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.712+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbbb0030070 con 0x7fbbc0102a90 2024-04-26T01:55:01.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.997+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbc01ac460 con 0x7fbbc0102a90 2024-04-26T01:55:01.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:01.999+0000 7fbbb6a00700 1 -- 172.21.15.52:0/346090345 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11886 (secure 0 0 0) 0x7fbbb00a4070 con 0x7fbbc0102a90 2024-04-26T01:55:01.998 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:55:01.998 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:54:45.951655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 msgr2=0x7fbba40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 0x7fbba40a6270 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0076d60 tx=0x7fbbac009260 comp rx=0 tx=0).stop 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 msgr2=0x7fbbc010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 0x7fbbc010ec40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0000c00 tx=0x7fbbb00047b0 comp rx=0 tx=0).stop 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 shutdown_connections 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbba40a3db0 0x7fbba40a6270 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbbc0108a50 0x7fbbc01a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:02.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbc0102a90 0x7fbbc010ec40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 --2- 172.21.15.52:0/346090345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbc0103450 0x7fbbc010f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.002+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 >> 172.21.15.52:0/346090345 conn(0x7fbbc0079ba0 msgr2=0x7fbbc0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.003+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 shutdown_connections 2024-04-26T01:55:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:02.003+0000 7fbbc5200700 1 -- 172.21.15.52:0/346090345 wait complete. 2024-04-26T01:55:02.002 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 44 2024-04-26T01:55:02.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.71 MB/sec execute 381 sec latency 26231.578 ms 2024-04-26T01:55:02.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:02 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/346090345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:02.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:02 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/346090345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:02.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:02 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/346090345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:03.081 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 76bc1138 --release --await' 2024-04-26T01:55:03.082 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 76bc1138 --release --await 2024-04-26T01:55:03.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.70 MB/sec execute 382 sec latency 27231.649 ms 2024-04-26T01:55:03.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:03 smithi052 ceph-mon[27851]: pgmap v313: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 682 KiB/s wr, 54 op/s 2024-04-26T01:55:03.543 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:03 smithi191 ceph-mon[32745]: pgmap v313: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 682 KiB/s wr, 54 op/s 2024-04-26T01:55:03.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:03 smithi183 ceph-mon[32752]: pgmap v313: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 682 KiB/s wr, 54 op/s 2024-04-26T01:55:04.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.69 MB/sec execute 383 sec latency 28231.761 ms 2024-04-26T01:55:05.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244855 2.69 MB/sec execute 384 sec latency 29231.900 ms 2024-04-26T01:55:05.307 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:05 smithi052 ceph-mon[27851]: pgmap v314: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 414 KiB/s wr, 43 op/s 2024-04-26T01:55:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:05 smithi191 ceph-mon[32745]: pgmap v314: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 414 KiB/s wr, 43 op/s 2024-04-26T01:55:05.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:05 smithi183 ceph-mon[32752]: pgmap v314: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 414 KiB/s wr, 43 op/s 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 -- 172.21.15.52:0/1230562352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 msgr2=0x7fce88109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 --2- 172.21.15.52:0/1230562352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce88109400 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fce70009990 tx=0x7fce70009ca0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 -- 172.21.15.52:0/1230562352 shutdown_connections 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 --2- 172.21.15.52:0/1230562352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce88115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 --2- 172.21.15.52:0/1230562352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 0x7fce8810f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 --2- 172.21.15.52:0/1230562352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce88109400 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 -- 172.21.15.52:0/1230562352 >> 172.21.15.52:0/1230562352 conn(0x7fce880787d0 msgr2=0x7fce88107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 -- 172.21.15.52:0/1230562352 shutdown_connections 2024-04-26T01:55:05.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.838+0000 7fce87200700 1 -- 172.21.15.52:0/1230562352 wait complete. 2024-04-26T01:55:05.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.839+0000 7fce87200700 1 Processor -- start 2024-04-26T01:55:05.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 -- start start 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce8819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 0x7fce881a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce88119140 con 0x7fce8810f030 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce88118fc0 con 0x7fce88108f80 2024-04-26T01:55:05.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.840+0000 7fce87200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce881192c0 con 0x7fce88109940 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce86800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 0x7fce881a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce8819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35084/0 (socket says 172.21.15.52:35084) 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 -- 172.21.15.52:0/1998031589 learned_addr learned my addr 172.21.15.52:0/1998031589 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 -- 172.21.15.52:0/1998031589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 msgr2=0x7fce8819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce8819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 -- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 msgr2=0x7fce881a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 0x7fce881a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 -- 172.21.15.52:0/1998031589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce881a45e0 con 0x7fce88109940 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.841+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce70000c00 tx=0x7fce700057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.842+0000 7fce7ea00700 1 -- 172.21.15.52:0/1998031589 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70027070 con 0x7fce88109940 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.842+0000 7fce7ea00700 1 -- 172.21.15.52:0/1998031589 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce70022470 con 0x7fce88109940 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.842+0000 7fce7ea00700 1 -- 172.21.15.52:0/1998031589 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70017650 con 0x7fce88109940 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.842+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce881a4870 con 0x7fce88109940 2024-04-26T01:55:05.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.842+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce881ac150 con 0x7fce88109940 2024-04-26T01:55:05.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.846+0000 7fce7ea00700 1 -- 172.21.15.52:0/1998031589 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fce7002a070 con 0x7fce88109940 2024-04-26T01:55:05.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 Processor -- start 2024-04-26T01:55:05.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 -- start start 2024-04-26T01:55:05.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce7ea00700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 0x7fce6c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2d9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce7ea00700 1 -- 172.21.15.52:0/1998031589 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7fce700d1430 con 0x7fce88109940 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5567dc21ef90 con 0x5567dc2cbea0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5567dc1c5830 con 0x5567dc2d7670 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce9798db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5567dc21af50 con 0x5567dc2cc7c0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.847+0000 7fce86800700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 0x7fce6c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43728/0 (socket says 172.21.15.52:43728) 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 -- 172.21.15.52:0/1941771885 learned_addr learned my addr 172.21.15.52:0/1941771885 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce86800700 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2d9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 -- 172.21.15.52:0/1941771885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 msgr2=0x5567dc2d7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 -- 172.21.15.52:0/1941771885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 msgr2=0x5567dc2d9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2d9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.848+0000 7fce85e00700 1 -- 172.21.15.52:0/1941771885 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567dc1bb780 con 0x5567dc2cbea0 2024-04-26T01:55:05.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.849+0000 7fce86800700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 0x7fce6c0a62f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fce80009ec0 tx=0x7fce80016450 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.849+0000 7fce85e00700 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fce78009d80 tx=0x7fce78011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0a9a6aa816d95a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.849+0000 7fce77400700 1 -- 172.21.15.52:0/1941771885 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce78004230 con 0x5567dc2cbea0 2024-04-26T01:55:05.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.849+0000 7fce77400700 1 -- 172.21.15.52:0/1941771885 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce78009e10 con 0x5567dc2cbea0 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.849+0000 7fce77400700 1 -- 172.21.15.52:0/1941771885 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce780197a0 con 0x5567dc2cbea0 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 -- 172.21.15.52:0/1941771885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 msgr2=0x5567dc2cc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fce78009d80 tx=0x7fce78011720 comp rx=0 tx=0).stop 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 -- 172.21.15.52:0/1941771885 shutdown_connections 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2d9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2cc280 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 --2- 172.21.15.52:0/1941771885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 -- 172.21.15.52:0/1941771885 >> 172.21.15.52:0/1941771885 conn(0x5567dc1fb050 msgr2=0x5567dc2c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 -- 172.21.15.52:0/1941771885 shutdown_connections 2024-04-26T01:55:05.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.850+0000 7fce9798db80 1 -- 172.21.15.52:0/1941771885 wait complete. 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 -- start start 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 Processor -- start 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2d7670 0x5567dc2d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5567dc2da100 con 0x5567dc2d7670 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5567dc1d8960 con 0x5567dc2cbea0 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5567dc2da280 con 0x5567dc2cc7c0 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.851+0000 7fce9798db80 10 client.0 _ll_register_callbacks cb 0x5567dc1b7820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:05.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce86800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2d7670 0x5567dc2d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce85e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41506/0 (socket says 172.21.15.52:41506) 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce85e00700 1 -- 172.21.15.52:0/2172856294 learned_addr learned my addr 172.21.15.52:0/2172856294 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35090/0 (socket says 172.21.15.52:35090) 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 -- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 msgr2=0x5567dc2d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 -- 172.21.15.52:0/2172856294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2d7670 msgr2=0x5567dc2d4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2d7670 0x5567dc2d4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce7fe00700 1 -- 172.21.15.52:0/2172856294 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567dc16c9d0 con 0x5567dc2cc7c0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.852+0000 7fce85e00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce7fe00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fce700ceb50 tx=0x7fce7009d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8cac5a4adc35b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce6be00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7009d430 con 0x5567dc2cc7c0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7fce7009d690 con 0x5567dc2cc7c0 2024-04-26T01:55:05.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce78009e10 con 0x5567dc2cc7c0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fce700a17e0 con 0x5567dc2cc7c0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce9798db80 10 client.26320 fetch_fsmap learned FSMap version 44 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5567dc1b5bf0 con 0x5567dc2cc7c0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.853+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce78009e10 con 0x5567dc2cc7c0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 44) ==== 8350+0+0 (secure 0 0 0) 0x7fce700b6b20 con 0x5567dc2cc7c0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 10 client.26320 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 10 client.26320 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 20 client.26320 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc2e6c90 0x5567dc2e9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 4 client.26320 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567dc22c780 con 0x5567dc2e6c90 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.854+0000 7fce85e00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc2e6c90 0x5567dc2e9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.855+0000 7fce85e00700 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc2e6c90 0x5567dc2e9080 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ff92137ba39c08 server_cookie=57e8e4b9965e6ae4 in_seq=0 out_seq=0 2024-04-26T01:55:05.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.855+0000 7fce6be00700 10 client.26320 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce6be00700 1 -- 172.21.15.52:0/2172856294 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567dc22c780 con 0x5567dc2e6c90 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce6be00700 10 client.26320 handle_command_reply: tid=0 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 client.26320 shutdown 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 msgr2=0x5567dc2d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fce700ceb50 tx=0x7fce7009d040 comp rx=0 tx=0).stop 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 shutdown_connections 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc2e6c90 0x5567dc2e9080 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2d7670 0x5567dc2d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cbea0 0x5567dc2d6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce6be00700 0 client.26320 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:05.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 --2- 172.21.15.52:0/2172856294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cc7c0 0x5567dc2d4100 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.856+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 >> 172.21.15.52:0/2172856294 conn(0x5567dc1fb050 msgr2=0x5567dc2c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.857+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 shutdown_connections 2024-04-26T01:55:05.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.857+0000 7fce9798db80 1 -- 172.21.15.52:0/2172856294 wait complete. 2024-04-26T01:55:05.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.857+0000 7fce9798db80 20 client.26320 trim_cache size 0 max 16384 2024-04-26T01:55:05.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.878+0000 7fce9798db80 1 Processor -- start 2024-04-26T01:55:05.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.878+0000 7fce9798db80 1 -- start start 2024-04-26T01:55:05.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 0x5567dc2e2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2d7670 0x5567dc2d61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b5ef0 con 0x5567dc2cbea0 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b3750 con 0x5567dc2cc7c0 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce9798db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b15c0 con 0x5567dc2d7670 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce85e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43740/0 (socket says 172.21.15.52:43740) 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce85e00700 1 -- 172.21.15.52:0/1240695605 learned_addr learned my addr 172.21.15.52:0/1240695605 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 0x5567dc2e2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.879+0000 7fce86800700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2d7670 0x5567dc2d61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 -- 172.21.15.52:0/1240695605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2d7670 msgr2=0x5567dc2d61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2d7670 0x5567dc2d61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 -- 172.21.15.52:0/1240695605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 msgr2=0x5567dc2d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 -- 172.21.15.52:0/1240695605 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567dc2e93f0 con 0x5567dc2cc7c0 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce85e00700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce7fe00700 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 0x5567dc2e2fc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fce700b6e90 tx=0x7fce70005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9128969c3acfbd88 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce76a00700 1 -- 172.21.15.52:0/1240695605 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce700ba070 con 0x5567dc2cc7c0 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce76a00700 1 -- 172.21.15.52:0/1240695605 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce700a6010 con 0x5567dc2cc7c0 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.880+0000 7fce76a00700 1 -- 172.21.15.52:0/1240695605 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70022920 con 0x5567dc2cc7c0 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 -- 172.21.15.52:0/1240695605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 msgr2=0x5567dc2e2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 0x5567dc2e2fc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fce700b6e90 tx=0x7fce70005f10 comp rx=0 tx=0).stop 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 -- 172.21.15.52:0/1240695605 shutdown_connections 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2d7670 0x5567dc2d61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cbea0 0x5567dc2d30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 --2- 172.21.15.52:0/1240695605 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2cc7c0 0x5567dc2e2fc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 -- 172.21.15.52:0/1240695605 >> 172.21.15.52:0/1240695605 conn(0x5567dc1fb050 msgr2=0x5567dc2cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 -- 172.21.15.52:0/1240695605 shutdown_connections 2024-04-26T01:55:05.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.881+0000 7fce9798db80 1 -- 172.21.15.52:0/1240695605 wait complete. 2024-04-26T01:55:05.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.882+0000 7fce9798db80 1 -- start start 2024-04-26T01:55:05.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.882+0000 7fce9798db80 1 Processor -- start 2024-04-26T01:55:05.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.882+0000 7fce9798db80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 0x5567dc2dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b1c10 con 0x5567dc2cc7c0 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b3080 con 0x5567dc2d7670 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fce881b22e0 con 0x5567dc2cbea0 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce9798db80 10 client.0 _ll_register_callbacks cb 0x5567dc1b7820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce86800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce85e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce7fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 0x5567dc2dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce85e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35096/0 (socket says 172.21.15.52:35096) 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce86800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2dc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41528/0 (socket says 172.21.15.52:41528) 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.883+0000 7fce85e00700 1 -- 172.21.15.52:0/3926361923 learned_addr learned my addr 172.21.15.52:0/3926361923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 -- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 msgr2=0x5567dc2dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 -- 172.21.15.52:0/3926361923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 msgr2=0x5567dc2dfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 0x5567dc2dfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 -- 172.21.15.52:0/3926361923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567dc2dcd40 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce7fe00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 0x5567dc2dfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce85e00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fce78000c00 tx=0x7fce78009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fda132b84d90e921 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce6be00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7802bb50 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8477+0+0 (secure 0 0 0) 0x7fce7802bcf0 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.884+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce700a6010 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.885+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fce7802b330 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.885+0000 7fce9798db80 10 client.26326 fetch_fsmap learned FSMap version 44 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.885+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5567dc34de20 con 0x5567dc2cbea0 2024-04-26T01:55:05.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.885+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce700a6010 con 0x5567dc2cbea0 2024-04-26T01:55:05.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.885+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 44) ==== 8350+0+0 (secure 0 0 0) 0x7fce78005500 con 0x5567dc2cbea0 2024-04-26T01:55:05.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 10 client.26326 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T01:55:05.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 10 client.26326 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:05.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 20 client.26326 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:05.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc34e420 0x5567dc350840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:05.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 4 client.26326 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "76bc1138", "release": true, "await": true}] 2024-04-26T01:55:05.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "76bc1138", "release": true, "await": true}) -- 0x5567dc226820 con 0x5567dc34e420 2024-04-26T01:55:05.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce7fe00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc34e420 0x5567dc350840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:05.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce7fe00700 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc34e420 0x5567dc350840 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c990e0256a40a4 server_cookie=e47f8e63d2bfcfc7 in_seq=0 out_seq=0 2024-04-26T01:55:05.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.886+0000 7fce6be00700 10 client.26326 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:05.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce6be00700 1 -- 172.21.15.52:0/3926361923 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5567dc226820 con 0x5567dc34e420 2024-04-26T01:55:05.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce6be00700 10 client.26326 handle_command_reply: tid=0 2024-04-26T01:55:05.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 client.26326 shutdown 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 msgr2=0x5567dc2df4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fce78000c00 tx=0x7fce78009260 comp rx=0 tx=0).stop 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 shutdown_connections 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5567dc34e420 0x5567dc350840 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5567dc2d7670 0x5567dc2dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5567dc2cbea0 0x5567dc2df4d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 --2- 172.21.15.52:0/3926361923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5567dc2cc7c0 0x5567dc2dfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 >> 172.21.15.52:0/3926361923 conn(0x5567dc1fb050 msgr2=0x5567dc2cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.897+0000 7fce6be00700 0 client.26326 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.898+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 shutdown_connections 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.898+0000 7fce9798db80 1 -- 172.21.15.52:0/3926361923 wait complete. 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.898+0000 7fce9798db80 20 client.26326 trim_cache size 0 max 16384 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 44, 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:55:05.897 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 28, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "76bc1138": { 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 28, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.1, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:55:05.898 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:05.899 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 msgr2=0x7fce6c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 0x7fce6c0a62f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fce80009ec0 tx=0x7fce80016450 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 msgr2=0x7fce8819fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce70000c00 tx=0x7fce700057c0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 shutdown_connections 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fce6c0a3e30 0x7fce6c0a62f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce8810f030 0x7fce881a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce88108f80 0x7fce8819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 --2- 172.21.15.52:0/1998031589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce88109940 0x7fce8819fb10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 >> 172.21.15.52:0/1998031589 conn(0x7fce880787d0 msgr2=0x7fce88079b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 shutdown_connections 2024-04-26T01:55:05.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:05.899+0000 7fce87200700 1 -- 172.21.15.52:0/1998031589 wait complete. 2024-04-26T01:55:06.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 244995 2.68 MB/sec execute 385 sec latency 30009.605 ms 2024-04-26T01:55:06.845 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '76bc1138', seconds elapsed: 39.4 2024-04-26T01:55:06.846 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.1 2024-04-26T01:55:06.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:55:07.124 INFO:tasks.workunit.client.0.smithi052.stdout: 1 245399 2.67 MB/sec execute 386 sec latency 34.977 ms 2024-04-26T01:55:07.382 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:07.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:07 smithi052 ceph-mon[27851]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 716 KiB/s wr, 62 op/s 2024-04-26T01:55:07.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:07 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:07.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:07 smithi052 ceph-mon[27851]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:55:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:07 smithi191 ceph-mon[32745]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 716 KiB/s wr, 62 op/s 2024-04-26T01:55:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:07 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:07 smithi191 ceph-mon[32745]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:55:07.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:07 smithi183 ceph-mon[32752]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 716 KiB/s wr, 62 op/s 2024-04-26T01:55:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:07 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:55:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:07 smithi183 ceph-mon[32752]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T01:55:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 -- 172.21.15.52:0/4235131520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 msgr2=0x7feb74074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 --2- 172.21.15.52:0/4235131520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb74074700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feb68009900 tx=0x7feb6802a480 comp rx=0 tx=0).stop 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 -- 172.21.15.52:0/4235131520 shutdown_connections 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 --2- 172.21.15.52:0/4235131520 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb7410e610 0x7feb74110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 --2- 172.21.15.52:0/4235131520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb74074c40 0x7feb7410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 --2- 172.21.15.52:0/4235131520 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb74074700 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.917+0000 7feb7a400640 1 -- 172.21.15.52:0/4235131520 >> 172.21.15.52:0/4235131520 conn(0x7feb740fd8a0 msgr2=0x7feb740ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.918+0000 7feb7a400640 1 -- 172.21.15.52:0/4235131520 shutdown_connections 2024-04-26T01:55:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.918+0000 7feb7a400640 1 -- 172.21.15.52:0/4235131520 wait complete. 2024-04-26T01:55:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.919+0000 7feb7a400640 1 Processor -- start 2024-04-26T01:55:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.919+0000 7feb7a400640 1 -- start start 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb741a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 0x7feb7419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feb74114230 con 0x7feb74074c40 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb741140b0 con 0x7feb74074320 2024-04-26T01:55:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb7a400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feb741143b0 con 0x7feb7410e610 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb73e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb73e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43758/0 (socket says 172.21.15.52:43758) 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb73e00640 1 -- 172.21.15.52:0/3046642693 learned_addr learned my addr 172.21.15.52:0/3046642693 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb79a00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 0x7feb7419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.920+0000 7feb79000640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb741a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 -- 172.21.15.52:0/3046642693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 msgr2=0x7feb7419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 0x7feb7419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 -- 172.21.15.52:0/3046642693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 msgr2=0x7feb741a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb741a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 -- 172.21.15.52:0/3046642693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7419aef0 con 0x7feb74074c40 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb79a00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 0x7feb7419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb73e00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d950 tx=0x7feb6000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.921+0000 7feb71600640 1 -- 172.21.15.52:0/3046642693 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60017070 con 0x7feb74074c40 2024-04-26T01:55:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.922+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7419b1e0 con 0x7feb74074c40 2024-04-26T01:55:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.922+0000 7feb71600640 1 -- 172.21.15.52:0/3046642693 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb60009c30 con 0x7feb74074c40 2024-04-26T01:55:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.922+0000 7feb71600640 1 -- 172.21.15.52:0/3046642693 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6000f620 con 0x7feb74074c40 2024-04-26T01:55:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.922+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb741a74f0 con 0x7feb74074c40 2024-04-26T01:55:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.926+0000 7feb71600640 1 -- 172.21.15.52:0/3046642693 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7feb60010460 con 0x7feb74074c40 2024-04-26T01:55:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.927+0000 7feb7e82e740 1 Processor -- start 2024-04-26T01:55:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.927+0000 7feb7e82e740 1 -- start start 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.927+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.927+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.927+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb7e82e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ef3cb10900 con 0x55ef3cbb5b90 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb7e82e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ef3cab6ef0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb7e82e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ef3cb0c8c0 con 0x55ef3cbaa9b0 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35120/0 (socket says 172.21.15.52:35120) 2024-04-26T01:55:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41548/0 (socket says 172.21.15.52:41548) 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 -- 172.21.15.52:0/2012206958 learned_addr learned my addr 172.21.15.52:0/2012206958 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb73e00640 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb71600640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 0x7feb5c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 -- 172.21.15.52:0/2012206958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 msgr2=0x55ef3cbaad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 -- 172.21.15.52:0/2012206958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 msgr2=0x55ef3cbb7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb79000640 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.928+0000 7feb71600640 1 -- 172.21.15.52:0/3046642693 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8565+0+0 (secure 0 0 0) 0x7feb600d03c0 con 0x7feb74074c40 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb79000640 1 -- 172.21.15.52:0/2012206958 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3c823210 con 0x55ef3cbab2d0 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb79a00640 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb79a00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 0x7feb5c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb79000640 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feb68009900 tx=0x7feb68032bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76e0693ab2b95e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb66000640 1 -- 172.21.15.52:0/2012206958 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68040070 con 0x55ef3cbab2d0 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb66000640 1 -- 172.21.15.52:0/2012206958 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb68004030 con 0x55ef3cbab2d0 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.929+0000 7feb66000640 1 -- 172.21.15.52:0/2012206958 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68035040 con 0x55ef3cbab2d0 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.930+0000 7feb7e82e740 1 -- 172.21.15.52:0/2012206958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 msgr2=0x55ef3cbb5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.930+0000 7feb7e82e740 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feb68009900 tx=0x7feb68032bd0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.930+0000 7feb79a00640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 0x7feb5c0a6250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005c40 tx=0x7feb6c004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 -- 172.21.15.52:0/2012206958 shutdown_connections 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb5650 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 --2- 172.21.15.52:0/2012206958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbaad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 -- 172.21.15.52:0/2012206958 >> 172.21.15.52:0/2012206958 conn(0x55ef3cba6ec0 msgr2=0x55ef3cba72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 -- 172.21.15.52:0/2012206958 shutdown_connections 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 -- 172.21.15.52:0/2012206958 wait complete. 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.931+0000 7feb7e82e740 1 -- start start 2024-04-26T01:55:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 Processor -- start 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbb2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ef3cbb8760 con 0x55ef3cbaa9b0 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ef3caca310 con 0x55ef3cbb5b90 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ef3cbb88e0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb7e82e740 10 client.0 _ll_register_callbacks cb 0x55ef3ca98cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb73e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb79a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb79000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbb2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb73e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35126/0 (socket says 172.21.15.52:35126) 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.932+0000 7feb73e00640 1 -- 172.21.15.52:0/3341187630 learned_addr learned my addr 172.21.15.52:0/3341187630 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 -- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbb5b90 msgr2=0x55ef3cbb3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 -- 172.21.15.52:0/3341187630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 msgr2=0x55ef3cbb2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbb2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 -- 172.21.15.52:0/3341187630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3c8ef620 con 0x55ef3cbab2d0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb73e00640 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7feb60000c00 tx=0x7feb6009bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5670ad9225654177 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60020040 con 0x55ef3cbab2d0 2024-04-26T01:55:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb68004030 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.933+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8565+0+0 (secure 0 0 0) 0x7feb600cdd00 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7feb600b5050 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 10 client.26338 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55ef3cbcd180 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb68004030 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7feb600a0240 con 0x55ef3cbab2d0 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 10 client.26338 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 10 client.26338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 20 client.26338 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cbce690 0x55ef3cbd0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 4 client.26338 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.934+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef3cb248b0 con 0x55ef3cbce690 2024-04-26T01:55:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.935+0000 7feb79000640 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cbce690 0x55ef3cbd0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.935+0000 7feb79000640 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cbce690 0x55ef3cbd0aa0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4690bc60763ff69 server_cookie=9768e460f1320de4 in_seq=0 out_seq=0 2024-04-26T01:55:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.935+0000 7feb5aa00640 10 client.26338 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb5aa00640 1 -- 172.21.15.52:0/3341187630 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef3cb248b0 con 0x55ef3cbce690 2024-04-26T01:55:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb5aa00640 10 client.26338 handle_command_reply: tid=0 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 client.26338 shutdown 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 msgr2=0x55ef3cbb3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7feb60000c00 tx=0x7feb6009bda0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 shutdown_connections 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cbce690 0x55ef3cbd0aa0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbb3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbb3010 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3341187630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbb2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.936+0000 7feb5aa00640 0 client.26338 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.937+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 >> 172.21.15.52:0/3341187630 conn(0x55ef3cba6ec0 msgr2=0x55ef3cbacbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.937+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 shutdown_connections 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.937+0000 7feb7e82e740 1 -- 172.21.15.52:0/3341187630 wait complete. 2024-04-26T01:55:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.937+0000 7feb7e82e740 20 client.26338 trim_cache size 0 max 16384 2024-04-26T01:55:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 Processor -- start 2024-04-26T01:55:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 -- start start 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbd0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbd63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feb741ae2d0 con 0x55ef3cbaa9b0 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb741abb40 con 0x55ef3cbab2d0 2024-04-26T01:55:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.957+0000 7feb7e82e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feb741a99b0 con 0x55ef3cbb5b90 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb79000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbd0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41572/0 (socket says 172.21.15.52:41572) 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 -- 172.21.15.52:0/3415624955 learned_addr learned my addr 172.21.15.52:0/3415624955 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb79a00640 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbd63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 -- 172.21.15.52:0/3415624955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbb5b90 msgr2=0x55ef3cbd63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbd63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 -- 172.21.15.52:0/3415624955 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 msgr2=0x55ef3cbd0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbd0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.958+0000 7feb73e00640 1 -- 172.21.15.52:0/3415624955 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3cbb1020 con 0x55ef3cbab2d0 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.959+0000 7feb73e00640 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb60098150 tx=0x7feb6009be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03cea98ddab8151 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.959+0000 7feb65600640 1 -- 172.21.15.52:0/3415624955 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600cdd00 con 0x55ef3cbab2d0 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.959+0000 7feb65600640 1 -- 172.21.15.52:0/3415624955 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb600cdea0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.959+0000 7feb65600640 1 -- 172.21.15.52:0/3415624955 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600a0380 con 0x55ef3cbab2d0 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 -- 172.21.15.52:0/3415624955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 msgr2=0x55ef3cbc9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb60098150 tx=0x7feb6009be10 comp rx=0 tx=0).stop 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 -- 172.21.15.52:0/3415624955 shutdown_connections 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbd63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc9e70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 --2- 172.21.15.52:0/3415624955 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbd0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 -- 172.21.15.52:0/3415624955 >> 172.21.15.52:0/3415624955 conn(0x55ef3cba6ec0 msgr2=0x55ef3cbade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 -- 172.21.15.52:0/3415624955 shutdown_connections 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.960+0000 7feb7e82e740 1 -- 172.21.15.52:0/3415624955 wait complete. 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 -- start start 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 Processor -- start 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbc65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbc3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7feb741aa000 con 0x55ef3cbb5b90 2024-04-26T01:55:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb741ab470 con 0x55ef3cbaa9b0 2024-04-26T01:55:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feb741aa6d0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.961+0000 7feb7e82e740 10 client.0 _ll_register_callbacks cb 0x7feb6009bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35158/0 (socket says 172.21.15.52:35158) 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 -- 172.21.15.52:0/710674522 learned_addr learned my addr 172.21.15.52:0/710674522 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbc65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb79a00640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbc3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 -- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbaa9b0 msgr2=0x55ef3cbc65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbc65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 -- 172.21.15.52:0/710674522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 msgr2=0x55ef3cbc3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbc3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb73e00640 1 -- 172.21.15.52:0/710674522 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3cbc4550 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.962+0000 7feb79a00640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbc3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb73e00640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb6009f000 tx=0x7feb6009bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84ae51868e1db805 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600cdd00 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb600cdea0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8565+0+0 (secure 0 0 0) 0x7feb600cf070 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7feb600a0680 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb7e82e740 10 client.26344 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.963+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55ef3cc34ab0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb600cdea0 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7feb6009b240 con 0x55ef3cbab2d0 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 10 client.26344 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 10 client.26344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 20 client.26344 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cc34f70 0x55ef3cc37390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 4 client.26344 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ef3cb1e320 con 0x55ef3cc34f70 2024-04-26T01:55:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.964+0000 7feb79000640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cc34f70 0x55ef3cc37390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.965+0000 7feb79000640 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cc34f70 0x55ef3cc37390 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e257a1be4530743 server_cookie=779c8ee5306281c7 in_seq=0 out_seq=0 2024-04-26T01:55:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.965+0000 7feb5aa00640 10 client.26344 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb5aa00640 1 -- 172.21.15.52:0/710674522 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ef3cb1e320 con 0x55ef3cc34f70 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb5aa00640 10 client.26344 handle_command_reply: tid=0 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 client.26344 shutdown 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 msgr2=0x55ef3cbc39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feb6009f000 tx=0x7feb6009bda0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 shutdown_connections 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ef3cc34f70 0x55ef3cc37390 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3cbb5b90 0x55ef3cbc3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ef3cbab2d0 0x55ef3cbc39d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 --2- 172.21.15.52:0/710674522 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ef3cbaa9b0 0x55ef3cbc65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb5aa00640 0 client.26344 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 >> 172.21.15.52:0/710674522 conn(0x55ef3cba6ec0 msgr2=0x55ef3cbb2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 shutdown_connections 2024-04-26T01:55:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.993+0000 7feb7e82e740 1 -- 172.21.15.52:0/710674522 wait complete. 2024-04-26T01:55:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.994+0000 7feb7e82e740 20 client.26344 trim_cache size 0 max 16384 2024-04-26T01:55:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:55:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:55:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:55:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.994+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 msgr2=0x7feb5c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.994+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 0x7feb5c0a6250 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb6c005c40 tx=0x7feb6c004a40 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.994+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 msgr2=0x7feb741a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.994+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d950 tx=0x7feb6000de10 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 shutdown_connections 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7feb5c0a3d90 0x7feb5c0a6250 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb7410e610 0x7feb7419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feb74074c40 0x7feb741a0ab0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 --2- 172.21.15.52:0/3046642693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb74074320 0x7feb741a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 >> 172.21.15.52:0/3046642693 conn(0x7feb740fd8a0 msgr2=0x7feb740fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 shutdown_connections 2024-04-26T01:55:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:07.995+0000 7feb7a400640 1 -- 172.21.15.52:0/3046642693 wait complete. 2024-04-26T01:55:08.125 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246445 2.68 MB/sec execute 387 sec latency 28.263 ms 2024-04-26T01:55:09.125 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246478 2.67 MB/sec execute 388 sec latency 968.741 ms 2024-04-26T01:55:09.351 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:09 smithi052 ceph-mon[27851]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 -- 172.21.15.52:0/650194678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f8010e3b0 msgr2=0x7f0f80115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 --2- 172.21.15.52:0/650194678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f8010e3b0 0x7f0f80115f80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009fc0 tx=0x7f0f7800b330 comp rx=0 tx=0).stop 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 -- 172.21.15.52:0/650194678 shutdown_connections 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 --2- 172.21.15.52:0/650194678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f8010e3b0 0x7f0f80115f80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 --2- 172.21.15.52:0/650194678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f801092d0 0x7f0f801096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 --2- 172.21.15.52:0/650194678 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f80109c80 0x7f0f8010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 -- 172.21.15.52:0/650194678 >> 172.21.15.52:0/650194678 conn(0x7f0f80076210 msgr2=0x7f0f80107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 -- 172.21.15.52:0/650194678 shutdown_connections 2024-04-26T01:55:09.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.613+0000 7f0f85200700 1 -- 172.21.15.52:0/650194678 wait complete. 2024-04-26T01:55:09.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.614+0000 7f0f85200700 1 Processor -- start 2024-04-26T01:55:09.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 -- start start 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f801092d0 0x7f0f8019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f8010e3b0 0x7f0f801a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f801192a0 con 0x7f0f80109c80 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80119120 con 0x7f0f8010e3b0 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.615+0000 7f0f85200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80119420 con 0x7f0f801092d0 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7de00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f801092d0 0x7f0f8019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43800/0 (socket says 172.21.15.52:43800) 2024-04-26T01:55:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 -- 172.21.15.52:0/1248141918 learned_addr learned my addr 172.21.15.52:0/1248141918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7e800700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f8010e3b0 0x7f0f801a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 -- 172.21.15.52:0/1248141918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f801092d0 msgr2=0x7f0f8019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f801092d0 0x7f0f8019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 -- 172.21.15.52:0/1248141918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f8010e3b0 msgr2=0x7f0f801a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f8010e3b0 0x7f0f801a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.616+0000 7f0f7d400700 1 -- 172.21.15.52:0/1248141918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801a4730 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f7d400700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00e970 tx=0x7f0f6c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c004a30 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f6c00cc70 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c005580 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f801a4a20 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.617+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f801ac2a0 con 0x7f0f80109c80 2024-04-26T01:55:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.618+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f80069730 con 0x7f0f80109c80 2024-04-26T01:55:09.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.621+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0f6c00c7c0 con 0x7f0f80109c80 2024-04-26T01:55:09.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.623+0000 7f0f76a00700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 0x7f0f640a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:09.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.623+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8565+0+0 (secure 0 0 0) 0x7f0f6c0d0af0 con 0x7f0f80109c80 2024-04-26T01:55:09.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.624+0000 7f0f7de00700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 0x7f0f640a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:09.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.624+0000 7f0f7de00700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 0x7f0f640a62f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005fd0 tx=0x7f0f70009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:09.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.626+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f6c053ff0 con 0x7f0f80109c80 2024-04-26T01:55:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:09 smithi191 ceph-mon[32745]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:09 smithi183 ceph-mon[32752]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:09.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.909+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f801a0ab0 con 0x7f0f80109c80 2024-04-26T01:55:09.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.911+0000 7f0f76a00700 1 -- 172.21.15.52:0/1248141918 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11888 (secure 0 0 0) 0x7f0f801a0ab0 con 0x7f0f80109c80 2024-04-26T01:55:09.910 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:55:09.910 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:55:05.952348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 msgr2=0x7f0f640a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 0x7f0f640a62f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005fd0 tx=0x7f0f70009310 comp rx=0 tx=0).stop 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 msgr2=0x7f0f8019fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00e970 tx=0x7f0f6c00ec80 comp rx=0 tx=0).stop 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 shutdown_connections 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f640a3e30 0x7f0f640a62f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f8010e3b0 0x7f0f801a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f801092d0 0x7f0f8019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 --2- 172.21.15.52:0/1248141918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f80109c80 0x7f0f8019fcb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.914+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 >> 172.21.15.52:0/1248141918 conn(0x7f0f80076210 msgr2=0x7f0f80076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.915+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 shutdown_connections 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:09.915+0000 7f0f85200700 1 -- 172.21.15.52:0/1248141918 wait complete. 2024-04-26T01:55:09.914 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 45 2024-04-26T01:55:10.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246478 2.67 MB/sec execute 389 sec latency 1968.885 ms 2024-04-26T01:55:10.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:10 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1248141918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:10.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:10 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1248141918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:10.685 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:10 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1248141918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:10.762 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0' 2024-04-26T01:55:10.762 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 82.1 --await-for 42.0 2024-04-26T01:55:11.125 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246478 2.66 MB/sec execute 390 sec latency 2968.967 ms 2024-04-26T01:55:11.270 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:11.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:11 smithi052 ceph-mon[27851]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:11 smithi191 ceph-mon[32745]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:11.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:11 smithi183 ceph-mon[32752]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 826 KiB/s wr, 60 op/s 2024-04-26T01:55:12.125 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246478 2.65 MB/sec execute 391 sec latency 3969.034 ms 2024-04-26T01:55:13.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 246927 2.65 MB/sec execute 392 sec latency 4681.399 ms 2024-04-26T01:55:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:13 smithi052 ceph-mon[27851]: pgmap v319: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-26T01:55:13.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:13 smithi052 ceph-mon[27851]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 -- 172.21.15.52:0/848602229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c109c80 msgr2=0x7ff92c10dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 --2- 172.21.15.52:0/848602229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c109c80 0x7ff92c10dcd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff918009990 tx=0x7ff918009ca0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 -- 172.21.15.52:0/848602229 shutdown_connections 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 --2- 172.21.15.52:0/848602229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c10e3b0 0x7ff92c115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 --2- 172.21.15.52:0/848602229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 --2- 172.21.15.52:0/848602229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c109c80 0x7ff92c10dcd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 -- 172.21.15.52:0/848602229 >> 172.21.15.52:0/848602229 conn(0x7ff92c076210 msgr2=0x7ff92c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 -- 172.21.15.52:0/848602229 shutdown_connections 2024-04-26T01:55:13.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.519+0000 7ff933400700 1 -- 172.21.15.52:0/848602229 wait complete. 2024-04-26T01:55:13.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.520+0000 7ff933400700 1 Processor -- start 2024-04-26T01:55:13.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 -- start start 2024-04-26T01:55:13.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c109c80 0x7ff92c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 0x7ff92c1a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c119270 con 0x7ff92c109c80 2024-04-26T01:55:13.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1190f0 con 0x7ff92c10e3b0 2024-04-26T01:55:13.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff933400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1193f0 con 0x7ff92c1092d0 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35200/0 (socket says 172.21.15.52:35200) 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff930a00700 1 -- 172.21.15.52:0/20477490 learned_addr learned my addr 172.21.15.52:0/20477490 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff931400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 0x7ff92c1a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.521+0000 7ff92be00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c109c80 0x7ff92c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 -- 172.21.15.52:0/20477490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 msgr2=0x7ff92c1a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 0x7ff92c1a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 -- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c109c80 msgr2=0x7ff92c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c109c80 0x7ff92c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 -- 172.21.15.52:0/20477490 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c1a4690 con 0x7ff92c1092d0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff931400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 0x7ff92c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.522+0000 7ff930a00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff92000d980 tx=0x7ff92000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.523+0000 7ff929600700 1 -- 172.21.15.52:0/20477490 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920016070 con 0x7ff92c1092d0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.523+0000 7ff929600700 1 -- 172.21.15.52:0/20477490 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff920004830 con 0x7ff92c1092d0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.523+0000 7ff929600700 1 -- 172.21.15.52:0/20477490 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920005540 con 0x7ff92c1092d0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.523+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c1a4980 con 0x7ff92c1092d0 2024-04-26T01:55:13.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.523+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c1ac200 con 0x7ff92c1092d0 2024-04-26T01:55:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.526+0000 7ff929600700 1 -- 172.21.15.52:0/20477490 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff920004220 con 0x7ff92c1092d0 2024-04-26T01:55:13.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.528+0000 7ff929600700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 0x7ff9140a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.528+0000 7ff929600700 1 -- 172.21.15.52:0/20477490 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7ff9200d0600 con 0x7ff92c1092d0 2024-04-26T01:55:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.528+0000 7ff93ddb1b80 1 Processor -- start 2024-04-26T01:55:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.528+0000 7ff92be00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 0x7ff9140a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 -- start start 2024-04-26T01:55:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecda950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cfdec20020 con 0x55cfdecd8560 2024-04-26T01:55:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cfdebc68c0 con 0x55cfdeccd6f0 2024-04-26T01:55:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cfdec1bfe0 con 0x55cfdecccdd0 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.529+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35216/0 (socket says 172.21.15.52:35216) 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 -- 172.21.15.52:0/705510540 learned_addr learned my addr 172.21.15.52:0/705510540 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff931400700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecda950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff92be00700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 -- 172.21.15.52:0/705510540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 msgr2=0x55cfdecd8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 -- 172.21.15.52:0/705510540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 msgr2=0x55cfdecda950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecda950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 -- 172.21.15.52:0/705510540 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfdebbc810 con 0x55cfdecccdd0 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff931400700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecda950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff92be00700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 0x7ff9140a6310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff92c10f630 tx=0x7ff918005fd0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.530+0000 7ff930a00700 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff9200a4c80 tx=0x7ff920004660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=390b52d6c8aab4f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.531+0000 7ff91e000700 1 -- 172.21.15.52:0/705510540 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b5070 con 0x55cfdecccdd0 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.531+0000 7ff91e000700 1 -- 172.21.15.52:0/705510540 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9200a08f0 con 0x55cfdecccdd0 2024-04-26T01:55:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.531+0000 7ff91e000700 1 -- 172.21.15.52:0/705510540 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92009ca00 con 0x55cfdecccdd0 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.531+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/705510540 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 msgr2=0x55cfdeccd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.531+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff9200a4c80 tx=0x7ff920004660 comp rx=0 tx=0).stop 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/705510540 shutdown_connections 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecda950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdeccd1b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/705510540 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/705510540 >> 172.21.15.52:0/705510540 conn(0x55cfdebfc0e0 msgr2=0x55cfdecc92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/705510540 shutdown_connections 2024-04-26T01:55:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/705510540 wait complete. 2024-04-26T01:55:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.532+0000 7ff93ddb1b80 1 -- start start 2024-04-26T01:55:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 Processor -- start 2024-04-26T01:55:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecd55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cfdecdb050 con 0x55cfdecd8560 2024-04-26T01:55:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cfdebd99f0 con 0x55cfdeccd6f0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cfdecdb1d0 con 0x55cfdecccdd0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.533+0000 7ff93ddb1b80 10 client.0 _ll_register_callbacks cb 0x55cfdebb88b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35232/0 (socket says 172.21.15.52:35232) 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff92be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 -- 172.21.15.52:0/2958118438 learned_addr learned my addr 172.21.15.52:0/2958118438 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff931400700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecd55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 -- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 msgr2=0x55cfdecd50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 -- 172.21.15.52:0/2958118438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 msgr2=0x55cfdecd55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecd55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff930a00700 1 -- 172.21.15.52:0/2958118438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfdeb6da60 con 0x55cfdecccdd0 2024-04-26T01:55:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff92be00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.534+0000 7ff931400700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecd55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff930a00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff920098f10 tx=0x7ff920004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2f230e0e908ace8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff912a00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b9030 con 0x55cfdecccdd0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7ff9200049e0 con 0x55cfdecccdd0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9200a08f0 con 0x55cfdecccdd0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff9200b5040 con 0x55cfdecccdd0 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff93ddb1b80 10 client.26362 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cfdebb6c80 con 0x55cfdecccdd0 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.535+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9200a08f0 con 0x55cfdecccdd0 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7ff9200d00c0 con 0x55cfdecccdd0 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 10 client.26362 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 10 client.26362 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 20 client.26362 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfdece7ce0 0x55cfdecea0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 4 client.26362 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfdec2d810 con 0x55cfdece7ce0 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.536+0000 7ff92be00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfdece7ce0 0x55cfdecea0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.537+0000 7ff92be00700 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfdece7ce0 0x55cfdecea0d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55b1bae8b806a0da server_cookie=55f2112d36c49e69 in_seq=0 out_seq=0 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.537+0000 7ff912a00700 10 client.26362 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.537+0000 7ff912a00700 1 -- 172.21.15.52:0/2958118438 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfdec2d810 con 0x55cfdece7ce0 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.537+0000 7ff912a00700 10 client.26362 handle_command_reply: tid=0 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.537+0000 7ff93ddb1b80 1 client.26362 shutdown 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 msgr2=0x55cfdecd7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff920098f10 tx=0x7ff920004fb0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 shutdown_connections 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfdece7ce0 0x55cfdecea0d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfdecd55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecd7da0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff912a00700 0 client.26362 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/2958118438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 >> 172.21.15.52:0/2958118438 conn(0x55cfdebfc0e0 msgr2=0x55cfdecc8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 shutdown_connections 2024-04-26T01:55:13.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/2958118438 wait complete. 2024-04-26T01:55:13.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.538+0000 7ff93ddb1b80 20 client.26362 trim_cache size 0 max 16384 2024-04-26T01:55:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 Processor -- start 2024-04-26T01:55:13.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 -- start start 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecd8560 0x55cfdeced100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b6230 con 0x55cfdeccd6f0 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b3a60 con 0x55cfdecd8560 2024-04-26T01:55:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.561+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b18d0 con 0x55cfdecccdd0 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff92be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff931400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecd8560 0x55cfdeced100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35246/0 (socket says 172.21.15.52:35246) 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 -- 172.21.15.52:0/1346440858 learned_addr learned my addr 172.21.15.52:0/1346440858 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 -- 172.21.15.52:0/1346440858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecd8560 msgr2=0x55cfdeced100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecd8560 0x55cfdeced100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 -- 172.21.15.52:0/1346440858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdeccd6f0 msgr2=0x55cfdecd43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 -- 172.21.15.52:0/1346440858 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfdeccec40 con 0x55cfdecccdd0 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff930a00700 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff92009c4d0 tx=0x7ff920004660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=866e5646d20d5950 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.562+0000 7ff91d600700 1 -- 172.21.15.52:0/1346440858 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b9020 con 0x55cfdecccdd0 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff91d600700 1 -- 172.21.15.52:0/1346440858 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9200049e0 con 0x55cfdecccdd0 2024-04-26T01:55:13.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff91d600700 1 -- 172.21.15.52:0/1346440858 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200bb2c0 con 0x55cfdecccdd0 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/1346440858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 msgr2=0x55cfdecc94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff92009c4d0 tx=0x7ff920004660 comp rx=0 tx=0).stop 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/1346440858 shutdown_connections 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecd8560 0x55cfdeced100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdecccdd0 0x55cfdecc94a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/1346440858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdeccd6f0 0x55cfdecd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.563+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/1346440858 >> 172.21.15.52:0/1346440858 conn(0x55cfdebfc0e0 msgr2=0x55cfdece3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.564+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/1346440858 shutdown_connections 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.564+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/1346440858 wait complete. 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.564+0000 7ff93ddb1b80 1 -- start start 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.564+0000 7ff93ddb1b80 1 Processor -- start 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfded41e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b1f20 con 0x55cfdecd8560 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b3390 con 0x55cfdecccdd0 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1b25f0 con 0x55cfdeccd6f0 2024-04-26T01:55:13.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff93ddb1b80 10 client.0 _ll_register_callbacks cb 0x55cfdebb88b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff930a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff92be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff930a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41662/0 (socket says 172.21.15.52:41662) 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff92be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35248/0 (socket says 172.21.15.52:35248) 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff930a00700 1 -- 172.21.15.52:0/30039954 learned_addr learned my addr 172.21.15.52:0/30039954 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.565+0000 7ff931400700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfded41e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 -- 172.21.15.52:0/30039954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 msgr2=0x55cfded47b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 -- 172.21.15.52:0/30039954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 msgr2=0x55cfded41e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfded41e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 -- 172.21.15.52:0/30039954 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfded42690 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff92be00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff931400700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfded41e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff930a00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff920004530 tx=0x7ff920004f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d9c39eca0eab1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff912a00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.566+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b9070 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7ff9200d00f0 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9200049e0 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff9200ad050 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff93ddb1b80 10 client.26369 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cfded42690 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.567+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9200049e0 con 0x55cfdecccdd0 2024-04-26T01:55:13.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7ff92009b4e0 con 0x55cfdecccdd0 2024-04-26T01:55:13.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 10 client.26369 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:13.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 10 client.26369 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:13.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 20 client.26369 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:13.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfded4f080 0x55cfded514a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:13.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 4 client.26369 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}] 2024-04-26T01:55:13.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.1, "await_for": 42.0}) -- 0x55cfdec278b0 con 0x55cfded4f080 2024-04-26T01:55:13.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.568+0000 7ff92be00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfded4f080 0x55cfded514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:13.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.569+0000 7ff92be00700 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfded4f080 0x55cfded514a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a81c491f119ab997 server_cookie=97a1dc46288426f6 in_seq=0 out_seq=0 2024-04-26T01:55:13.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.569+0000 7ff912a00700 10 client.26369 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff912a00700 1 -- 172.21.15.52:0/30039954 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55cfdec278b0 con 0x55cfded4f080 2024-04-26T01:55:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff912a00700 10 client.26369 handle_command_reply: tid=0 2024-04-26T01:55:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 client.26369 shutdown 2024-04-26T01:55:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 msgr2=0x55cfded47610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff920004530 tx=0x7ff920004f10 comp rx=0 tx=0).stop 2024-04-26T01:55:13.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 shutdown_connections 2024-04-26T01:55:13.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cfded4f080 0x55cfded514a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cfdecd8560 0x55cfded41e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cfdecccdd0 0x55cfded47610 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 --2- 172.21.15.52:0/30039954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cfdeccd6f0 0x55cfded47b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff912a00700 0 client.26369 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 >> 172.21.15.52:0/30039954 conn(0x55cfdebfc0e0 msgr2=0x55cfdecc8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 shutdown_connections 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.627+0000 7ff93ddb1b80 1 -- 172.21.15.52:0/30039954 wait complete. 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.628+0000 7ff93ddb1b80 20 client.26369 trim_cache size 0 max 16384 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 45, 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:55:13.627 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 30, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "cc1e02e0": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 30, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.1, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:13.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.628+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 msgr2=0x7ff9140a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.628+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 0x7ff9140a6310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff92c10f630 tx=0x7ff918005fd0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 msgr2=0x7ff92c19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff92000d980 tx=0x7ff92000dc90 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 shutdown_connections 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff9140a3e50 0x7ff9140a6310 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff92c10e3b0 0x7ff92c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c1092d0 0x7ff92c19f680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 --2- 172.21.15.52:0/20477490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff92c109c80 0x7ff92c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 >> 172.21.15.52:0/20477490 conn(0x7ff92c076210 msgr2=0x7ff92c076730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 shutdown_connections 2024-04-26T01:55:13.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:13.629+0000 7ff933400700 1 -- 172.21.15.52:0/20477490 wait complete. 2024-04-26T01:55:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:13 smithi191 ceph-mon[32745]: pgmap v319: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-26T01:55:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:13 smithi191 ceph-mon[32745]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:55:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:13 smithi183 ceph-mon[32752]: pgmap v319: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-26T01:55:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:13 smithi183 ceph-mon[32752]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T01:55:14.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.65 MB/sec execute 393 sec latency 520.786 ms 2024-04-26T01:55:14.605 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'cc1e02e0', quiesce took 7.8 seconds. Will release after: 22.1 2024-04-26T01:55:15.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.64 MB/sec execute 394 sec latency 1520.951 ms 2024-04-26T01:55:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:15 smithi191 ceph-mon[32745]: pgmap v321: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 81 op/s 2024-04-26T01:55:15.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:15 smithi183 ceph-mon[32752]: pgmap v321: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 81 op/s 2024-04-26T01:55:15.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:15 smithi052 ceph-mon[27851]: pgmap v321: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 81 op/s 2024-04-26T01:55:16.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.63 MB/sec execute 395 sec latency 2521.092 ms 2024-04-26T01:55:17.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.63 MB/sec execute 396 sec latency 3521.212 ms 2024-04-26T01:55:17.515 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:17 smithi052 ceph-mon[27851]: pgmap v322: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 973 KiB/s wr, 88 op/s 2024-04-26T01:55:17.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:17 smithi191 ceph-mon[32745]: pgmap v322: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 973 KiB/s wr, 88 op/s 2024-04-26T01:55:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:17 smithi183 ceph-mon[32752]: pgmap v322: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 973 KiB/s wr, 88 op/s 2024-04-26T01:55:18.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.62 MB/sec execute 397 sec latency 4521.305 ms 2024-04-26T01:55:19.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.61 MB/sec execute 398 sec latency 5521.435 ms 2024-04-26T01:55:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:19 smithi191 ceph-mon[32745]: pgmap v323: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:19 smithi183 ceph-mon[32752]: pgmap v323: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:19 smithi052 ceph-mon[27851]: pgmap v323: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:20.126 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.61 MB/sec execute 399 sec latency 6521.560 ms 2024-04-26T01:55:21.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.60 MB/sec execute 400 sec latency 7521.688 ms 2024-04-26T01:55:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 -- 172.21.15.52:0/1869806637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048103830 msgr2=0x7fa04810bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 --2- 172.21.15.52:0/1869806637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048103830 0x7fa04810bd10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa034009860 tx=0x7fa03402a3d0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 -- 172.21.15.52:0/1869806637 shutdown_connections 2024-04-26T01:55:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 --2- 172.21.15.52:0/1869806637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa04810c330 0x7fa04810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 --2- 172.21.15.52:0/1869806637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048103830 0x7fa04810bd10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 --2- 172.21.15.52:0/1869806637 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa048102f10 0x7fa0481032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 -- 172.21.15.52:0/1869806637 >> 172.21.15.52:0/1869806637 conn(0x7fa0480fc770 msgr2=0x7fa0480feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 -- 172.21.15.52:0/1869806637 shutdown_connections 2024-04-26T01:55:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.144+0000 7fa04e800640 1 -- 172.21.15.52:0/1869806637 wait complete. 2024-04-26T01:55:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 Processor -- start 2024-04-26T01:55:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 -- start start 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 0x7fa048100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 0x7fa0481010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04810c330 0x7fa0481015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa048112010 con 0x7fa048103830 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa048111e90 con 0x7fa048102f10 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.145+0000 7fa04e800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa048112190 con 0x7fa04810c330 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04ca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 0x7fa0481010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 0x7fa048100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04ca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 0x7fa0481010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33670/0 (socket says 172.21.15.52:33670) 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04ca00640 1 -- 172.21.15.52:0/1796892596 learned_addr learned my addr 172.21.15.52:0/1796892596 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04de00640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04810c330 0x7fa0481015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 -- 172.21.15.52:0/1796892596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04810c330 msgr2=0x7fa0481015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04810c330 0x7fa0481015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 -- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 msgr2=0x7fa0481010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 0x7fa0481010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 -- 172.21.15.52:0/1796892596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0481a6ab0 con 0x7fa048102f10 2024-04-26T01:55:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.146+0000 7fa04d400640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 0x7fa048100b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa03800ef80 tx=0x7fa03800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.147+0000 7fa046000640 1 -- 172.21.15.52:0/1796892596 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038003970 con 0x7fa048102f10 2024-04-26T01:55:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.147+0000 7fa046000640 1 -- 172.21.15.52:0/1796892596 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa038004020 con 0x7fa048102f10 2024-04-26T01:55:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.147+0000 7fa046000640 1 -- 172.21.15.52:0/1796892596 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03800b960 con 0x7fa048102f10 2024-04-26T01:55:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.147+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0481a6da0 con 0x7fa048102f10 2024-04-26T01:55:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.147+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0481a7260 con 0x7fa048102f10 2024-04-26T01:55:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa052c3c740 1 Processor -- start 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa052c3c740 1 -- start start 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa046000640 1 -- 172.21.15.52:0/1796892596 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa03801d060 con 0x7fa048102f10 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.150+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b126ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa052c3c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5637b11c8900 con 0x5637b12629b0 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa052c3c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637b116eef0 con 0x5637b12632d0 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa052c3c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637b11c48c0 con 0x5637b126db90 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33684/0 (socket says 172.21.15.52:33684) 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 -- 172.21.15.52:0/1311425227 learned_addr learned my addr 172.21.15.52:0/1311425227 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04d400640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04ca00640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b126ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 -- 172.21.15.52:0/1311425227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 msgr2=0x5637b126ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b126ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 -- 172.21.15.52:0/1311425227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 msgr2=0x5637b126d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04de00640 1 -- 172.21.15.52:0/1311425227 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b0edb210 con 0x5637b12629b0 2024-04-26T01:55:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.151+0000 7fa04d400640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa04de00640 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c20 tx=0x7fa040011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abbd0a6d6122f89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa03e600640 1 -- 172.21.15.52:0/1311425227 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04001e070 con 0x5637b12629b0 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa03e600640 1 -- 172.21.15.52:0/1311425227 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040019ae0 con 0x5637b12629b0 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa03e600640 1 -- 172.21.15.52:0/1311425227 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04000e900 con 0x5637b12629b0 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa046000640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 0x7fa02c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa046000640 1 -- 172.21.15.52:0/1796892596 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7fa03801b070 con 0x7fa048102f10 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.152+0000 7fa04d400640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 0x7fa02c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- 172.21.15.52:0/1311425227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 msgr2=0x5637b1262d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c20 tx=0x7fa040011810 comp rx=0 tx=0).stop 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- 172.21.15.52:0/1311425227 shutdown_connections 2024-04-26T01:55:21.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b126ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1311425227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1262d90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- 172.21.15.52:0/1311425227 >> 172.21.15.52:0/1311425227 conn(0x5637b125eec0 msgr2=0x5637b125f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- 172.21.15.52:0/1311425227 shutdown_connections 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- 172.21.15.52:0/1311425227 wait complete. 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa04d400640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 0x7fa02c0a6250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5637b1263f50 tx=0x7fa03809c340 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.153+0000 7fa052c3c740 1 -- start start 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 Processor -- start 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b126aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b126db90 0x5637b126b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5637b12707c0 con 0x5637b12632d0 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637b1182310 con 0x5637b126db90 2024-04-26T01:55:21.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637b1270940 con 0x5637b12629b0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa052c3c740 10 client.0 _ll_register_callbacks cb 0x5637b1150cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa04ca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa04ca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33690/0 (socket says 172.21.15.52:33690) 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.154+0000 7fa04ca00640 1 -- 172.21.15.52:0/4112889936 learned_addr learned my addr 172.21.15.52:0/4112889936 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04de00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b126aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04de00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b126db90 0x5637b126b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 -- 172.21.15.52:0/4112889936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 msgr2=0x5637b126aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b126aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 -- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b126db90 msgr2=0x5637b126b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b126db90 0x5637b126b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 -- 172.21.15.52:0/4112889936 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b0fa7620 con 0x5637b12632d0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa034000c00 tx=0x7fa034030bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d084bdb8b377a735 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa032a00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:55:21.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.155+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034004050 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa040019ae0 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7fa0340041f0 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa0340043f0 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa052c3c740 10 client.16245 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5637b127c980 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040019ae0 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fa0340047e0 con 0x5637b12632d0 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa052c3c740 10 client.16245 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.156+0000 7fa052c3c740 10 client.16245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa052c3c740 20 client.16245 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b127df50 0x5637b1280390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa052c3c740 4 client.16245 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637b11dc8b0 con 0x5637b127df50 2024-04-26T01:55:21.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa04de00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b127df50 0x5637b1280390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa04de00640 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b127df50 0x5637b1280390 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=573327ce62298d3c server_cookie=33ae6e1c32c5220d in_seq=0 out_seq=0 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.157+0000 7fa032a00640 10 client.16245 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.158+0000 7fa032a00640 1 -- 172.21.15.52:0/4112889936 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637b11dc8b0 con 0x5637b127df50 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.158+0000 7fa032a00640 10 client.16245 handle_command_reply: tid=0 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.158+0000 7fa052c3c740 1 client.16245 shutdown 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 msgr2=0x5637b126af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa034000c00 tx=0x7fa034030bb0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 shutdown_connections 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b127df50 0x5637b1280390 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b126db90 0x5637b126b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12632d0 0x5637b126af70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa032a00640 0 client.16245 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4112889936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b126aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 >> 172.21.15.52:0/4112889936 conn(0x5637b125eec0 msgr2=0x5637b1264c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 shutdown_connections 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 1 -- 172.21.15.52:0/4112889936 wait complete. 2024-04-26T01:55:21.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.159+0000 7fa052c3c740 20 client.16245 trim_cache size 0 max 16384 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 Processor -- start 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 -- start start 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b1285ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b24c0 con 0x5637b12629b0 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481afd30 con 0x5637b12632d0 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.175+0000 7fa052c3c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481adba0 con 0x5637b126db90 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33694/0 (socket says 172.21.15.52:33694) 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04de00640 1 -- 172.21.15.52:0/4245948037 learned_addr learned my addr 172.21.15.52:0/4245948037 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04de00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b1285ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 -- 172.21.15.52:0/4245948037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 msgr2=0x5637b1285ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b1285ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 -- 172.21.15.52:0/4245948037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 msgr2=0x5637b1279680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04ca00640 1 -- 172.21.15.52:0/4245948037 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637b125ecf0 con 0x5637b12632d0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.176+0000 7fa04de00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa04ca00640 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126b2d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa034005e00 tx=0x7fa034004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b60cf29d400baa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa03dc00640 1 -- 172.21.15.52:0/4245948037 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0340035f0 con 0x5637b12632d0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa03dc00640 1 -- 172.21.15.52:0/4245948037 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa034004e70 con 0x5637b12632d0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa03dc00640 1 -- 172.21.15.52:0/4245948037 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034003b70 con 0x5637b12632d0 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa052c3c740 1 -- 172.21.15.52:0/4245948037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 msgr2=0x5637b126b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126b2d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa034005e00 tx=0x7fa034004590 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa052c3c740 1 -- 172.21.15.52:0/4245948037 shutdown_connections 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b126db90 0x5637b1285ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.177+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b126b2d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 --2- 172.21.15.52:0/4245948037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b12629b0 0x5637b1279680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 -- 172.21.15.52:0/4245948037 >> 172.21.15.52:0/4245948037 conn(0x5637b125eec0 msgr2=0x5637b1269320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 -- 172.21.15.52:0/4245948037 shutdown_connections 2024-04-26T01:55:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 -- 172.21.15.52:0/4245948037 wait complete. 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 -- start start 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.178+0000 7fa052c3c740 1 Processor -- start 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b1275d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b126db90 0x5637b1272c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481ae1f0 con 0x5637b126db90 2024-04-26T01:55:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481af660 con 0x5637b12632d0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481ae8c0 con 0x5637b12629b0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa052c3c740 10 client.0 _ll_register_callbacks cb 0x7fa0340096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:52664/0 (socket says 172.21.15.52:52664) 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 -- 172.21.15.52:0/1882493416 learned_addr learned my addr 172.21.15.52:0/1882493416 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04ca00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b1275d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b126db90 0x5637b1272c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 -- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 msgr2=0x5637b1275d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b1275d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 -- 172.21.15.52:0/1882493416 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b126db90 msgr2=0x5637b1272c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b126db90 0x5637b1272c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.179+0000 7fa04de00640 1 -- 172.21.15.52:0/1882493416 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637b1273480 con 0x5637b12629b0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa04de00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa0400098e0 tx=0x7fa040030040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f3245c01e3bc7ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa032a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04001e040 con 0x5637b12629b0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa034004e70 con 0x5637b12629b0 2024-04-26T01:55:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7fa040019430 con 0x5637b12629b0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa040019630 con 0x5637b12629b0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa052c3c740 10 client.26386 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.180+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5637b12e3f50 con 0x5637b12629b0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa034004e70 con 0x5637b12629b0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fa040003b60 con 0x5637b12629b0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 10 client.26386 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 10 client.26386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 20 client.26386 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b12e43d0 0x5637b12e67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 4 client.26386 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5637b11d6320 con 0x5637b12e43d0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.181+0000 7fa04ca00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b12e43d0 0x5637b12e67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.182+0000 7fa04ca00640 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b12e43d0 0x5637b12e67c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca819e4eddb7015 server_cookie=6efaa4bfd6c39995 in_seq=0 out_seq=0 2024-04-26T01:55:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.182+0000 7fa032a00640 10 client.26386 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa032a00640 1 -- 172.21.15.52:0/1882493416 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5637b11d6320 con 0x5637b12e43d0 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa032a00640 10 client.26386 handle_command_reply: tid=0 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 client.26386 shutdown 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 msgr2=0x5637b12757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa0400098e0 tx=0x7fa040030040 comp rx=0 tx=0).stop 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 shutdown_connections 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637b12e43d0 0x5637b12e67c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637b126db90 0x5637b1272c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa032a00640 0 client.26386 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637b12632d0 0x5637b1275d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 --2- 172.21.15.52:0/1882493416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637b12629b0 0x5637b12757d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 >> 172.21.15.52:0/1882493416 conn(0x5637b125eec0 msgr2=0x5637b125d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.212+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 shutdown_connections 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa052c3c740 1 -- 172.21.15.52:0/1882493416 wait complete. 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa052c3c740 20 client.26386 trim_cache size 0 max 16384 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:55:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 msgr2=0x7fa02c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 0x7fa02c0a6250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5637b1263f50 tx=0x7fa03809c340 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 msgr2=0x7fa048100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.213+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 0x7fa048100b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa03800ef80 tx=0x7fa03800c800 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 shutdown_connections 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa02c0a3d90 0x7fa02c0a6250 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa04810c330 0x7fa0481015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa048103830 0x7fa0481010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 --2- 172.21.15.52:0/1796892596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa048102f10 0x7fa048100b60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 >> 172.21.15.52:0/1796892596 conn(0x7fa0480fc770 msgr2=0x7fa0480fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 shutdown_connections 2024-04-26T01:55:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:21.214+0000 7fa04e800640 1 -- 172.21.15.52:0/1796892596 wait complete. 2024-04-26T01:55:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:21 smithi191 ceph-mon[32745]: pgmap v324: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:21 smithi183 ceph-mon[32752]: pgmap v324: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:21 smithi052 ceph-mon[27851]: pgmap v324: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 961 KiB/s wr, 87 op/s 2024-04-26T01:55:22.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.59 MB/sec execute 401 sec latency 8521.789 ms 2024-04-26T01:55:23.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.59 MB/sec execute 402 sec latency 9521.966 ms 2024-04-26T01:55:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:23 smithi191 ceph-mon[32745]: pgmap v325: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 170 KiB/s wr, 28 op/s 2024-04-26T01:55:23.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:23 smithi183 ceph-mon[32752]: pgmap v325: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 170 KiB/s wr, 28 op/s 2024-04-26T01:55:23.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:23 smithi052 ceph-mon[27851]: pgmap v325: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 170 KiB/s wr, 28 op/s 2024-04-26T01:55:24.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.58 MB/sec execute 403 sec latency 10522.131 ms 2024-04-26T01:55:25.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.58 MB/sec execute 404 sec latency 11522.276 ms 2024-04-26T01:55:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:25 smithi191 ceph-mon[32745]: pgmap v326: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 156 KiB/s wr, 26 op/s 2024-04-26T01:55:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:25 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:25.678 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:25 smithi183 ceph-mon[32752]: pgmap v326: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 156 KiB/s wr, 26 op/s 2024-04-26T01:55:25.678 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:25 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:25 smithi052 ceph-mon[27851]: pgmap v326: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 156 KiB/s wr, 26 op/s 2024-04-26T01:55:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:25 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:55:26.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.57 MB/sec execute 405 sec latency 12522.366 ms 2024-04-26T01:55:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:26 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:26 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:26 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:26 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:26 smithi183 ceph-mon[32752]: pgmap v327: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 230 KiB/s wr, 34 op/s 2024-04-26T01:55:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:26 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:26 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:26 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:26 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:26 smithi052 ceph-mon[27851]: pgmap v327: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 230 KiB/s wr, 34 op/s 2024-04-26T01:55:27.125 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:26 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:27.125 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:26 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:27.125 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:26 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:27.125 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:26 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:27.125 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:26 smithi191 ceph-mon[32745]: pgmap v327: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 230 KiB/s wr, 34 op/s 2024-04-26T01:55:27.127 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.56 MB/sec execute 406 sec latency 13522.434 ms 2024-04-26T01:55:28.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.56 MB/sec execute 407 sec latency 14522.545 ms 2024-04-26T01:55:28.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:28 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:28 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:28.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:28 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:29.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.55 MB/sec execute 408 sec latency 15522.672 ms 2024-04-26T01:55:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:29 smithi191 ceph-mon[32745]: pgmap v328: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:29 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:55:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:29 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:55:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:29 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:55:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:29 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:55:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:29 smithi183 ceph-mon[32752]: pgmap v328: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:29 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:55:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:29 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:55:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:29 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:55:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:29 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:55:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:29 smithi052 ceph-mon[27851]: pgmap v328: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:29 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T01:55:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:29 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T01:55:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:29 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T01:55:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:29 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T01:55:30.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.54 MB/sec execute 409 sec latency 16522.877 ms 2024-04-26T01:55:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:30.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:30.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:30 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:30.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:30 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:55:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:30 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:55:31.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.54 MB/sec execute 410 sec latency 17523.031 ms 2024-04-26T01:55:31.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:31 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:55:31.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:31 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:55:31.804 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:31 smithi191 ceph-mon[32745]: pgmap v329: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:31 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:55:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:31 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:55:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:31 smithi183 ceph-mon[32752]: pgmap v329: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:31 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T01:55:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:31 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T01:55:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:31 smithi052 ceph-mon[27851]: pgmap v329: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 122 KiB/s wr, 15 op/s 2024-04-26T01:55:32.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.53 MB/sec execute 411 sec latency 18523.123 ms 2024-04-26T01:55:33.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.53 MB/sec execute 412 sec latency 19523.282 ms 2024-04-26T01:55:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:33 smithi191 ceph-mon[32745]: pgmap v330: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 128 KiB/s wr, 19 op/s 2024-04-26T01:55:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:33 smithi183 ceph-mon[32752]: pgmap v330: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 128 KiB/s wr, 19 op/s 2024-04-26T01:55:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:33 smithi052 ceph-mon[27851]: pgmap v330: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 128 KiB/s wr, 19 op/s 2024-04-26T01:55:34.128 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.52 MB/sec execute 413 sec latency 20523.420 ms 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 -- 172.21.15.52:0/1717888470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c100d60 msgr2=0x7eff1c101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 --2- 172.21.15.52:0/1717888470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c100d60 0x7eff1c101140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7eff04009900 tx=0x7eff0402a400 comp rx=0 tx=0).stop 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 -- 172.21.15.52:0/1717888470 shutdown_connections 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 --2- 172.21.15.52:0/1717888470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c10e910 0x7eff1c110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 --2- 172.21.15.52:0/1717888470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 0x7eff1c10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 --2- 172.21.15.52:0/1717888470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c100d60 0x7eff1c101140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 -- 172.21.15.52:0/1717888470 >> 172.21.15.52:0/1717888470 conn(0x7eff1c077fd0 msgr2=0x7eff1c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 -- 172.21.15.52:0/1717888470 shutdown_connections 2024-04-26T01:55:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.362+0000 7eff21c00640 1 -- 172.21.15.52:0/1717888470 wait complete. 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.363+0000 7eff21c00640 1 Processor -- start 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.363+0000 7eff21c00640 1 -- start start 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 0x7eff1c1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c10e910 0x7eff1c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c114420 con 0x7eff1c101680 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1142a0 con 0x7eff1c10e910 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff21c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1145a0 con 0x7eff1c100d60 2024-04-26T01:55:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff1aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 0x7eff1c1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff1aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54202/0 (socket says 172.21.15.52:54202) 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff1aa00640 1 -- 172.21.15.52:0/769383331 learned_addr learned my addr 172.21.15.52:0/769383331 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.364+0000 7eff1b400640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c10e910 0x7eff1c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 -- 172.21.15.52:0/769383331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c10e910 msgr2=0x7eff1c19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c10e910 0x7eff1c19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 -- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 msgr2=0x7eff1c1a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 0x7eff1c1a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 -- 172.21.15.52:0/769383331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c19afd0 con 0x7eff1c100d60 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff1aa00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7eff04005ab0 tx=0x7eff04003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff13400640 1 -- 172.21.15.52:0/769383331 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0403f070 con 0x7eff1c100d60 2024-04-26T01:55:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff13400640 1 -- 172.21.15.52:0/769383331 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff04031b60 con 0x7eff1c100d60 2024-04-26T01:55:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff13400640 1 -- 172.21.15.52:0/769383331 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff04035030 con 0x7eff1c100d60 2024-04-26T01:55:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c19b260 con 0x7eff1c100d60 2024-04-26T01:55:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.365+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c1a74e0 con 0x7eff1c100d60 2024-04-26T01:55:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 Processor -- start 2024-04-26T01:55:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff13400640 1 -- 172.21.15.52:0/769383331 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7eff04042070 con 0x7eff1c100d60 2024-04-26T01:55:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 -- start start 2024-04-26T01:55:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x5621061704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x562106172df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5621060cb900 con 0x562106170a00 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562106071ef0 con 0x562106166140 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.369+0000 7eff24bec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5621060c78c0 con 0x562106165820 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x562106172df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54204/0 (socket says 172.21.15.52:54204) 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 -- 172.21.15.52:0/2387498068 learned_addr learned my addr 172.21.15.52:0/2387498068 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1aa00640 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x5621061704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 -- 172.21.15.52:0/2387498068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 msgr2=0x5621061704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x5621061704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 -- 172.21.15.52:0/2387498068 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 msgr2=0x562106172df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x562106172df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.370+0000 7eff1b400640 1 -- 172.21.15.52:0/2387498068 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562105dde210 con 0x562106165820 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff1b400640 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7eff14009ba0 tx=0x7eff14011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=704f43268647d813 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff13400640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 0x7eff000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff0be00640 1 -- 172.21.15.52:0/2387498068 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14004220 con 0x562106165820 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff0be00640 1 -- 172.21.15.52:0/2387498068 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff14009e10 con 0x562106165820 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff0be00640 1 -- 172.21.15.52:0/2387498068 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14019690 con 0x562106165820 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff13400640 1 -- 172.21.15.52:0/769383331 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7eff040f0e00 con 0x7eff1c100d60 2024-04-26T01:55:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.371+0000 7eff1aa00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 0x7eff000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- 172.21.15.52:0/2387498068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 msgr2=0x562106165c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7eff14009ba0 tx=0x7eff14011810 comp rx=0 tx=0).stop 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff1aa00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 0x7eff000a6300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x562106166dc0 tx=0x7eff04009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- 172.21.15.52:0/2387498068 shutdown_connections 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x562106172df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x5621061704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 --2- 172.21.15.52:0/2387498068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x562106165c00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- 172.21.15.52:0/2387498068 >> 172.21.15.52:0/2387498068 conn(0x562106161d30 msgr2=0x562106162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- 172.21.15.52:0/2387498068 shutdown_connections 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- 172.21.15.52:0/2387498068 wait complete. 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.372+0000 7eff24bec740 1 -- start start 2024-04-26T01:55:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 Processor -- start 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x56210616d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562106173630 con 0x562106170a00 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562106085310 con 0x562106166140 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5621061737b0 con 0x562106165820 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff24bec740 10 client.0 _ll_register_callbacks cb 0x562106053cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff1b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff1a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff1b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34704/0 (socket says 172.21.15.52:34704) 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff1a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51494/0 (socket says 172.21.15.52:51494) 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.373+0000 7eff1b400640 1 -- 172.21.15.52:0/2958217614 learned_addr learned my addr 172.21.15.52:0/2958217614 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x56210616d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 -- 172.21.15.52:0/2958217614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 msgr2=0x56210616d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x56210616d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 -- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 msgr2=0x56210616def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 -- 172.21.15.52:0/2958217614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562105eaa620 con 0x562106170a00 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1a000640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff1b400640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff140045c0 tx=0x7eff14003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8448bbc7f51ff30c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.374+0000 7eff08c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14028450 con 0x562106170a00 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7eff140285f0 con 0x562106170a00 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff14009e10 con 0x562106170a00 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7eff14036050 con 0x562106170a00 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff24bec740 10 client.16263 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56210617f8f0 con 0x562106170a00 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.375+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff14009e10 con 0x562106170a00 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7eff140277d0 con 0x562106170a00 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 10 client.16263 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 20 client.16263 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562106180cf0 0x5621061830e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 4 client.16263 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621060df8b0 con 0x562106180cf0 2024-04-26T01:55:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.376+0000 7eff1a000640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562106180cf0 0x5621061830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.377+0000 7eff1a000640 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562106180cf0 0x5621061830e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c35725680f6b5af server_cookie=b6e3f873860bb4ad in_seq=0 out_seq=0 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.377+0000 7eff08c00640 10 client.16263 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff08c00640 1 -- 172.21.15.52:0/2958217614 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621060df8b0 con 0x562106180cf0 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff08c00640 10 client.16263 handle_command_reply: tid=0 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 client.16263 shutdown 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 msgr2=0x56210616e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff140045c0 tx=0x7eff14003dc0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 shutdown_connections 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562106180cf0 0x5621061830e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106170a00 0x56210616e430 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106166140 0x56210616def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 --2- 172.21.15.52:0/2958217614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106165820 0x56210616d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 >> 172.21.15.52:0/2958217614 conn(0x562106161d30 msgr2=0x562106167a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff08c00640 0 client.16263 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.378+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 shutdown_connections 2024-04-26T01:55:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.379+0000 7eff24bec740 1 -- 172.21.15.52:0/2958217614 wait complete. 2024-04-26T01:55:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.379+0000 7eff24bec740 20 client.16263 trim_cache size 0 max 16384 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 Processor -- start 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 -- start start 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106165820 0x56210617c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106170a00 0x562106188de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1b2690 con 0x562106166140 2024-04-26T01:55:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1aff00 con 0x562106165820 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.395+0000 7eff24bec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1add70 con 0x562106170a00 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34712/0 (socket says 172.21.15.52:34712) 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 -- 172.21.15.52:0/2180581970 learned_addr learned my addr 172.21.15.52:0/2180581970 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1b400640 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106165820 0x56210617c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1b400640 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106170a00 0x562106188de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 -- 172.21.15.52:0/2180581970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106170a00 msgr2=0x562106188de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106170a00 0x562106188de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 -- 172.21.15.52:0/2180581970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106165820 msgr2=0x56210617c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106165820 0x56210617c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.396+0000 7eff1a000640 1 -- 172.21.15.52:0/2180581970 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562106183450 con 0x562106166140 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.397+0000 7eff1a000640 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005fd0 tx=0x7eff0c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd64533e159bdb07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.397+0000 7eff0b400640 1 -- 172.21.15.52:0/2180581970 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c025070 con 0x562106166140 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.397+0000 7eff0b400640 1 -- 172.21.15.52:0/2180581970 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff0c01ce10 con 0x562106166140 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.397+0000 7eff0b400640 1 -- 172.21.15.52:0/2180581970 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c005020 con 0x562106166140 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- 172.21.15.52:0/2180581970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 msgr2=0x5621061888a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005fd0 tx=0x7eff0c009500 comp rx=0 tx=0).stop 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- 172.21.15.52:0/2180581970 shutdown_connections 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106170a00 0x562106188de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106166140 0x5621061888a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 --2- 172.21.15.52:0/2180581970 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106165820 0x56210617c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- 172.21.15.52:0/2180581970 >> 172.21.15.52:0/2180581970 conn(0x562106161d30 msgr2=0x562106162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- 172.21.15.52:0/2180581970 shutdown_connections 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- 172.21.15.52:0/2180581970 wait complete. 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 -- start start 2024-04-26T01:55:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.398+0000 7eff24bec740 1 Processor -- start 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 0x562106178c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106170a00 0x562106175bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ae3c0 con 0x562106165820 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1af830 con 0x562106170a00 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1aea90 con 0x562106166140 2024-04-26T01:55:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff24bec740 10 client.0 _ll_register_callbacks cb 0x7eff140044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff1b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff1b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34714/0 (socket says 172.21.15.52:34714) 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff1b400640 1 -- 172.21.15.52:0/3132055631 learned_addr learned my addr 172.21.15.52:0/3132055631 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff1a000640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 0x562106178c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.399+0000 7eff1b400640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106170a00 0x562106175bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 -- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106170a00 msgr2=0x562106175bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106170a00 0x562106175bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 -- 172.21.15.52:0/3132055631 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 msgr2=0x562106178720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 -- 172.21.15.52:0/3132055631 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562106176390 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1b400640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff1a000640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 0x562106178c60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5621061709d0 tx=0x7eff0c020210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884ec39134da1246 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff08c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c02d020 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7eff0c0205d0 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.400+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff0c01ce10 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7eff0c038050 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 10 client.26404 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5621061e6b20 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff0c01ce10 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7eff0c005320 con 0x562106166140 2024-04-26T01:55:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 10 client.26404 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 20 client.26404 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5621061e7320 0x5621061e9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 4 client.26404 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.401+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621060d9320 con 0x5621061e7320 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.402+0000 7eff1b400640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5621061e7320 0x5621061e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.402+0000 7eff1b400640 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5621061e7320 0x5621061e9740 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328a33dad47cc572 server_cookie=69cbd28861a869b9 in_seq=0 out_seq=0 2024-04-26T01:55:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.402+0000 7eff08c00640 10 client.26404 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff08c00640 1 -- 172.21.15.52:0/3132055631 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621060d9320 con 0x5621061e7320 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff08c00640 10 client.26404 handle_command_reply: tid=0 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 client.26404 shutdown 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 msgr2=0x562106178c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 0x562106178c60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5621061709d0 tx=0x7eff0c020210 comp rx=0 tx=0).stop 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 shutdown_connections 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5621061e7320 0x5621061e9740 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562106170a00 0x562106175bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562106166140 0x562106178c60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 --2- 172.21.15.52:0/3132055631 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562106165820 0x562106178720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff08c00640 0 client.26404 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 >> 172.21.15.52:0/3132055631 conn(0x562106161d30 msgr2=0x56210616d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 shutdown_connections 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 1 -- 172.21.15.52:0/3132055631 wait complete. 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.424+0000 7eff24bec740 20 client.26404 trim_cache size 0 max 16384 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:55:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:55:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 msgr2=0x7eff000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 0x7eff000a6300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x562106166dc0 tx=0x7eff04009260 comp rx=0 tx=0).stop 2024-04-26T01:55:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 msgr2=0x7eff1c1a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7eff04005ab0 tx=0x7eff04003c10 comp rx=0 tx=0).stop 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 shutdown_connections 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7eff000a3e40 0x7eff000a6300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff1c10e910 0x7eff1c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7eff1c101680 0x7eff1c1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 --2- 172.21.15.52:0/769383331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff1c100d60 0x7eff1c1a0690 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.425+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 >> 172.21.15.52:0/769383331 conn(0x7eff1c077fd0 msgr2=0x7eff1c10fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.426+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 shutdown_connections 2024-04-26T01:55:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:34.426+0000 7eff21c00640 1 -- 172.21.15.52:0/769383331 wait complete. 2024-04-26T01:55:35.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.51 MB/sec execute 414 sec latency 21523.581 ms 2024-04-26T01:55:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:35 smithi191 ceph-mon[32745]: pgmap v331: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-26T01:55:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:35 smithi183 ceph-mon[32752]: pgmap v331: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-26T01:55:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:35 smithi052 ceph-mon[27851]: pgmap v331: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-26T01:55:36.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.51 MB/sec execute 415 sec latency 22523.711 ms 2024-04-26T01:55:36.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:55:37.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.50 MB/sec execute 416 sec latency 23523.804 ms 2024-04-26T01:55:37.198 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:37.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:37 smithi052 ceph-mon[27851]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 115 KiB/s wr, 21 op/s 2024-04-26T01:55:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:37 smithi191 ceph-mon[32745]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 115 KiB/s wr, 21 op/s 2024-04-26T01:55:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:37 smithi183 ceph-mon[32752]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 115 KiB/s wr, 21 op/s 2024-04-26T01:55:38.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.50 MB/sec execute 417 sec latency 24523.907 ms 2024-04-26T01:55:39.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.49 MB/sec execute 418 sec latency 25523.991 ms 2024-04-26T01:55:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 -- 172.21.15.52:0/2756220601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874108a30 msgr2=0x7fa874108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 --2- 172.21.15.52:0/2756220601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874108a30 0x7fa874108e10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa8680098f0 tx=0x7fa868009c00 comp rx=0 tx=0).stop 2024-04-26T01:55:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 -- 172.21.15.52:0/2756220601 shutdown_connections 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 --2- 172.21.15.52:0/2756220601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa87406b0f0 0x7fa87410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 --2- 172.21.15.52:0/2756220601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874108a30 0x7fa874108e10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 --2- 172.21.15.52:0/2756220601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406a750 0x7fa87406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 -- 172.21.15.52:0/2756220601 >> 172.21.15.52:0/2756220601 conn(0x7fa8740788f0 msgr2=0x7fa874078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 -- 172.21.15.52:0/2756220601 shutdown_connections 2024-04-26T01:55:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.418+0000 7fa87b400700 1 -- 172.21.15.52:0/2756220601 wait complete. 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.419+0000 7fa87b400700 1 Processor -- start 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.419+0000 7fa87b400700 1 -- start start 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 0x7fa87419b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa874108a30 0x7fa87419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa874112c00 con 0x7fa874108a30 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa874112a80 con 0x7fa87406b0f0 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa87b400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa874112d80 con 0x7fa87406a750 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa878a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa878a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55238/0 (socket says 172.21.15.52:55238) 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa878a00700 1 -- 172.21.15.52:0/3609315734 learned_addr learned my addr 172.21.15.52:0/3609315734 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:39.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.420+0000 7fa873e00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 0x7fa87419b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa879400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa874108a30 0x7fa87419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 -- 172.21.15.52:0/3609315734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 msgr2=0x7fa87419b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 0x7fa87419b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 -- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa874108a30 msgr2=0x7fa87419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa874108a30 0x7fa87419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 -- 172.21.15.52:0/3609315734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8741a03a0 con 0x7fa87406a750 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa873e00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 0x7fa87419b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.421+0000 7fa878a00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa868009230 tx=0x7fa86800fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.422+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868005220 con 0x7fa87406a750 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.422+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86800b560 con 0x7fa87406a750 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.422+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8741a0630 con 0x7fa87406a750 2024-04-26T01:55:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.422+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868003b40 con 0x7fa87406a750 2024-04-26T01:55:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.422+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8741a7f10 con 0x7fa87406a750 2024-04-26T01:55:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.423+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa874069730 con 0x7fa87406a750 2024-04-26T01:55:39.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.425+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa868003d00 con 0x7fa87406a750 2024-04-26T01:55:39.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.427+0000 7fa871600700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 0x7fa85c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:39.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.427+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7fa8680d1810 con 0x7fa87406a750 2024-04-26T01:55:39.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.427+0000 7fa873e00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 0x7fa85c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:39.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.428+0000 7fa873e00700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 0x7fa85c0a62f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa874079030 tx=0x7fa860009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:39.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.431+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa868013030 con 0x7fa87406a750 2024-04-26T01:55:39.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:39 smithi052 ceph-mon[27851]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:39.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.717+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa87419c6f0 con 0x7fa87406a750 2024-04-26T01:55:39.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.719+0000 7fa871600700 1 -- 172.21.15.52:0/3609315734 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11888 (secure 0 0 0) 0x7fa86809dee0 con 0x7fa87406a750 2024-04-26T01:55:39.718 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:55:39.718 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:55:05.952348+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:55:39.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 msgr2=0x7fa85c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:39.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 0x7fa85c0a62f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa874079030 tx=0x7fa860009260 comp rx=0 tx=0).stop 2024-04-26T01:55:39.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 msgr2=0x7fa87419b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa868009230 tx=0x7fa86800fb20 comp rx=0 tx=0).stop 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 shutdown_connections 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa85c0a3e30 0x7fa85c0a62f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa874108a30 0x7fa87419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa87406a750 0x7fa87419b390 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 --2- 172.21.15.52:0/3609315734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa87406b0f0 0x7fa87419b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.722+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 >> 172.21.15.52:0/3609315734 conn(0x7fa8740788f0 msgr2=0x7fa87410df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.723+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 shutdown_connections 2024-04-26T01:55:39.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:39.723+0000 7fa87b400700 1 -- 172.21.15.52:0/3609315734 wait complete. 2024-04-26T01:55:39.722 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 45 2024-04-26T01:55:39.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:39 smithi191 ceph-mon[32745]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:39 smithi183 ceph-mon[32752]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:40.129 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.48 MB/sec execute 419 sec latency 26524.072 ms 2024-04-26T01:55:40.600 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id cc1e02e0 --release --await' 2024-04-26T01:55:40.601 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id cc1e02e0 --release --await 2024-04-26T01:55:40.813 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:40 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3609315734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:40.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:40 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3609315734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:40.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:40 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3609315734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:55:41.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:55:41.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.48 MB/sec execute 420 sec latency 27524.640 ms 2024-04-26T01:55:41.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:41 smithi191 ceph-mon[32745]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:41 smithi183 ceph-mon[32752]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:41 smithi052 ceph-mon[27851]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-26T01:55:42.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.47 MB/sec execute 421 sec latency 28524.756 ms 2024-04-26T01:55:43.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 247395 2.47 MB/sec execute 422 sec latency 29524.896 ms 2024-04-26T01:55:43.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 -- 172.21.15.52:0/3900727769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48105890 msgr2=0x7f6a481098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 --2- 172.21.15.52:0/3900727769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48105890 0x7f6a481098e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6a340099c0 tx=0x7f6a34009cd0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 -- 172.21.15.52:0/3900727769 shutdown_connections 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 --2- 172.21.15.52:0/3900727769 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48109fc0 0x7f6a48111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 --2- 172.21.15.52:0/3900727769 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48104ee0 0x7f6a481052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 --2- 172.21.15.52:0/3900727769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48105890 0x7f6a481098e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 -- 172.21.15.52:0/3900727769 >> 172.21.15.52:0/3900727769 conn(0x7f6a48100770 msgr2=0x7f6a48102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 -- 172.21.15.52:0/3900727769 shutdown_connections 2024-04-26T01:55:43.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.287+0000 7f6a4d800700 1 -- 172.21.15.52:0/3900727769 wait complete. 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.288+0000 7f6a4d800700 1 Processor -- start 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 -- start start 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 0x7f6a4819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 0x7f6a4819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6a48114ea0 con 0x7f6a48109fc0 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6a48114d20 con 0x7f6a48104ee0 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a4d800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6a48115020 con 0x7f6a48105890 2024-04-26T01:55:43.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a47400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a47400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34164/0 (socket says 172.21.15.52:34164) 2024-04-26T01:55:43.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a47400700 1 -- 172.21.15.52:0/2960734859 learned_addr learned my addr 172.21.15.52:0/2960734859 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:43.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.289+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 0x7f6a4819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a46000700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 0x7f6a4819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 -- 172.21.15.52:0/2960734859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 msgr2=0x7f6a4819b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 0x7f6a4819b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 -- 172.21.15.52:0/2960734859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 msgr2=0x7f6a4819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 0x7f6a4819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 -- 172.21.15.52:0/2960734859 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a481a03a0 con 0x7f6a48109fc0 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a46000700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 0x7f6a4819b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 0x7f6a4819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:43.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.290+0000 7f6a47400700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000c370 tx=0x7f6a4000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.291+0000 7f6a3f400700 1 -- 172.21.15.52:0/2960734859 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a400092e0 con 0x7f6a48109fc0 2024-04-26T01:55:43.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.291+0000 7f6a3f400700 1 -- 172.21.15.52:0/2960734859 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a40003a00 con 0x7f6a48109fc0 2024-04-26T01:55:43.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.291+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a481a0630 con 0x7f6a48109fc0 2024-04-26T01:55:43.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.291+0000 7f6a3f400700 1 -- 172.21.15.52:0/2960734859 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a400046f0 con 0x7f6a48109fc0 2024-04-26T01:55:43.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.291+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a481a7f10 con 0x7f6a48109fc0 2024-04-26T01:55:43.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.294+0000 7f6a3f400700 1 -- 172.21.15.52:0/2960734859 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6a4001a020 con 0x7f6a48109fc0 2024-04-26T01:55:43.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.296+0000 7f6a3f400700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 0x7f6a2c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.296+0000 7f6a58270b80 1 Processor -- start 2024-04-26T01:55:43.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.296+0000 7f6a3f400700 1 -- 172.21.15.52:0/2960734859 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f6a400cffe0 con 0x7f6a48109fc0 2024-04-26T01:55:43.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.296+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 0x7f6a2c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.296+0000 7f6a58270b80 1 -- start start 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8db120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8eece0 0x557c0a8f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557c0a82df90 con 0x557c0a8db660 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557c0a7d4830 con 0x557c0a8dad40 2024-04-26T01:55:43.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a58270b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557c0a829f50 con 0x557c0a8eece0 2024-04-26T01:55:43.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34176/0 (socket says 172.21.15.52:34176) 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 -- 172.21.15.52:0/2526628334 learned_addr learned my addr 172.21.15.52:0/2526628334 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 0x7f6a2c0a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a38011490 tx=0x7f6a38009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a47400700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8eece0 0x557c0a8f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 -- 172.21.15.52:0/2526628334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8eece0 msgr2=0x557c0a8f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8eece0 0x557c0a8f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 -- 172.21.15.52:0/2526628334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 msgr2=0x557c0a8db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8db120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.297+0000 7f6a46000700 1 -- 172.21.15.52:0/2526628334 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0a7ca780 con 0x557c0a8db660 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.298+0000 7f6a46a00700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8db120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.298+0000 7f6a46000700 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6a340099c0 tx=0x7f6a340036e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788567cd1aafac20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.298+0000 7f6a33e00700 1 -- 172.21.15.52:0/2526628334 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34028070 con 0x557c0a8db660 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.298+0000 7f6a33e00700 1 -- 172.21.15.52:0/2526628334 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a34005570 con 0x557c0a8db660 2024-04-26T01:55:43.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.298+0000 7f6a33e00700 1 -- 172.21.15.52:0/2526628334 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34003c40 con 0x557c0a8db660 2024-04-26T01:55:43.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 -- 172.21.15.52:0/2526628334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 msgr2=0x557c0a8ee7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6a340099c0 tx=0x7f6a340036e0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 -- 172.21.15.52:0/2526628334 shutdown_connections 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8eece0 0x557c0a8f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8db120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 --2- 172.21.15.52:0/2526628334 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8db660 0x557c0a8ee7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 -- 172.21.15.52:0/2526628334 >> 172.21.15.52:0/2526628334 conn(0x557c0a80a050 msgr2=0x557c0a8d7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 -- 172.21.15.52:0/2526628334 shutdown_connections 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.299+0000 7f6a58270b80 1 -- 172.21.15.52:0/2526628334 wait complete. 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.300+0000 7f6a58270b80 1 -- start start 2024-04-26T01:55:43.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.300+0000 7f6a58270b80 1 Processor -- start 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557c0a8f1770 con 0x557c0a8eece0 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557c0a7e7960 con 0x557c0a8dad40 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557c0a8f18f0 con 0x557c0a8db660 2024-04-26T01:55:43.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a58270b80 10 client.0 _ll_register_callbacks cb 0x557c0a7c6820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a46000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a46a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a46000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55286/0 (socket says 172.21.15.52:55286) 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a46a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:40710/0 (socket says 172.21.15.52:40710) 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.301+0000 7f6a46000700 1 -- 172.21.15.52:0/1595840042 learned_addr learned my addr 172.21.15.52:0/1595840042 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a47400700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 -- 172.21.15.52:0/1595840042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 msgr2=0x557c0a8eb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 -- 172.21.15.52:0/1595840042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 msgr2=0x557c0a8ebbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8ebbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 -- 172.21.15.52:0/1595840042 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c0a77b9d0 con 0x557c0a8dad40 2024-04-26T01:55:43.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46000700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:55:43.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6a38011be0 tx=0x7f6a380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e7807925a92dad0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.302+0000 7f6a30c00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3801f8d0 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f6a3801fa70 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a34005570 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a3801fc70 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a58270b80 10 client.26423 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x557c0a7c4bf0 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.303+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a34005570 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f6a38028470 con 0x557c0a8dad40 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 10 client.26423 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 10 client.26423 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 20 client.26423 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a8fe3a0 0x557c0a9007b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 4 client.26423 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c0a83b780 con 0x557c0a8fe3a0 2024-04-26T01:55:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.304+0000 7f6a46000700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a8fe3a0 0x557c0a9007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.305+0000 7f6a46000700 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a8fe3a0 0x557c0a9007b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a872153011d469d9 server_cookie=108889ffa615f4f7 in_seq=0 out_seq=0 2024-04-26T01:55:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.305+0000 7f6a30c00700 10 client.26423 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.306+0000 7f6a30c00700 1 -- 172.21.15.52:0/1595840042 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c0a83b780 con 0x557c0a8fe3a0 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.306+0000 7f6a30c00700 10 client.26423 handle_command_reply: tid=0 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.306+0000 7f6a58270b80 1 client.26423 shutdown 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.306+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 msgr2=0x557c0a8ee380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.306+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6a38011be0 tx=0x7f6a380047c0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 shutdown_connections 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a8fe3a0 0x557c0a9007b0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8dad40 0x557c0a8ee380 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a30c00700 0 client.26423 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:43.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1595840042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 >> 172.21.15.52:0/1595840042 conn(0x557c0a80a050 msgr2=0x557c0a8d6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 shutdown_connections 2024-04-26T01:55:43.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 1 -- 172.21.15.52:0/1595840042 wait complete. 2024-04-26T01:55:43.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.307+0000 7f6a58270b80 20 client.26423 trim_cache size 0 max 16384 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.331+0000 7f6a58270b80 1 Processor -- start 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.331+0000 7f6a58270b80 1 -- start start 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.331+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.331+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8db660 0x557c0a8ea620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a58270b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6a481ada10 con 0x557c0a8eece0 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a58270b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6a481ab240 con 0x557c0a8db660 2024-04-26T01:55:43.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a58270b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4807b050 con 0x557c0a8dad40 2024-04-26T01:55:43.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55292/0 (socket says 172.21.15.52:55292) 2024-04-26T01:55:43.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 -- 172.21.15.52:0/1804593627 learned_addr learned my addr 172.21.15.52:0/1804593627 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46000700 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8db660 0x557c0a8ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a47400700 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 -- 172.21.15.52:0/1804593627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8db660 msgr2=0x557c0a8ea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8db660 0x557c0a8ea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 -- 172.21.15.52:0/1804593627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 msgr2=0x557c0a8e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.332+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.333+0000 7f6a46a00700 1 -- 172.21.15.52:0/1804593627 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0a8dc750 con 0x557c0a8dad40 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.333+0000 7f6a46a00700 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a38000c00 tx=0x7f6a38004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddb44ecd5a9a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.333+0000 7f6a33400700 1 -- 172.21.15.52:0/1804593627 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38028cb0 con 0x557c0a8dad40 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.333+0000 7f6a33400700 1 -- 172.21.15.52:0/1804593627 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a38023070 con 0x557c0a8dad40 2024-04-26T01:55:43.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.333+0000 7f6a33400700 1 -- 172.21.15.52:0/1804593627 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38031d80 con 0x557c0a8dad40 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 -- 172.21.15.52:0/1804593627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 msgr2=0x557c0a8d7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a38000c00 tx=0x7f6a38004530 comp rx=0 tx=0).stop 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 -- 172.21.15.52:0/1804593627 shutdown_connections 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8eece0 0x557c0a8e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8dad40 0x557c0a8d7410 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 --2- 172.21.15.52:0/1804593627 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8db660 0x557c0a8ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 -- 172.21.15.52:0/1804593627 >> 172.21.15.52:0/1804593627 conn(0x557c0a80a050 msgr2=0x557c0a8e6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.334+0000 7f6a58270b80 1 -- 172.21.15.52:0/1804593627 shutdown_connections 2024-04-26T01:55:43.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.335+0000 7f6a58270b80 1 -- 172.21.15.52:0/1804593627 wait complete. 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.335+0000 7f6a58270b80 1 -- start start 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.335+0000 7f6a58270b80 1 Processor -- start 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8dad40 0x557c0a8f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8eece0 0x557c0a8f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6a481a9720 con 0x557c0a8dad40 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6a481aab70 con 0x557c0a8eece0 2024-04-26T01:55:43.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6a481a9dd0 con 0x557c0a8db660 2024-04-26T01:55:43.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a58270b80 10 client.0 _ll_register_callbacks cb 0x557c0a7c6820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:43.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a46000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a46a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8dad40 0x557c0a8f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a46000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55300/0 (socket says 172.21.15.52:55300) 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a46000700 1 -- 172.21.15.52:0/964709727 learned_addr learned my addr 172.21.15.52:0/964709727 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.336+0000 7f6a47400700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8eece0 0x557c0a8f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 -- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8eece0 msgr2=0x557c0a8f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8eece0 0x557c0a8f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 -- 172.21.15.52:0/964709727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8dad40 msgr2=0x557c0a8f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8dad40 0x557c0a8f1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 -- 172.21.15.52:0/964709727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c0a964a50 con 0x557c0a8db660 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a46000700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400b580 tx=0x7f6a3400b5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4764df67417b989a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:43.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.337+0000 7f6a30c00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34037060 con 0x557c0a8db660 2024-04-26T01:55:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f6a34003d70 con 0x557c0a8db660 2024-04-26T01:55:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a38023070 con 0x557c0a8db660 2024-04-26T01:55:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a34028040 con 0x557c0a8db660 2024-04-26T01:55:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a58270b80 10 client.26428 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x557c0a965d90 con 0x557c0a8db660 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a38023070 con 0x557c0a8db660 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.338+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f6a34031630 con 0x557c0a8db660 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 10 client.26428 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 10 client.26428 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 20 client.26428 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a966060 0x557c0a968450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 4 client.26428 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "cc1e02e0", "release": true, "await": true}] 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "cc1e02e0", "release": true, "await": true}) -- 0x557c0a835820 con 0x557c0a966060 2024-04-26T01:55:43.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.339+0000 7f6a46a00700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a966060 0x557c0a968450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:43.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.340+0000 7f6a46a00700 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a966060 0x557c0a968450 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a36140d3fe61789 server_cookie=b2f31b03a11b5e42 in_seq=0 out_seq=0 2024-04-26T01:55:43.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.340+0000 7f6a30c00700 10 client.26428 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a30c00700 1 -- 172.21.15.52:0/964709727 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557c0a835820 con 0x557c0a966060 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a30c00700 10 client.26428 handle_command_reply: tid=0 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 client.26428 shutdown 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 msgr2=0x557c0a8f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400b580 tx=0x7f6a3400b5b0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 shutdown_connections 2024-04-26T01:55:43.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557c0a966060 0x557c0a968450 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557c0a8eece0 0x557c0a8f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557c0a8dad40 0x557c0a8f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 --2- 172.21.15.52:0/964709727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557c0a8db660 0x557c0a8f20b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.344+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 >> 172.21.15.52:0/964709727 conn(0x557c0a80a050 msgr2=0x557c0a8eceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.345+0000 7f6a30c00700 0 client.26428 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.345+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 shutdown_connections 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.345+0000 7f6a58270b80 1 -- 172.21.15.52:0/964709727 wait complete. 2024-04-26T01:55:43.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.345+0000 7f6a58270b80 20 client.26428 trim_cache size 0 max 16384 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 45, 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 32, 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "cc1e02e0": { 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 32, 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:55:43.345 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.1, 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:43.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 msgr2=0x7f6a2c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 0x7f6a2c0a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a38011490 tx=0x7f6a38009260 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 msgr2=0x7f6a4819fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6a4000c370 tx=0x7f6a4000c680 comp rx=0 tx=0).stop 2024-04-26T01:55:43.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 shutdown_connections 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6a2c0a3db0 0x7f6a2c0a6270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a48109fc0 0x7f6a4819fc60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6a48104ee0 0x7f6a4819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.346+0000 7f6a4d800700 1 --2- 172.21.15.52:0/2960734859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a48105890 0x7f6a4819b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.347+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 >> 172.21.15.52:0/2960734859 conn(0x7f6a48100770 msgr2=0x7f6a48102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.347+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 shutdown_connections 2024-04-26T01:55:43.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:55:43.347+0000 7f6a4d800700 1 -- 172.21.15.52:0/2960734859 wait complete. 2024-04-26T01:55:43.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:43 smithi052 ceph-mon[27851]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-04-26T01:55:43.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:43 smithi191 ceph-mon[32745]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-04-26T01:55:43.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:43 smithi183 ceph-mon[32752]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-04-26T01:55:44.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 248419 2.48 MB/sec execute 423 sec latency 29740.952 ms 2024-04-26T01:55:44.345 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'cc1e02e0', seconds elapsed: 37.5 2024-04-26T01:55:45.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 248740 2.48 MB/sec execute 424 sec latency 24.991 ms 2024-04-26T01:55:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:45 smithi191 ceph-mon[32745]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 133 KiB/s wr, 18 op/s 2024-04-26T01:55:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:45 smithi183 ceph-mon[32752]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 133 KiB/s wr, 18 op/s 2024-04-26T01:55:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:45 smithi052 ceph-mon[27851]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 133 KiB/s wr, 18 op/s 2024-04-26T01:55:46.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 249310 2.48 MB/sec execute 425 sec latency 25.808 ms 2024-04-26T01:55:47.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 250021 2.48 MB/sec execute 426 sec latency 27.108 ms 2024-04-26T01:55:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 -- 172.21.15.52:0/1198974697 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1998102f10 msgr2=0x7f19981032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 --2- 172.21.15.52:0/1198974697 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1998102f10 0x7f19981032f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1990009ca0 tx=0x7f199002a950 comp rx=0 tx=0).stop 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 -- 172.21.15.52:0/1198974697 shutdown_connections 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 --2- 172.21.15.52:0/1198974697 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f199810c330 0x7f199810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 --2- 172.21.15.52:0/1198974697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998103830 0x7f199810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 --2- 172.21.15.52:0/1198974697 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1998102f10 0x7f19981032f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 -- 172.21.15.52:0/1198974697 >> 172.21.15.52:0/1198974697 conn(0x7f19980fc770 msgr2=0x7f19980feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 -- 172.21.15.52:0/1198974697 shutdown_connections 2024-04-26T01:55:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.579+0000 7f199f200640 1 -- 172.21.15.52:0/1198974697 wait complete. 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.580+0000 7f199f200640 1 Processor -- start 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.580+0000 7f199f200640 1 -- start start 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998102f10 0x7f19981a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f199810c330 0x7f199819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1998111ff0 con 0x7f199810c330 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1998111e70 con 0x7f1998103830 2024-04-26T01:55:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199f200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1998112170 con 0x7f1998102f10 2024-04-26T01:55:47.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f199810c330 0x7f199819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998102f10 0x7f19981a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53358/0 (socket says 172.21.15.52:53358) 2024-04-26T01:55:47.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 -- 172.21.15.52:0/875117097 learned_addr learned my addr 172.21.15.52:0/875117097 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 -- 172.21.15.52:0/875117097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998102f10 msgr2=0x7f19981a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998102f10 0x7f19981a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 -- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f199810c330 msgr2=0x7f199819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f199810c330 0x7f199819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.581+0000 7f199d400640 1 -- 172.21.15.52:0/875117097 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199819adb0 con 0x7f1998103830 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f199d400640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f198800e9f0 tx=0x7f198800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f198ea00640 1 -- 172.21.15.52:0/875117097 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198800ccc0 con 0x7f1998103830 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f198ea00640 1 -- 172.21.15.52:0/875117097 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f198800ce60 con 0x7f1998103830 2024-04-26T01:55:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199819b0a0 con 0x7f1998103830 2024-04-26T01:55:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f198ea00640 1 -- 172.21.15.52:0/875117097 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988010430 con 0x7f1998103830 2024-04-26T01:55:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.582+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19981a72f0 con 0x7f1998103830 2024-04-26T01:55:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.585+0000 7f198ea00640 1 -- 172.21.15.52:0/875117097 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1988004000 con 0x7f1998103830 2024-04-26T01:55:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.585+0000 7f19a35c7740 1 Processor -- start 2024-04-26T01:55:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 -- start start 2024-04-26T01:55:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce6a2d0 0x55868ce74650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce76f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55868cdcf900 con 0x55868ce6a2d0 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55868cd75ef0 con 0x55868ce699b0 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f19a35c7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868cdcb8c0 con 0x55868ce74b90 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f199e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f199e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53368/0 (socket says 172.21.15.52:53368) 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f199e800640 1 -- 172.21.15.52:0/1230700667 learned_addr learned my addr 172.21.15.52:0/1230700667 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f199de00640 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce6a2d0 0x55868ce74650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.586+0000 7f199d400640 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce76f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 -- 172.21.15.52:0/1230700667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 msgr2=0x55868ce76f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce76f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 -- 172.21.15.52:0/1230700667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce6a2d0 msgr2=0x55868ce74650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce6a2d0 0x55868ce74650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 -- 172.21.15.52:0/1230700667 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55868cae2210 con 0x55868ce699b0 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f198ea00640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 0x7f19840a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199e800640 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1994009c10 tx=0x7f1994011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba351b19003a6c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f198ea00640 1 -- 172.21.15.52:0/875117097 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f19880d0a70 con 0x7f1998103830 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f199de00640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 0x7f19840a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f1983000640 1 -- 172.21.15.52:0/1230700667 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994004230 con 0x55868ce699b0 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f1983000640 1 -- 172.21.15.52:0/1230700667 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19940043d0 con 0x55868ce699b0 2024-04-26T01:55:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.587+0000 7f1983000640 1 -- 172.21.15.52:0/1230700667 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994019680 con 0x55868ce699b0 2024-04-26T01:55:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 -- 172.21.15.52:0/1230700667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 msgr2=0x55868ce69d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f199de00640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 0x7f19840a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f199002af40 tx=0x7f1990005d80 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1994009c10 tx=0x7f1994011810 comp rx=0 tx=0).stop 2024-04-26T01:55:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 -- 172.21.15.52:0/1230700667 shutdown_connections 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce76f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce6a2d0 0x55868ce74650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 --2- 172.21.15.52:0/1230700667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce699b0 0x55868ce69d90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 -- 172.21.15.52:0/1230700667 >> 172.21.15.52:0/1230700667 conn(0x55868ce65ec0 msgr2=0x55868ce662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 -- 172.21.15.52:0/1230700667 shutdown_connections 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.588+0000 7f19a35c7740 1 -- 172.21.15.52:0/1230700667 wait complete. 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.589+0000 7f19a35c7740 1 -- start start 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.589+0000 7f19a35c7740 1 Processor -- start 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.589+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce71b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce72060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55868ce777c0 con 0x55868ce699b0 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55868cd89310 con 0x55868ce6a2d0 2024-04-26T01:55:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55868ce77940 con 0x55868ce74b90 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f19a35c7740 10 client.0 _ll_register_callbacks cb 0x55868cd57cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce71b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce71b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52786/0 (socket says 172.21.15.52:52786) 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce72060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199de00640 1 -- 172.21.15.52:0/3227596319 learned_addr learned my addr 172.21.15.52:0/3227596319 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199e800640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 -- 172.21.15.52:0/3227596319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 msgr2=0x55868ce725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 -- 172.21.15.52:0/3227596319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 msgr2=0x55868ce71b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce71b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 -- 172.21.15.52:0/3227596319 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55868cbae620 con 0x55868ce6a2d0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199e800640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f199d400640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce72060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19880ce160 tx=0x7f1988098d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f366afd2e7ab97cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.590+0000 7f1977400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880d0e60 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f19880ce3e0 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19940043d0 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f19880042f0 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f19a35c7740 10 client.26441 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55868ce83a20 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.591+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19940043d0 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f198809dce0 con 0x55868ce6a2d0 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 10 client.26441 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 10 client.26441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 20 client.26441 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ce84f30 0x55868ce873a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 4 client.26441 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:55:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55868cde38b0 con 0x55868ce84f30 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.592+0000 7f199de00640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ce84f30 0x55868ce873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.593+0000 7f199de00640 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ce84f30 0x55868ce873a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18196523f61ace6 server_cookie=efce90d185d4d92d in_seq=0 out_seq=0 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.593+0000 7f1977400640 10 client.26441 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.593+0000 7f1977400640 1 -- 172.21.15.52:0/3227596319 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55868cde38b0 con 0x55868ce84f30 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.593+0000 7f1977400640 10 client.26441 handle_command_reply: tid=0 2024-04-26T01:55:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.593+0000 7f19a35c7740 1 client.26441 shutdown 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 msgr2=0x55868ce72060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce72060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19880ce160 tx=0x7f1988098d10 comp rx=0 tx=0).stop 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 shutdown_connections 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ce84f30 0x55868ce873a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce74b90 0x55868ce725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce72060 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 --2- 172.21.15.52:0/3227596319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce71b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 >> 172.21.15.52:0/3227596319 conn(0x55868ce65ec0 msgr2=0x55868ce6bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f1977400640 0 client.26441 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 shutdown_connections 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 1 -- 172.21.15.52:0/3227596319 wait complete. 2024-04-26T01:55:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.594+0000 7f19a35c7740 20 client.26441 trim_cache size 0 max 16384 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.610+0000 7f19a35c7740 1 Processor -- start 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.610+0000 7f19a35c7740 1 -- start start 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce6a2d0 0x55868ce6bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 0x55868ce8cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f19981b25a0 con 0x55868ce699b0 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f19981afe10 con 0x55868ce74b90 2024-04-26T01:55:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f19a35c7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19981adc80 con 0x55868ce6a2d0 2024-04-26T01:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52792/0 (socket says 172.21.15.52:52792) 2024-04-26T01:55:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 -- 172.21.15.52:0/476397135 learned_addr learned my addr 172.21.15.52:0/476397135 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199d400640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce6a2d0 0x55868ce6bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199e800640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 0x55868ce8cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 -- 172.21.15.52:0/476397135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce6a2d0 msgr2=0x55868ce6bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce6a2d0 0x55868ce6bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 -- 172.21.15.52:0/476397135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 msgr2=0x55868ce8cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 0x55868ce8cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199de00640 1 -- 172.21.15.52:0/476397135 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55868ce70090 con 0x55868ce699b0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.611+0000 7f199e800640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 0x55868ce8cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.612+0000 7f199de00640 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1990003990 tx=0x7f1990042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53831ef93cc1fce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.612+0000 7f1982600640 1 -- 172.21.15.52:0/476397135 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199005b070 con 0x55868ce699b0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.612+0000 7f1982600640 1 -- 172.21.15.52:0/476397135 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1990030d40 con 0x55868ce699b0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.612+0000 7f1982600640 1 -- 172.21.15.52:0/476397135 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1990034040 con 0x55868ce699b0 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- 172.21.15.52:0/476397135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 msgr2=0x55868ce80680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1990003990 tx=0x7f1990042000 comp rx=0 tx=0).stop 2024-04-26T01:55:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- 172.21.15.52:0/476397135 shutdown_connections 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce74b90 0x55868ce8cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce6a2d0 0x55868ce6bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 --2- 172.21.15.52:0/476397135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce699b0 0x55868ce80680 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- 172.21.15.52:0/476397135 >> 172.21.15.52:0/476397135 conn(0x55868ce65ec0 msgr2=0x55868ce6b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- 172.21.15.52:0/476397135 shutdown_connections 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- 172.21.15.52:0/476397135 wait complete. 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.613+0000 7f19a35c7740 1 -- start start 2024-04-26T01:55:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 Processor -- start 2024-04-26T01:55:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce7abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 0x55868ce7b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ae2d0 con 0x55868ce74b90 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f19981af740 con 0x55868ce6a2d0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19981ae9a0 con 0x55868ce699b0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f19a35c7740 10 client.0 _ll_register_callbacks cb 0x7f1988009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.614+0000 7f199de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57958/0 (socket says 172.21.15.52:57958) 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 -- 172.21.15.52:0/848746724 learned_addr learned my addr 172.21.15.52:0/848746724 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199e800640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 0x55868ce7b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199d400640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce7abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 -- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 msgr2=0x55868ce7abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce7abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 -- 172.21.15.52:0/848746724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 msgr2=0x55868ce7b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:55:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 0x55868ce7b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 -- 172.21.15.52:0/848746724 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55868ceea5c0 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199e800640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 0x55868ce7b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f199de00640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1990030fa0 tx=0x7f199002f8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a9929405062ad8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f1977400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199005b070 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f1990055050 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.615+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1990030d40 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f1990067050 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 10 client.26446 fetch_fsmap learned FSMap version 45 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55868ceeb8d0 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1990030d40 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f199003a780 con 0x55868ce699b0 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 10 client.26446 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 10 client.26446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:55:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 20 client.26446 populate_metadata read hostname 'smithi052' 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ceec020 0x55868ceee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 4 client.26446 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.616+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55868cddd320 con 0x55868ceec020 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.617+0000 7f199d400640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ceec020 0x55868ceee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.617+0000 7f199d400640 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ceec020 0x55868ceee440 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1901b22098a47a32 server_cookie=ee4d44016c263382 in_seq=0 out_seq=0 2024-04-26T01:55:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.617+0000 7f1977400640 10 client.26446 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f1977400640 1 -- 172.21.15.52:0/848746724 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55868cddd320 con 0x55868ceec020 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f1977400640 10 client.26446 handle_command_reply: tid=0 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 client.26446 shutdown 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 msgr2=0x55868ce7a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1990030fa0 tx=0x7f199002f8f0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 shutdown_connections 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55868ceec020 0x55868ceee440 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55868ce74b90 0x55868ce7b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55868ce6a2d0 0x55868ce7abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 --2- 172.21.15.52:0/848746724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55868ce699b0 0x55868ce7a6a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f1977400640 0 client.26446 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 >> 172.21.15.52:0/848746724 conn(0x55868ce65ec0 msgr2=0x55868ce73be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 shutdown_connections 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.643+0000 7f19a35c7740 1 -- 172.21.15.52:0/848746724 wait complete. 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.644+0000 7f19a35c7740 20 client.26446 trim_cache size 0 max 16384 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:55:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:55:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.644+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 msgr2=0x7f19840a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.644+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 0x7f19840a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f199002af40 tx=0x7f1990005d80 comp rx=0 tx=0).stop 2024-04-26T01:55:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.644+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 msgr2=0x7f19981a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:55:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.644+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f198800e9f0 tx=0x7f198800edf0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 shutdown_connections 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f19840a3d90 0x7f19840a6250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f199810c330 0x7f199819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1998103830 0x7f19981a0970 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 --2- 172.21.15.52:0/875117097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1998102f10 0x7f19981a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 >> 172.21.15.52:0/875117097 conn(0x7f19980fc770 msgr2=0x7f1998101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 shutdown_connections 2024-04-26T01:55:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:55:47.645+0000 7f199f200640 1 -- 172.21.15.52:0/875117097 wait complete. 2024-04-26T01:55:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:47 smithi191 ceph-mon[32745]: pgmap v337: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 302 KiB/s wr, 44 op/s 2024-04-26T01:55:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:47 smithi183 ceph-mon[32752]: pgmap v337: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 302 KiB/s wr, 44 op/s 2024-04-26T01:55:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:47 smithi052 ceph-mon[27851]: pgmap v337: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 302 KiB/s wr, 44 op/s 2024-04-26T01:55:48.130 INFO:tasks.workunit.client.0.smithi052.stdout: 1 250033 2.48 MB/sec execute 427 sec latency 960.745 ms 2024-04-26T01:55:49.131 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251260 2.49 MB/sec execute 428 sec latency 35.652 ms 2024-04-26T01:55:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:49 smithi191 ceph-mon[32745]: pgmap v338: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:49 smithi183 ceph-mon[32752]: pgmap v338: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:49 smithi052 ceph-mon[27851]: pgmap v338: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:50.131 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.49 MB/sec execute 429 sec latency 628.526 ms 2024-04-26T01:55:51.131 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.49 MB/sec execute 430 sec latency 1628.666 ms 2024-04-26T01:55:51.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:51 smithi191 ceph-mon[32745]: pgmap v339: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:51.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:51 smithi183 ceph-mon[32752]: pgmap v339: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:51 smithi052 ceph-mon[27851]: pgmap v339: 97 pgs: 97 active+clean; 93 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 281 KiB/s wr, 36 op/s 2024-04-26T01:55:52.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.48 MB/sec execute 431 sec latency 2628.819 ms 2024-04-26T01:55:53.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.47 MB/sec execute 432 sec latency 3628.950 ms 2024-04-26T01:55:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:53 smithi191 ceph-mon[32745]: pgmap v340: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 512 KiB/s wr, 79 op/s 2024-04-26T01:55:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:53 smithi183 ceph-mon[32752]: pgmap v340: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 512 KiB/s wr, 79 op/s 2024-04-26T01:55:53.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:53 smithi052 ceph-mon[27851]: pgmap v340: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 512 KiB/s wr, 79 op/s 2024-04-26T01:55:54.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.47 MB/sec execute 433 sec latency 4629.168 ms 2024-04-26T01:55:54.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:54 smithi191 ceph-mon[32745]: pgmap v341: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 400 KiB/s wr, 68 op/s 2024-04-26T01:55:54.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:54 smithi183 ceph-mon[32752]: pgmap v341: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 400 KiB/s wr, 68 op/s 2024-04-26T01:55:54.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:54 smithi052 ceph-mon[27851]: pgmap v341: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 400 KiB/s wr, 68 op/s 2024-04-26T01:55:55.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.46 MB/sec execute 434 sec latency 5629.262 ms 2024-04-26T01:55:56.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.46 MB/sec execute 435 sec latency 6629.406 ms 2024-04-26T01:55:57.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.45 MB/sec execute 436 sec latency 7629.527 ms 2024-04-26T01:55:57.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:57 smithi052 ceph-mon[27851]: pgmap v342: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 417 KiB/s wr, 73 op/s 2024-04-26T01:55:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:57 smithi191 ceph-mon[32745]: pgmap v342: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 417 KiB/s wr, 73 op/s 2024-04-26T01:55:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:57 smithi183 ceph-mon[32752]: pgmap v342: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 417 KiB/s wr, 73 op/s 2024-04-26T01:55:58.132 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251818 2.45 MB/sec execute 437 sec latency 8629.629 ms 2024-04-26T01:55:59.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 251969 2.44 MB/sec execute 438 sec latency 9396.916 ms 2024-04-26T01:55:59.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:55:59 smithi052 ceph-mon[27851]: pgmap v343: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:55:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:55:59 smithi191 ceph-mon[32745]: pgmap v343: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:55:59.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:55:59 smithi183 ceph-mon[32752]: pgmap v343: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:56:00.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 252274 2.44 MB/sec execute 439 sec latency 30.185 ms 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 -- 172.21.15.52:0/3226354963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30103850 msgr2=0x7f3f3010bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 --2- 172.21.15.52:0/3226354963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30103850 0x7f3f3010bd30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009900 tx=0x7f3f1c02a480 comp rx=0 tx=0).stop 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 -- 172.21.15.52:0/3226354963 shutdown_connections 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 --2- 172.21.15.52:0/3226354963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f3010c350 0x7f3f3010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 --2- 172.21.15.52:0/3226354963 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f30103850 0x7f3f3010bd30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 --2- 172.21.15.52:0/3226354963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f30103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 -- 172.21.15.52:0/3226354963 >> 172.21.15.52:0/3226354963 conn(0x7f3f300fc770 msgr2=0x7f3f300feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 -- 172.21.15.52:0/3226354963 shutdown_connections 2024-04-26T01:56:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.791+0000 7f3f36800640 1 -- 172.21.15.52:0/3226354963 wait complete. 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.792+0000 7f3f36800640 1 Processor -- start 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.792+0000 7f3f36800640 1 -- start start 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f30103850 0x7f3f301a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 0x7f3f3019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30111fc0 con 0x7f3f30102f30 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30111e40 con 0x7f3f30103850 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f36800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30112140 con 0x7f3f3010c350 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35108/0 (socket says 172.21.15.52:35108) 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f35400640 1 -- 172.21.15.52:0/2718490193 learned_addr learned my addr 172.21.15.52:0/2718490193 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f35e00640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 0x7f3f3019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.793+0000 7f3f34a00640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f30103850 0x7f3f301a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 -- 172.21.15.52:0/2718490193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 msgr2=0x7f3f3019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 0x7f3f3019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 -- 172.21.15.52:0/2718490193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f30103850 msgr2=0x7f3f301a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f30103850 0x7f3f301a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 -- 172.21.15.52:0/2718490193 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3019ad40 con 0x7f3f30102f30 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35e00640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 0x7f3f3019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f35400640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3f2400d950 tx=0x7f3f2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.794+0000 7f3f2e000640 1 -- 172.21.15.52:0/2718490193 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24017030 con 0x7f3f30102f30 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.795+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3019b030 con 0x7f3f30102f30 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.795+0000 7f3f2e000640 1 -- 172.21.15.52:0/2718490193 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f24004020 con 0x7f3f30102f30 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.795+0000 7f3f2e000640 1 -- 172.21.15.52:0/2718490193 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240047a0 con 0x7f3f30102f30 2024-04-26T01:56:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.795+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f301a72d0 con 0x7f3f30102f30 2024-04-26T01:56:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.799+0000 7f3f3abba740 1 Processor -- start 2024-04-26T01:56:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.799+0000 7f3f3abba740 1 -- start start 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.799+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be69b0 0x556e57be6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.799+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 0x556e57bf1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f3abba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556e57b4c900 con 0x556e57be72d0 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f3abba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556e57af2ef0 con 0x556e57be69b0 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f3abba740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556e57b488c0 con 0x556e57bf1b90 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f2e000640 1 -- 172.21.15.52:0/2718490193 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3f24009a30 con 0x7f3f30102f30 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f35e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be69b0 0x556e57be6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39342/0 (socket says 172.21.15.52:39342) 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 -- 172.21.15.52:0/3550133523 learned_addr learned my addr 172.21.15.52:0/3550133523 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f35400640 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 0x556e57bf1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 -- 172.21.15.52:0/3550133523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be69b0 msgr2=0x556e57be6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be69b0 0x556e57be6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 -- 172.21.15.52:0/3550133523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 msgr2=0x556e57bf1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 0x556e57bf1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f34a00640 1 -- 172.21.15.52:0/3550133523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e5785f210 con 0x556e57bf1b90 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.800+0000 7f3f35400640 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 0x556e57bf1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.801+0000 7f3f34a00640 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009900 tx=0x7f3f1c032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e27e52001c9f89c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.801+0000 7f3f22600640 1 -- 172.21.15.52:0/3550133523 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c040070 con 0x556e57bf1b90 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.801+0000 7f3f22600640 1 -- 172.21.15.52:0/3550133523 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f1c004030 con 0x556e57bf1b90 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.801+0000 7f3f22600640 1 -- 172.21.15.52:0/3550133523 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c035040 con 0x556e57bf1b90 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.801+0000 7f3f2e000640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 0x7f3f180a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f2e000640 1 -- 172.21.15.52:0/2718490193 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f3f240d0230 con 0x7f3f30102f30 2024-04-26T01:56:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f35e00640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 0x7f3f180a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 -- 172.21.15.52:0/3550133523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 msgr2=0x556e57bf3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009900 tx=0x7f3f1c032b20 comp rx=0 tx=0).stop 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 -- 172.21.15.52:0/3550133523 shutdown_connections 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57bf1b90 0x556e57bf3f80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be72d0 0x556e57bf1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 --2- 172.21.15.52:0/3550133523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be69b0 0x556e57be6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 -- 172.21.15.52:0/3550133523 >> 172.21.15.52:0/3550133523 conn(0x556e57be2ec0 msgr2=0x556e57be32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 -- 172.21.15.52:0/3550133523 shutdown_connections 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f3abba740 1 -- 172.21.15.52:0/3550133523 wait complete. 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.802+0000 7f3f35e00640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 0x7f3f180a61d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009fd0 tx=0x7f3f28009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.803+0000 7f3f3abba740 1 -- start start 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.803+0000 7f3f3abba740 1 Processor -- start 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 0x556e57beea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57bf1b90 0x556e57bef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556e57bf4760 con 0x556e57bf1b90 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556e57b06310 con 0x556e57be72d0 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556e57bf48e0 con 0x556e57be69b0 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f3abba740 10 client.0 _ll_register_callbacks cb 0x556e57ad4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f35e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57bf1b90 0x556e57bef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f35400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 0x556e57beea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:50962/0 (socket says 172.21.15.52:50962) 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 -- 172.21.15.52:0/2784080050 learned_addr learned my addr 172.21.15.52:0/2784080050 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f35400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 0x556e57beea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39348/0 (socket says 172.21.15.52:39348) 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 -- 172.21.15.52:0/2784080050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 msgr2=0x556e57beea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 0x556e57beea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 -- 172.21.15.52:0/2784080050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57bf1b90 msgr2=0x556e57bef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57bf1b90 0x556e57bef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.804+0000 7f3f34a00640 1 -- 172.21.15.52:0/2784080050 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e5792b620 con 0x556e57be72d0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f34a00640 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c032ec0 tx=0x7f3f1c031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71b1b8964ed61bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f16a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c035070 con 0x556e57be72d0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f3f1c005430 con 0x556e57be72d0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f1c004030 con 0x556e57be72d0 2024-04-26T01:56:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3f1c040040 con 0x556e57be72d0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f3abba740 10 client.26459 fetch_fsmap learned FSMap version 45 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556e57c00890 con 0x556e57be72d0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.805+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f1c004030 con 0x556e57be72d0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f3f1c004be0 con 0x556e57be72d0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 10 client.26459 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 10 client.26459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 20 client.26459 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c01be0 0x556e57c03fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 4 client.26459 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e57b608b0 con 0x556e57c01be0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.806+0000 7f3f35400640 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c01be0 0x556e57c03fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.807+0000 7f3f35400640 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c01be0 0x556e57c03fd0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9be4aa0e1320ff25 server_cookie=1e444537af280ae3 in_seq=0 out_seq=0 2024-04-26T01:56:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.807+0000 7f3f16a00640 10 client.26459 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.807+0000 7f3f16a00640 1 -- 172.21.15.52:0/2784080050 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e57b608b0 con 0x556e57c01be0 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.807+0000 7f3f16a00640 10 client.26459 handle_command_reply: tid=0 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.807+0000 7f3f3abba740 1 client.26459 shutdown 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 msgr2=0x556e57beef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c032ec0 tx=0x7f3f1c031400 comp rx=0 tx=0).stop 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 shutdown_connections 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c01be0 0x556e57c03fd0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57bf1b90 0x556e57bef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57be72d0 0x556e57beef80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f16a00640 0 client.26459 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 --2- 172.21.15.52:0/2784080050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be69b0 0x556e57beea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 >> 172.21.15.52:0/2784080050 conn(0x556e57be2ec0 msgr2=0x556e57be8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 shutdown_connections 2024-04-26T01:56:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 1 -- 172.21.15.52:0/2784080050 wait complete. 2024-04-26T01:56:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.808+0000 7f3f3abba740 20 client.26459 trim_cache size 0 max 16384 2024-04-26T01:56:00.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.824+0000 7f3f3abba740 1 Processor -- start 2024-04-26T01:56:00.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.824+0000 7f3f3abba740 1 -- start start 2024-04-26T01:56:00.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bfd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bedba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57be93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b2510 con 0x556e57be69b0 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301afd80 con 0x556e57bf1b90 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f3abba740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301adbf0 con 0x556e57be72d0 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bfd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57be93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bfd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35138/0 (socket says 172.21.15.52:35138) 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35400640 1 -- 172.21.15.52:0/4000621135 learned_addr learned my addr 172.21.15.52:0/4000621135 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:00.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f34a00640 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bedba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 -- 172.21.15.52:0/4000621135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 msgr2=0x556e57bedba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bedba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 -- 172.21.15.52:0/4000621135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 msgr2=0x556e57bfd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bfd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f35e00640 1 -- 172.21.15.52:0/4000621135 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e57be9900 con 0x556e57bf1b90 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.825+0000 7f3f34a00640 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.826+0000 7f3f35e00640 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57be93c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3f28005fd0 tx=0x7f3f2801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5938999640f5cb1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.826+0000 7f3f21c00640 1 -- 172.21.15.52:0/4000621135 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800fe60 con 0x556e57bf1b90 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.826+0000 7f3f21c00640 1 -- 172.21.15.52:0/4000621135 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f28003c90 con 0x556e57bf1b90 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.826+0000 7f3f21c00640 1 -- 172.21.15.52:0/4000621135 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28005280 con 0x556e57bf1b90 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- 172.21.15.52:0/4000621135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 msgr2=0x556e57be93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57be93c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3f28005fd0 tx=0x7f3f2801b000 comp rx=0 tx=0).stop 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- 172.21.15.52:0/4000621135 shutdown_connections 2024-04-26T01:56:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57be93c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 --2- 172.21.15.52:0/4000621135 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bfd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- 172.21.15.52:0/4000621135 >> 172.21.15.52:0/4000621135 conn(0x556e57be2ec0 msgr2=0x556e57beea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- 172.21.15.52:0/4000621135 shutdown_connections 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- 172.21.15.52:0/4000621135 wait complete. 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 -- start start 2024-04-26T01:56:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.827+0000 7f3f3abba740 1 Processor -- start 2024-04-26T01:56:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bf7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57bf75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301ae240 con 0x556e57be69b0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301af6b0 con 0x556e57bf1b90 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301ae910 con 0x556e57be72d0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f3abba740 10 client.0 _ll_register_callbacks cb 0x7f3f1c009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f35400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35154/0 (socket says 172.21.15.52:35154) 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f35400640 1 -- 172.21.15.52:0/291802868 learned_addr learned my addr 172.21.15.52:0/291802868 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f35e00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57bf75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.828+0000 7f3f34a00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bf7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 -- 172.21.15.52:0/291802868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 msgr2=0x556e57bf7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bf7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 -- 172.21.15.52:0/291802868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 msgr2=0x556e57bf6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 -- 172.21.15.52:0/291802868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e57c660b0 con 0x556e57bf1b90 2024-04-26T01:56:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35400640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f34a00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bf7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f35e00640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57bf75c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3f28012740 tx=0x7f3f28005ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822004dfffc259c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f16a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.829+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2802d070 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f28003c90 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8521+0+0 (secure 0 0 0) 0x7f3f28012160 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3f2803f050 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 10 client.26465 fetch_fsmap learned FSMap version 45 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556e57c673c0 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f28003c90 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f3f2803a030 con 0x556e57bf1b90 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 10 client.26465 fetch_fsmap finished waiting for FSMap version 45 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 20 client.26465 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c67b70 0x556e57c69f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 4 client.26465 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.830+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e57b5a320 con 0x556e57c67b70 2024-04-26T01:56:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.831+0000 7f3f35400640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c67b70 0x556e57c69f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.831+0000 7f3f35400640 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c67b70 0x556e57c69f90 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0da0a1ff250cef server_cookie=88a2c544ca016aff in_seq=0 out_seq=0 2024-04-26T01:56:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.831+0000 7f3f16a00640 10 client.26465 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f16a00640 1 -- 172.21.15.52:0/291802868 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e57b5a320 con 0x556e57c67b70 2024-04-26T01:56:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f16a00640 10 client.26465 handle_command_reply: tid=0 2024-04-26T01:56:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 client.26465 shutdown 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 msgr2=0x556e57bf75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57bf75c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3f28012740 tx=0x7f3f28005ad0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 shutdown_connections 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556e57c67b70 0x556e57c69f90 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556e57bf1b90 0x556e57bf75c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556e57be72d0 0x556e57bf7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 --2- 172.21.15.52:0/291802868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556e57be69b0 0x556e57bf6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 >> 172.21.15.52:0/291802868 conn(0x556e57be2ec0 msgr2=0x556e57beda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.856+0000 7f3f16a00640 0 client.26465 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.857+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 shutdown_connections 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.857+0000 7f3f3abba740 1 -- 172.21.15.52:0/291802868 wait complete. 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.857+0000 7f3f3abba740 20 client.26465 trim_cache size 0 max 16384 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:56:00.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:56:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 msgr2=0x7f3f180a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 0x7f3f180a61d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009fd0 tx=0x7f3f28009500 comp rx=0 tx=0).stop 2024-04-26T01:56:00.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 msgr2=0x7f3f301a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3f2400d950 tx=0x7f3f2400de10 comp rx=0 tx=0).stop 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 shutdown_connections 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f180a3d10 0x7f3f180a61d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f3010c350 0x7f3f3019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f30103850 0x7f3f301a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 --2- 172.21.15.52:0/2718490193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3f30102f30 0x7f3f301a0410 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 >> 172.21.15.52:0/2718490193 conn(0x7f3f300fc770 msgr2=0x7f3f30101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 shutdown_connections 2024-04-26T01:56:00.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:00.858+0000 7f3f36800640 1 -- 172.21.15.52:0/2718490193 wait complete. 2024-04-26T01:56:01.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 252447 2.43 MB/sec execute 440 sec latency 449.871 ms 2024-04-26T01:56:01.455 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:01 smithi191 ceph-mon[32745]: pgmap v344: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:56:01.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:01 smithi052 ceph-mon[27851]: pgmap v344: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:56:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:01 smithi183 ceph-mon[32752]: pgmap v344: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 248 KiB/s wr, 47 op/s 2024-04-26T01:56:02.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 252447 2.43 MB/sec execute 441 sec latency 1450.013 ms 2024-04-26T01:56:03.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 252479 2.42 MB/sec execute 442 sec latency 2381.100 ms 2024-04-26T01:56:03.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:03 smithi052 ceph-mon[27851]: pgmap v345: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 877 KiB/s wr, 98 op/s 2024-04-26T01:56:03.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:03 smithi191 ceph-mon[32745]: pgmap v345: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 877 KiB/s wr, 98 op/s 2024-04-26T01:56:03.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:03 smithi183 ceph-mon[32752]: pgmap v345: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 877 KiB/s wr, 98 op/s 2024-04-26T01:56:04.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 252929 2.42 MB/sec execute 443 sec latency 61.228 ms 2024-04-26T01:56:04.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:04 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:04.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:04 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:04.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:04 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:05.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 253572 2.42 MB/sec execute 444 sec latency 175.043 ms 2024-04-26T01:56:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:05 smithi191 ceph-mon[32745]: pgmap v346: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 645 KiB/s wr, 55 op/s 2024-04-26T01:56:05.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:05 smithi183 ceph-mon[32752]: pgmap v346: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 645 KiB/s wr, 55 op/s 2024-04-26T01:56:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:05 smithi052 ceph-mon[27851]: pgmap v346: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 645 KiB/s wr, 55 op/s 2024-04-26T01:56:06.133 INFO:tasks.workunit.client.0.smithi052.stdout: 1 253572 2.42 MB/sec execute 445 sec latency 1175.184 ms 2024-04-26T01:56:07.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 253572 2.41 MB/sec execute 446 sec latency 2175.361 ms 2024-04-26T01:56:07.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:07 smithi191 ceph-mon[32745]: pgmap v347: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 895 KiB/s wr, 96 op/s 2024-04-26T01:56:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:07 smithi191 ceph-mon[32745]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:56:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:07 smithi183 ceph-mon[32752]: pgmap v347: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 895 KiB/s wr, 96 op/s 2024-04-26T01:56:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:07 smithi183 ceph-mon[32752]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:56:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:07 smithi052 ceph-mon[27851]: pgmap v347: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 895 KiB/s wr, 96 op/s 2024-04-26T01:56:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:07 smithi052 ceph-mon[27851]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T01:56:08.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 254021 2.41 MB/sec execute 447 sec latency 2884.603 ms 2024-04-26T01:56:09.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255419 2.43 MB/sec execute 448 sec latency 24.394 ms 2024-04-26T01:56:09.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:09 smithi191 ceph-mon[32745]: pgmap v349: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:09 smithi183 ceph-mon[32752]: pgmap v349: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:09 smithi052 ceph-mon[27851]: pgmap v349: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:10.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255493 2.42 MB/sec execute 449 sec latency 889.275 ms 2024-04-26T01:56:11.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255493 2.42 MB/sec execute 450 sec latency 1889.409 ms 2024-04-26T01:56:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:11 smithi191 ceph-mon[32745]: pgmap v350: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:11.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:11 smithi183 ceph-mon[32752]: pgmap v350: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:11.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:11 smithi052 ceph-mon[27851]: pgmap v350: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-26T01:56:12.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255493 2.41 MB/sec execute 451 sec latency 2889.525 ms 2024-04-26T01:56:13.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255500 2.41 MB/sec execute 452 sec latency 3839.572 ms 2024-04-26T01:56:13.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:13 smithi191 ceph-mon[32745]: pgmap v351: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 479 KiB/s wr, 78 op/s 2024-04-26T01:56:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:13 smithi183 ceph-mon[32752]: pgmap v351: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 479 KiB/s wr, 78 op/s 2024-04-26T01:56:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:13 smithi052 ceph-mon[27851]: pgmap v351: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 479 KiB/s wr, 78 op/s 2024-04-26T01:56:14.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 -- 172.21.15.52:0/2841478532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc104bb0 msgr2=0x7f3efc108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 --2- 172.21.15.52:0/2841478532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc104bb0 0x7f3efc108bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009900 tx=0x7f3ee402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 -- 172.21.15.52:0/2841478532 shutdown_connections 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 --2- 172.21.15.52:0/2841478532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc1092d0 0x7f3efc110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 --2- 172.21.15.52:0/2841478532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc104bb0 0x7f3efc108bf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 --2- 172.21.15.52:0/2841478532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc104200 0x7f3efc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 -- 172.21.15.52:0/2841478532 >> 172.21.15.52:0/2841478532 conn(0x7f3efc0ffa60 msgr2=0x7f3efc101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 -- 172.21.15.52:0/2841478532 shutdown_connections 2024-04-26T01:56:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.003+0000 7f3efb200640 1 -- 172.21.15.52:0/2841478532 wait complete. 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.004+0000 7f3efb200640 1 Processor -- start 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.004+0000 7f3efb200640 1 -- start start 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc104200 0x7f3efc1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc19a5d0 0x7f3efc19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1141f0 con 0x7f3efc1092d0 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc114070 con 0x7f3efc104200 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efb200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc114370 con 0x7f3efc19a5d0 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59012/0 (socket says 172.21.15.52:59012) 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 -- 172.21.15.52:0/2418231538 learned_addr learned my addr 172.21.15.52:0/2418231538 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc104200 0x7f3efc1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3efa800640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc19a5d0 0x7f3efc19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 -- 172.21.15.52:0/2418231538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc19a5d0 msgr2=0x7f3efc19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc19a5d0 0x7f3efc19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 -- 172.21.15.52:0/2418231538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc104200 msgr2=0x7f3efc1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc104200 0x7f3efc1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.005+0000 7f3ef3e00640 1 -- 172.21.15.52:0/2418231538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc19b0f0 con 0x7f3efc1092d0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4032fd0 tx=0x7f3ee4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3ef2a00640 1 -- 172.21.15.52:0/2418231538 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee403f030 con 0x7f3efc1092d0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3ef2a00640 1 -- 172.21.15.52:0/2418231538 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee4004490 con 0x7f3efc1092d0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc19b380 con 0x7f3efc1092d0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3ef2a00640 1 -- 172.21.15.52:0/2418231538 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4035070 con 0x7f3efc1092d0 2024-04-26T01:56:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.006+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc1a7570 con 0x7f3efc1092d0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.009+0000 7f3f03792740 1 Processor -- start 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.009+0000 7f3f03792740 1 -- start start 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f6164ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562f615a8900 con 0x562f616432d0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562f6154eef0 con 0x562f6164db90 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3f03792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562f615a48c0 con 0x562f616429b0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3ef9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3efa800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3efa800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35766/0 (socket says 172.21.15.52:35766) 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3efa800640 1 -- 172.21.15.52:0/1992097218 learned_addr learned my addr 172.21.15.52:0/1992097218 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3ef9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59026/0 (socket says 172.21.15.52:59026) 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.010+0000 7f3ef2a00640 1 -- 172.21.15.52:0/2418231538 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3ee4042030 con 0x7f3efc1092d0 2024-04-26T01:56:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 -- 172.21.15.52:0/1992097218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 msgr2=0x562f6164ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f6164ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 -- 172.21.15.52:0/1992097218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 msgr2=0x562f6164d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 -- 172.21.15.52:0/1992097218 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f612bb210 con 0x562f616429b0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3efa800640 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009f50 tx=0x7f3ef40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7000ac4d385ef063 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3eeb000640 1 -- 172.21.15.52:0/1992097218 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004b20 con 0x562f616429b0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3eeb000640 1 -- 172.21.15.52:0/1992097218 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ef4004cc0 con 0x562f616429b0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.011+0000 7f3eeb000640 1 -- 172.21.15.52:0/1992097218 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4005330 con 0x562f616429b0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3ef2a00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 0x7f3ee00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 -- 172.21.15.52:0/1992097218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 msgr2=0x562f61642d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009f50 tx=0x7f3ef40118d0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3ef2a00640 1 -- 172.21.15.52:0/2418231538 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3ee40f07a0 con 0x7f3efc1092d0 2024-04-26T01:56:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 -- 172.21.15.52:0/1992097218 shutdown_connections 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f6164ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 0x7f3ee00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f6164d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 --2- 172.21.15.52:0/1992097218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61642d90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 -- 172.21.15.52:0/1992097218 >> 172.21.15.52:0/1992097218 conn(0x562f6163eec0 msgr2=0x562f6163f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 -- 172.21.15.52:0/1992097218 shutdown_connections 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.012+0000 7f3f03792740 1 -- 172.21.15.52:0/1992097218 wait complete. 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.013+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 0x7f3ee00a61d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x562f61643f50 tx=0x7f3eec005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.013+0000 7f3f03792740 1 -- start start 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.013+0000 7f3f03792740 1 Processor -- start 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f6164b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f6164b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562f616507c0 con 0x562f6164db90 2024-04-26T01:56:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562f61562310 con 0x562f616432d0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562f61650940 con 0x562f616429b0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3f03792740 10 client.0 _ll_register_callbacks cb 0x562f61530cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3ef9e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3efa800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f6164b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3ef9e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35770/0 (socket says 172.21.15.52:35770) 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3ef9e00640 1 -- 172.21.15.52:0/912236784 learned_addr learned my addr 172.21.15.52:0/912236784 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f6164b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3efa800640 1 -- 172.21.15.52:0/912236784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 msgr2=0x562f6164aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3efa800640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3efa800640 1 -- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 msgr2=0x562f6164b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.014+0000 7f3efa800640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f6164b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3efa800640 1 -- 172.21.15.52:0/912236784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f61387620 con 0x562f6164db90 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3efa800640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f6164b570 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3ef401b040 tx=0x7f3ef400fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a5763ea159e950 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3edf400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4028070 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3ef4005090 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.015+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef4004cc0 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3ef4009be0 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 10 client.16338 fetch_fsmap learned FSMap version 46 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562f6165c9e0 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ef4004cc0 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 46) ==== 8358+0+0 (secure 0 0 0) 0x7f3ef4024730 con 0x562f6164db90 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 10 client.16338 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 20 client.16338 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f6165ddf0 0x562f616601e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 4 client.16338 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.016+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f615bc8b0 con 0x562f6165ddf0 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.017+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f6165ddf0 0x562f616601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.017+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f6165ddf0 0x562f616601e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd69c83b620d232e server_cookie=6942318ecc7ff2e7 in_seq=0 out_seq=0 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.017+0000 7f3edf400640 10 client.16338 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.018+0000 7f3edf400640 1 -- 172.21.15.52:0/912236784 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f615bc8b0 con 0x562f6165ddf0 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.018+0000 7f3edf400640 10 client.16338 handle_command_reply: tid=0 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.018+0000 7f3f03792740 1 client.16338 shutdown 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.018+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 msgr2=0x562f6164b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.018+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f6164b570 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3ef401b040 tx=0x7f3ef400fbf0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 shutdown_connections 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f6165ddf0 0x562f616601e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f6164b570 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f6164b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 --2- 172.21.15.52:0/912236784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f6164aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 >> 172.21.15.52:0/912236784 conn(0x562f6163eec0 msgr2=0x562f61644c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3edf400640 0 client.16338 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 shutdown_connections 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 1 -- 172.21.15.52:0/912236784 wait complete. 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.019+0000 7f3f03792740 20 client.16338 trim_cache size 0 max 16384 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.035+0000 7f3f03792740 1 Processor -- start 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.035+0000 7f3f03792740 1 -- start start 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61659680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f61665eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1b25e0 con 0x562f616432d0 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1afe20 con 0x562f6164db90 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3f03792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1adc90 con 0x562f616429b0 2024-04-26T01:56:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59054/0 (socket says 172.21.15.52:59054) 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1103820398 learned_addr learned my addr 172.21.15.52:0/1103820398 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3efa800640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f61665eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.036+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61659680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1103820398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 msgr2=0x562f61659680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61659680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1103820398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 msgr2=0x562f61665eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f61665eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1103820398 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f6163a820 con 0x562f616432d0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61659680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40edf50 tx=0x7f3ee40bc6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb87eddcc4365fb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3eea600640 1 -- 172.21.15.52:0/1103820398 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4049040 con 0x562f616432d0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3eea600640 1 -- 172.21.15.52:0/1103820398 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee40c0af0 con 0x562f616432d0 2024-04-26T01:56:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.037+0000 7f3eea600640 1 -- 172.21.15.52:0/1103820398 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40b8e60 con 0x562f616432d0 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 -- 172.21.15.52:0/1103820398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 msgr2=0x562f61665970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40edf50 tx=0x7f3ee40bc6d0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 -- 172.21.15.52:0/1103820398 shutdown_connections 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f6164db90 0x562f61665eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f616432d0 0x562f61665970 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 --2- 172.21.15.52:0/1103820398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61659680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 -- 172.21.15.52:0/1103820398 >> 172.21.15.52:0/1103820398 conn(0x562f6163eec0 msgr2=0x562f61645db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 -- 172.21.15.52:0/1103820398 shutdown_connections 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.038+0000 7f3f03792740 1 -- 172.21.15.52:0/1103820398 wait complete. 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 -- start start 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 Processor -- start 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61653160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f616bb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1ae2e0 con 0x562f6164db90 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1af750 con 0x562f616432d0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1ae9b0 con 0x562f616429b0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.039+0000 7f3f03792740 10 client.0 _ll_register_callbacks cb 0x7f3ef400f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3efa800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f616bb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45806/0 (socket says 172.21.15.52:45806) 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1927936306 learned_addr learned my addr 172.21.15.52:0/1927936306 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61653160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1927936306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 msgr2=0x562f61653160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61653160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1927936306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 msgr2=0x562f616bb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f616bb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 -- 172.21.15.52:0/1927936306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f616bb7e0 con 0x562f616432d0 2024-04-26T01:56:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3ef3e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ee403b510 tx=0x7f3ee40bc970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f6cd872883b3532 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.040+0000 7f3edf400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40da060 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3ee40ee170 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ee40c0af0 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3ee4049070 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 10 client.26480 fetch_fsmap learned FSMap version 46 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562f616c3d60 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ee40c0af0 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 46) ==== 8358+0+0 (secure 0 0 0) 0x7f3ee404c060 con 0x562f616432d0 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 10 client.26480 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 10 client.26480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 20 client.26480 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f616c42a0 0x562f616c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 4 client.26480 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.041+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562f615b6320 con 0x562f616c42a0 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.042+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f616c42a0 0x562f616c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.042+0000 7f3ef9e00640 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f616c42a0 0x562f616c66c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd626b0f5b3f548e server_cookie=90991fd827911c4d in_seq=0 out_seq=0 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.042+0000 7f3edf400640 10 client.26480 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3edf400640 1 -- 172.21.15.52:0/1927936306 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562f615b6320 con 0x562f616c42a0 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3edf400640 10 client.26480 handle_command_reply: tid=0 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3f03792740 1 client.26480 shutdown 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 msgr2=0x562f616536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3ee403b510 tx=0x7f3ee40bc970 comp rx=0 tx=0).stop 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 shutdown_connections 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.073+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562f616c42a0 0x562f616c66c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562f6164db90 0x562f616bb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562f616432d0 0x562f616536a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 --2- 172.21.15.52:0/1927936306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f616429b0 0x562f61653160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3edf400640 0 client.26480 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 >> 172.21.15.52:0/1927936306 conn(0x562f6163eec0 msgr2=0x562f6164aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 shutdown_connections 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 1 -- 172.21.15.52:0/1927936306 wait complete. 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.074+0000 7f3f03792740 20 client.26480 trim_cache size 0 max 16384 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:56:14.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:56:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 msgr2=0x7f3ee00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 0x7f3ee00a61d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x562f61643f50 tx=0x7f3eec005f90 comp rx=0 tx=0).stop 2024-04-26T01:56:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 msgr2=0x7f3efc1a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4032fd0 tx=0x7f3ee4031a20 comp rx=0 tx=0).stop 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 shutdown_connections 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ee00a3d10 0x7f3ee00a61d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3efc19a5d0 0x7f3efc19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3efc1092d0 0x7f3efc1a0960 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 --2- 172.21.15.52:0/2418231538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3efc104200 0x7f3efc1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 >> 172.21.15.52:0/2418231538 conn(0x7f3efc0ffa60 msgr2=0x7f3efc100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 shutdown_connections 2024-04-26T01:56:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:14.075+0000 7f3efb200640 1 -- 172.21.15.52:0/2418231538 wait complete. 2024-04-26T01:56:14.134 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255560 2.40 MB/sec execute 453 sec latency 820.818 ms 2024-04-26T01:56:14.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:14 smithi191 ceph-mon[32745]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:56:14.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:14 smithi183 ceph-mon[32752]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:56:14.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:14 smithi052 ceph-mon[27851]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T01:56:15.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255560 2.40 MB/sec execute 454 sec latency 1820.907 ms 2024-04-26T01:56:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:15 smithi191 ceph-mon[32745]: pgmap v353: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 224 KiB/s wr, 36 op/s 2024-04-26T01:56:15.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:15 smithi183 ceph-mon[32752]: pgmap v353: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 224 KiB/s wr, 36 op/s 2024-04-26T01:56:15.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:15 smithi052 ceph-mon[27851]: pgmap v353: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 224 KiB/s wr, 36 op/s 2024-04-26T01:56:16.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255560 2.39 MB/sec execute 455 sec latency 2821.060 ms 2024-04-26T01:56:17.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255560 2.39 MB/sec execute 456 sec latency 3821.190 ms 2024-04-26T01:56:17.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:17 smithi191 ceph-mon[32745]: pgmap v354: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 259 KiB/s wr, 51 op/s 2024-04-26T01:56:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:17 smithi183 ceph-mon[32752]: pgmap v354: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 259 KiB/s wr, 51 op/s 2024-04-26T01:56:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:17 smithi052 ceph-mon[27851]: pgmap v354: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 259 KiB/s wr, 51 op/s 2024-04-26T01:56:18.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255567 2.38 MB/sec execute 457 sec latency 4759.785 ms 2024-04-26T01:56:19.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 255847 2.38 MB/sec execute 458 sec latency 31.463 ms 2024-04-26T01:56:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:19 smithi191 ceph-mon[32745]: pgmap v355: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:19 smithi183 ceph-mon[32752]: pgmap v355: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:19 smithi052 ceph-mon[27851]: pgmap v355: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:20.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 256489 2.38 MB/sec execute 459 sec latency 28.334 ms 2024-04-26T01:56:21.135 INFO:tasks.workunit.client.0.smithi052.stdout: 1 257061 2.38 MB/sec execute 460 sec latency 35.968 ms 2024-04-26T01:56:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:21 smithi191 ceph-mon[32745]: pgmap v356: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:21 smithi183 ceph-mon[32752]: pgmap v356: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:21 smithi052 ceph-mon[27851]: pgmap v356: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 256 KiB/s wr, 50 op/s 2024-04-26T01:56:22.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 257113 2.38 MB/sec execute 461 sec latency 916.373 ms 2024-04-26T01:56:23.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 257561 2.37 MB/sec execute 462 sec latency 1628.311 ms 2024-04-26T01:56:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:23 smithi191 ceph-mon[32745]: pgmap v357: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 358 KiB/s wr, 53 op/s 2024-04-26T01:56:23.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:23 smithi183 ceph-mon[32752]: pgmap v357: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 358 KiB/s wr, 53 op/s 2024-04-26T01:56:23.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:23 smithi052 ceph-mon[27851]: pgmap v357: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 358 KiB/s wr, 53 op/s 2024-04-26T01:56:24.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 258949 2.39 MB/sec execute 463 sec latency 21.415 ms 2024-04-26T01:56:25.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259038 2.39 MB/sec execute 464 sec latency 870.234 ms 2024-04-26T01:56:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:25 smithi191 ceph-mon[32745]: pgmap v358: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 329 KiB/s wr, 49 op/s 2024-04-26T01:56:25.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:25 smithi183 ceph-mon[32752]: pgmap v358: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 329 KiB/s wr, 49 op/s 2024-04-26T01:56:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:25 smithi052 ceph-mon[27851]: pgmap v358: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 329 KiB/s wr, 49 op/s 2024-04-26T01:56:26.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259038 2.38 MB/sec execute 465 sec latency 1870.374 ms 2024-04-26T01:56:27.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259038 2.38 MB/sec execute 466 sec latency 2870.462 ms 2024-04-26T01:56:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 -- 172.21.15.52:0/1106255456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 msgr2=0x7f3acc10bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/1106255456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc10bd10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009900 tx=0x7f3ab402a480 comp rx=0 tx=0).stop 2024-04-26T01:56:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 -- 172.21.15.52:0/1106255456 shutdown_connections 2024-04-26T01:56:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/1106255456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc10c330 0x7f3acc10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/1106255456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc10bd10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/1106255456 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc102f10 0x7f3acc1032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 -- 172.21.15.52:0/1106255456 >> 172.21.15.52:0/1106255456 conn(0x7f3acc0fc770 msgr2=0x7f3acc0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 -- 172.21.15.52:0/1106255456 shutdown_connections 2024-04-26T01:56:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.228+0000 7f3ad0a00640 1 -- 172.21.15.52:0/1106255456 wait complete. 2024-04-26T01:56:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.229+0000 7f3ad0a00640 1 Processor -- start 2024-04-26T01:56:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.229+0000 7f3ad0a00640 1 -- start start 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc102f10 0x7f3acc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 0x7f3acc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc112010 con 0x7f3acc103830 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc111e90 con 0x7f3acc102f10 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ad0a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc112190 con 0x7f3acc10c330 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ac1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ac1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59694/0 (socket says 172.21.15.52:59694) 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3ac1e00640 1 -- 172.21.15.52:0/3612967075 learned_addr learned my addr 172.21.15.52:0/3612967075 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3acb200640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 0x7f3acc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.230+0000 7f3aca800640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc102f10 0x7f3acc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 -- 172.21.15.52:0/3612967075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc102f10 msgr2=0x7f3acc1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc102f10 0x7f3acc1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 -- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 msgr2=0x7f3acc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 -- 172.21.15.52:0/3612967075 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc19d5c0 con 0x7f3acc10c330 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3acb200640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 0x7f3acc19ce90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009fd0 tx=0x7f3ac400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3ac3e00640 1 -- 172.21.15.52:0/3612967075 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400ce20 con 0x7f3acc10c330 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc19d8b0 con 0x7f3acc10c330 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3ac3e00640 1 -- 172.21.15.52:0/3612967075 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac4005020 con 0x7f3acc10c330 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3ac3e00640 1 -- 172.21.15.52:0/3612967075 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4010690 con 0x7f3acc10c330 2024-04-26T01:56:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.231+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc1a5140 con 0x7f3acc10c330 2024-04-26T01:56:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ad4172740 1 Processor -- start 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ac3e00640 1 -- 172.21.15.52:0/3612967075 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3ac40054d0 con 0x7f3acc10c330 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ad4172740 1 -- start start 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45809b0 0x55f6a4580d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.235+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a458df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ad4172740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f6a44e6900 con 0x55f6a45812d0 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ad4172740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f6a448cef0 con 0x55f6a458bb90 2024-04-26T01:56:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ad4172740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f6a44e28c0 con 0x55f6a45809b0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59700/0 (socket says 172.21.15.52:59700) 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 -- 172.21.15.52:0/2233693417 learned_addr learned my addr 172.21.15.52:0/2233693417 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45809b0 0x55f6a4580d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a458df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 -- 172.21.15.52:0/2233693417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45809b0 msgr2=0x55f6a4580d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45809b0 0x55f6a4580d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 -- 172.21.15.52:0/2233693417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 msgr2=0x55f6a458df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a458df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3aca800640 1 -- 172.21.15.52:0/2233693417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6a41f9210 con 0x55f6a45812d0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.236+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a458df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3aca800640 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009d20 tx=0x7f3abc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b65f71558ea2cf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3ac3e00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 0x7f3ab00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3abb400640 1 -- 172.21.15.52:0/2233693417 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc01e070 con 0x55f6a45812d0 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3ac3e00640 1 -- 172.21.15.52:0/3612967075 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3ac40d0ea0 con 0x7f3acc10c330 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3abb400640 1 -- 172.21.15.52:0/2233693417 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc019ae0 con 0x55f6a45812d0 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3abb400640 1 -- 172.21.15.52:0/2233693417 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc00ea80 con 0x55f6a45812d0 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.237+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 0x7f3ab00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- 172.21.15.52:0/2233693417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 msgr2=0x55f6a458b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009d20 tx=0x7f3abc011810 comp rx=0 tx=0).stop 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- 172.21.15.52:0/2233693417 shutdown_connections 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a458df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45812d0 0x55f6a458b650 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 --2- 172.21.15.52:0/2233693417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45809b0 0x55f6a4580d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 0x7f3ab00a61d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0fee70 tx=0x7f3ab4009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- 172.21.15.52:0/2233693417 >> 172.21.15.52:0/2233693417 conn(0x55f6a457cec0 msgr2=0x55f6a457d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- 172.21.15.52:0/2233693417 shutdown_connections 2024-04-26T01:56:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- 172.21.15.52:0/2233693417 wait complete. 2024-04-26T01:56:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.238+0000 7f3ad4172740 1 -- start start 2024-04-26T01:56:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 Processor -- start 2024-04-26T01:56:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a4588fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f6a458e7c0 con 0x55f6a45809b0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f6a44a0310 con 0x55f6a45812d0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f6a458e940 con 0x55f6a458bb90 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3ad4172740 10 client.0 _ll_register_callbacks cb 0x55f6a446ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.239+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59710/0 (socket says 172.21.15.52:59710) 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 -- 172.21.15.52:0/3277058049 learned_addr learned my addr 172.21.15.52:0/3277058049 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3acb200640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a4588fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 -- 172.21.15.52:0/3277058049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 msgr2=0x55f6a45894f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45894f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 -- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 msgr2=0x55f6a4588fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a4588fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 -- 172.21.15.52:0/3277058049 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6a42c5620 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aca800640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005890 tx=0x7f3abc00e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99aaee86e3ae7fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.240+0000 7f3aafe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc01e040 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3abc004440 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3abc019ae0 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3abc030050 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3ad4172740 10 client.16356 fetch_fsmap learned FSMap version 46 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f6a459a9a0 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc019ae0 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.241+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 46) ==== 8358+0+0 (secure 0 0 0) 0x7f3abc00e800 con 0x55f6a45809b0 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 10 client.16356 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 20 client.16356 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a459bda0 0x55f6a459e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 4 client.16356 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6a44fa8b0 con 0x55f6a459bda0 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.242+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a459bda0 0x55f6a459e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.243+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a459bda0 0x55f6a459e190 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9561039fd58144 server_cookie=d79518c99ccbb45e in_seq=0 out_seq=0 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.243+0000 7f3aafe00640 10 client.16356 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.243+0000 7f3aafe00640 1 -- 172.21.15.52:0/3277058049 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6a44fa8b0 con 0x55f6a459bda0 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.243+0000 7f3aafe00640 10 client.16356 handle_command_reply: tid=0 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 client.16356 shutdown 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 msgr2=0x55f6a4588a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005890 tx=0x7f3abc00e4f0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 shutdown_connections 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a459bda0 0x55f6a459e190 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45894f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a4588fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3aafe00640 0 client.16356 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3277058049 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4588a70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 >> 172.21.15.52:0/3277058049 conn(0x55f6a457cec0 msgr2=0x55f6a4582bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 shutdown_connections 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 1 -- 172.21.15.52:0/3277058049 wait complete. 2024-04-26T01:56:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.244+0000 7f3ad4172740 20 client.16356 trim_cache size 0 max 16384 2024-04-26T01:56:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.261+0000 7f3ad4172740 1 Processor -- start 2024-04-26T01:56:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.261+0000 7f3ad4172740 1 -- start start 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4597680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45812d0 0x55f6a45a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a45a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1b02c0 con 0x55f6a45809b0 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1adb00 con 0x55f6a458bb90 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ad4172740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1ab970 con 0x55f6a45812d0 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4597680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3acb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a45a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4597680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59716/0 (socket says 172.21.15.52:59716) 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3aca800640 1 -- 172.21.15.52:0/1525372436 learned_addr learned my addr 172.21.15.52:0/1525372436 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.262+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45812d0 0x55f6a45a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 -- 172.21.15.52:0/1525372436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45812d0 msgr2=0x55f6a45a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45812d0 0x55f6a45a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 -- 172.21.15.52:0/1525372436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 msgr2=0x55f6a4597680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4597680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 -- 172.21.15.52:0/1525372436 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6a4581100 con 0x55f6a458bb90 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3acb200640 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a45a3ed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40ce590 tx=0x7f3ac4099130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda845b9ce561bae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3abaa00640 1 -- 172.21.15.52:0/1525372436 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40b5070 con 0x55f6a458bb90 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3abaa00640 1 -- 172.21.15.52:0/1525372436 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac40a5a50 con 0x55f6a458bb90 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.263+0000 7f3abaa00640 1 -- 172.21.15.52:0/1525372436 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40a55c0 con 0x55f6a458bb90 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- 172.21.15.52:0/1525372436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 msgr2=0x55f6a45a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a45a3ed0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40ce590 tx=0x7f3ac4099130 comp rx=0 tx=0).stop 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- 172.21.15.52:0/1525372436 shutdown_connections 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a458bb90 0x55f6a45a3ed0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a45812d0 0x55f6a45a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 --2- 172.21.15.52:0/1525372436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a4597680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- 172.21.15.52:0/1525372436 >> 172.21.15.52:0/1525372436 conn(0x55f6a457cec0 msgr2=0x55f6a4583d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- 172.21.15.52:0/1525372436 shutdown_connections 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- 172.21.15.52:0/1525372436 wait complete. 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.264+0000 7f3ad4172740 1 -- start start 2024-04-26T01:56:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 Processor -- start 2024-04-26T01:56:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a45fa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45fad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1abfc0 con 0x55f6a45809b0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1ad430 con 0x55f6a45812d0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1ac690 con 0x55f6a458bb90 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3ad4172740 10 client.0 _ll_register_callbacks cb 0x7f3abc009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3acb200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3aca800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59722/0 (socket says 172.21.15.52:59722) 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.265+0000 7f3aca800640 1 -- 172.21.15.52:0/3660142647 learned_addr learned my addr 172.21.15.52:0/3660142647 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a45fa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 -- 172.21.15.52:0/3660142647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 msgr2=0x55f6a45fad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45fad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 -- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 msgr2=0x55f6a45fa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a45fa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 -- 172.21.15.52:0/3660142647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6a4592b90 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3acb200640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45fad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a45fa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aca800640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00a010 tx=0x7f3abc00e460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcd648e6bd94ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.266+0000 7f3aafe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc038020 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f3abc0052b0 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac40a5a50 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3abc043050 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3ad4172740 10 client.16362 fetch_fsmap learned FSMap version 46 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f6a4593ea0 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac40a5a50 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 46) ==== 8358+0+0 (secure 0 0 0) 0x7f3abc003bc0 con 0x55f6a45809b0 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.267+0000 7f3ad4172740 10 client.16362 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T01:56:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ad4172740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ad4172740 20 client.16362 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a460ae00 0x55f6a460d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ad4172740 4 client.16362 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6a44f4320 con 0x55f6a460ae00 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.268+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a460ae00 0x55f6a460d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.269+0000 7f3ac1e00640 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a460ae00 0x55f6a460d1f0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa09534cec5536dd server_cookie=5ad3a7ae8ffe27b4 in_seq=0 out_seq=0 2024-04-26T01:56:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.269+0000 7f3aafe00640 10 client.16362 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3aafe00640 1 -- 172.21.15.52:0/3660142647 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6a44f4320 con 0x55f6a460ae00 2024-04-26T01:56:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3aafe00640 10 client.16362 handle_command_reply: tid=0 2024-04-26T01:56:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 client.16362 shutdown 2024-04-26T01:56:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 msgr2=0x55f6a45fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00a010 tx=0x7f3abc00e460 comp rx=0 tx=0).stop 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 shutdown_connections 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f6a460ae00 0x55f6a460d1f0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f6a458bb90 0x55f6a45fad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f6a45812d0 0x55f6a45fa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 --2- 172.21.15.52:0/3660142647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f6a45809b0 0x55f6a45fc030 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 >> 172.21.15.52:0/3660142647 conn(0x55f6a457cec0 msgr2=0x55f6a4582330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.300+0000 7f3aafe00640 0 client.16362 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.301+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 shutdown_connections 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.301+0000 7f3ad4172740 1 -- 172.21.15.52:0/3660142647 wait complete. 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.301+0000 7f3ad4172740 20 client.16362 trim_cache size 0 max 16384 2024-04-26T01:56:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:56:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:56:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:56:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:56:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 msgr2=0x7f3ab00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 0x7f3ab00a61d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0fee70 tx=0x7f3ab4009260 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 msgr2=0x7f3acc19ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 0x7f3acc19ce90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009fd0 tx=0x7f3ac400ef10 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 shutdown_connections 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3ab00a3d10 0x7f3ab00a61d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3acc10c330 0x7f3acc19ce90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3acc103830 0x7f3acc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 --2- 172.21.15.52:0/3612967075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3acc102f10 0x7f3acc1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 >> 172.21.15.52:0/3612967075 conn(0x7f3acc0fc770 msgr2=0x7f3acc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 shutdown_connections 2024-04-26T01:56:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:27.302+0000 7f3ad0a00640 1 -- 172.21.15.52:0/3612967075 wait complete. 2024-04-26T01:56:27.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:27 smithi191 ceph-mon[32745]: pgmap v359: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 450 KiB/s wr, 70 op/s 2024-04-26T01:56:27.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:27 smithi183 ceph-mon[32752]: pgmap v359: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 450 KiB/s wr, 70 op/s 2024-04-26T01:56:27.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:27 smithi052 ceph-mon[27851]: pgmap v359: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 450 KiB/s wr, 70 op/s 2024-04-26T01:56:28.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259050 2.37 MB/sec execute 467 sec latency 3802.649 ms 2024-04-26T01:56:29.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259105 2.37 MB/sec execute 468 sec latency 807.514 ms 2024-04-26T01:56:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:29 smithi191 ceph-mon[32745]: pgmap v360: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:29.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:29 smithi183 ceph-mon[32752]: pgmap v360: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:29.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:29 smithi052 ceph-mon[27851]: pgmap v360: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:30.136 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259112 2.36 MB/sec execute 469 sec latency 1781.745 ms 2024-04-26T01:56:30.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:30 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:30.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:30 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:30.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:30 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:56:31.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 259495 2.36 MB/sec execute 470 sec latency 24.744 ms 2024-04-26T01:56:31.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:31 smithi191 ceph-mon[32745]: pgmap v361: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:31.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:31 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:31.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:31 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:31.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:31 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:56:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:31 smithi183 ceph-mon[32752]: pgmap v361: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:31 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:31 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:31 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:56:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:31 smithi052 ceph-mon[27851]: pgmap v361: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 385 KiB/s wr, 53 op/s 2024-04-26T01:56:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:31 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:56:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:31 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:56:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:31 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:56:32.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 260231 2.36 MB/sec execute 471 sec latency 24.516 ms 2024-04-26T01:56:33.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 260744 2.36 MB/sec execute 472 sec latency 140.427 ms 2024-04-26T01:56:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:33 smithi191 ceph-mon[32745]: pgmap v362: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-04-26T01:56:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:33 smithi183 ceph-mon[32752]: pgmap v362: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-04-26T01:56:33.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:33 smithi052 ceph-mon[27851]: pgmap v362: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 1.0 MiB/s wr, 93 op/s 2024-04-26T01:56:34.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 261963 2.37 MB/sec execute 473 sec latency 36.443 ms 2024-04-26T01:56:34.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:34 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:34.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:34 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:34.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:34 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:35.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262584 2.38 MB/sec execute 474 sec latency 493.971 ms 2024-04-26T01:56:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:35 smithi191 ceph-mon[32745]: pgmap v363: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-04-26T01:56:35.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:35 smithi183 ceph-mon[32752]: pgmap v363: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-04-26T01:56:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:35 smithi052 ceph-mon[27851]: pgmap v363: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 810 KiB/s wr, 67 op/s 2024-04-26T01:56:36.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262584 2.37 MB/sec execute 475 sec latency 1494.084 ms 2024-04-26T01:56:37.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262584 2.37 MB/sec execute 476 sec latency 2494.221 ms 2024-04-26T01:56:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:37 smithi191 ceph-mon[32745]: pgmap v364: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-26T01:56:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:37 smithi183 ceph-mon[32752]: pgmap v364: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-26T01:56:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:37 smithi052 ceph-mon[27851]: pgmap v364: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-26T01:56:38.137 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262591 2.36 MB/sec execute 477 sec latency 3427.402 ms 2024-04-26T01:56:39.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262651 2.36 MB/sec execute 478 sec latency 828.115 ms 2024-04-26T01:56:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:39 smithi191 ceph-mon[32745]: pgmap v365: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:39.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:39 smithi183 ceph-mon[32752]: pgmap v365: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:39 smithi052 ceph-mon[27851]: pgmap v365: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:40.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262651 2.35 MB/sec execute 479 sec latency 1828.243 ms 2024-04-26T01:56:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 -- 172.21.15.52:0/2555341168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c100b40 msgr2=0x7f136c100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 --2- 172.21.15.52:0/2555341168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c100b40 0x7f136c100f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f135c009900 tx=0x7f135c02a3f0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 -- 172.21.15.52:0/2555341168 shutdown_connections 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 --2- 172.21.15.52:0/2555341168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c10e6f0 0x7f136c110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 --2- 172.21.15.52:0/2555341168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 0x7f136c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 --2- 172.21.15.52:0/2555341168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c100b40 0x7f136c100f20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 -- 172.21.15.52:0/2555341168 >> 172.21.15.52:0/2555341168 conn(0x7f136c0fc790 msgr2=0x7f136c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 -- 172.21.15.52:0/2555341168 shutdown_connections 2024-04-26T01:56:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.453+0000 7f136b200640 1 -- 172.21.15.52:0/2555341168 wait complete. 2024-04-26T01:56:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.454+0000 7f136b200640 1 Processor -- start 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.454+0000 7f136b200640 1 -- start start 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 0x7f136c1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 0x7f136c1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f136c114240 con 0x7f136c10e6f0 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1140c0 con 0x7f136c100b40 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1143c0 con 0x7f136c1014f0 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40270/0 (socket says 172.21.15.52:40270) 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f136a800640 1 -- 172.21.15.52:0/495623553 learned_addr learned my addr 172.21.15.52:0/495623553 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f1361400640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 0x7f136c1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.455+0000 7f1369e00640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 0x7f136c1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 -- 172.21.15.52:0/495623553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 msgr2=0x7f136c1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 0x7f136c1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 -- 172.21.15.52:0/495623553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 msgr2=0x7f136c1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 0x7f136c1a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 -- 172.21.15.52:0/495623553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c19adf0 con 0x7f136c10e6f0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f1369e00640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 0x7f136c1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136a800640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f136400dac0 tx=0x7f136400df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f1363400640 1 -- 172.21.15.52:0/495623553 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364017030 con 0x7f136c10e6f0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f1363400640 1 -- 172.21.15.52:0/495623553 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1364003e20 con 0x7f136c10e6f0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f1363400640 1 -- 172.21.15.52:0/495623553 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364010460 con 0x7f136c10e6f0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.456+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c19b0e0 con 0x7f136c10e6f0 2024-04-26T01:56:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.457+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1a7330 con 0x7f136c10e6f0 2024-04-26T01:56:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.460+0000 7f1373f7b740 1 Processor -- start 2024-04-26T01:56:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.460+0000 7f1363400640 1 -- 172.21.15.52:0/495623553 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f136401a070 con 0x7f136c10e6f0 2024-04-26T01:56:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.460+0000 7f1373f7b740 1 -- start start 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564d8d646900 con 0x564d8d6e91c0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564d8d5ecef0 con 0x564d8d6f43a0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1373f7b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564d8d6428c0 con 0x564d8d6e9ae0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1361400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45300/0 (socket says 172.21.15.52:45300) 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 -- 172.21.15.52:0/4237968555 learned_addr learned my addr 172.21.15.52:0/4237968555 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1361400640 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 -- 172.21.15.52:0/4237968555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 msgr2=0x564d8d6f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 -- 172.21.15.52:0/4237968555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 msgr2=0x564d8d6e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6e95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1369e00640 1 -- 172.21.15.52:0/4237968555 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d8d359210 con 0x564d8d6e9ae0 2024-04-26T01:56:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.461+0000 7f1361400640 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f1369e00640 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f135c009900 tx=0x7f135c032c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ab3690486e4e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f1363400640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 0x7f13500a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f1363400640 1 -- 172.21.15.52:0/495623553 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f13640d05b0 con 0x7f136c10e6f0 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f135b400640 1 -- 172.21.15.52:0/4237968555 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c03f070 con 0x564d8d6e9ae0 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f1361400640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 0x7f13500a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f135b400640 1 -- 172.21.15.52:0/4237968555 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135c032e10 con 0x564d8d6e9ae0 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.462+0000 7f135b400640 1 -- 172.21.15.52:0/4237968555 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c035070 con 0x564d8d6e9ae0 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 -- 172.21.15.52:0/4237968555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 msgr2=0x564d8d6f3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1361400640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 0x7f13500a61d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1354009710 tx=0x7f1354005850 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f135c009900 tx=0x7f135c032c20 comp rx=0 tx=0).stop 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 -- 172.21.15.52:0/4237968555 shutdown_connections 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f3e60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 --2- 172.21.15.52:0/4237968555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 -- 172.21.15.52:0/4237968555 >> 172.21.15.52:0/4237968555 conn(0x564d8d6e56d0 msgr2=0x564d8d6e5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 -- 172.21.15.52:0/4237968555 shutdown_connections 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.463+0000 7f1373f7b740 1 -- 172.21.15.52:0/4237968555 wait complete. 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.464+0000 7f1373f7b740 1 -- start start 2024-04-26T01:56:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.464+0000 7f1373f7b740 1 Processor -- start 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.464+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6f1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564d8d6f6f70 con 0x564d8d6e9ae0 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564d8d600310 con 0x564d8d6f43a0 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564d8d6f70f0 con 0x564d8d6e91c0 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1373f7b740 10 client.0 _ll_register_callbacks cb 0x564d8d5cecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40280/0 (socket says 172.21.15.52:40280) 2024-04-26T01:56:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 -- 172.21.15.52:0/876928861 learned_addr learned my addr 172.21.15.52:0/876928861 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1369e00640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f136a800640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 -- 172.21.15.52:0/876928861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 msgr2=0x564d8d6f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6f1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 -- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 msgr2=0x564d8d6f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1361400640 1 -- 172.21.15.52:0/876928861 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d8d425620 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.465+0000 7f1369e00640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f1361400640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f136c0ffe90 tx=0x7f1354004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801be4334dd85eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f134fe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354009c80 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f1354019030 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f135c032e10 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1354011be0 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f1373f7b740 10 client.16374 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.466+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564d8d7031f0 con 0x564d8d6e9ae0 2024-04-26T01:56:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135c032e10 con 0x564d8d6e9ae0 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f135401e3f0 con 0x564d8d6e9ae0 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 10 client.16374 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 10 client.16374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 20 client.16374 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d704770 0x564d8d706bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 4 client.16374 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d8d65a8b0 con 0x564d8d704770 2024-04-26T01:56:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.467+0000 7f1369e00640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d704770 0x564d8d706bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.468+0000 7f1369e00640 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d704770 0x564d8d706bc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea777bc6eb06fdc server_cookie=17868b732640c3ac in_seq=0 out_seq=0 2024-04-26T01:56:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.468+0000 7f134fe00640 10 client.16374 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f134fe00640 1 -- 172.21.15.52:0/876928861 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d8d65a8b0 con 0x564d8d704770 2024-04-26T01:56:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f134fe00640 10 client.16374 handle_command_reply: tid=0 2024-04-26T01:56:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 client.16374 shutdown 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 msgr2=0x564d8d6f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f136c0ffe90 tx=0x7f1354004730 comp rx=0 tx=0).stop 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 shutdown_connections 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d704770 0x564d8d706bc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d6f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f1860 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 --2- 172.21.15.52:0/876928861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f134fe00640 0 client.16374 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 >> 172.21.15.52:0/876928861 conn(0x564d8d6e56d0 msgr2=0x564d8d6eb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 shutdown_connections 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.469+0000 7f1373f7b740 1 -- 172.21.15.52:0/876928861 wait complete. 2024-04-26T01:56:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.470+0000 7f1373f7b740 20 client.16374 trim_cache size 0 max 16384 2024-04-26T01:56:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.486+0000 7f1373f7b740 1 Processor -- start 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.486+0000 7f1373f7b740 1 -- start start 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.486+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6ffe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.486+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d70c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d70ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1373f7b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1ae190 con 0x564d8d6e9ae0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1373f7b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1ab9d0 con 0x564d8d6f43a0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1373f7b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1a9840 con 0x564d8d6e91c0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1361400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d70c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d70ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1361400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d70c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40288/0 (socket says 172.21.15.52:40288) 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1361400640 1 -- 172.21.15.52:0/3770972829 learned_addr learned my addr 172.21.15.52:0/3770972829 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f1369e00640 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6ffe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 -- 172.21.15.52:0/3770972829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 msgr2=0x564d8d6ffe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6ffe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 -- 172.21.15.52:0/3770972829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 msgr2=0x564d8d70c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d70c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.487+0000 7f136a800640 1 -- 172.21.15.52:0/3770972829 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d8d6e9910 con 0x564d8d6f43a0 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.488+0000 7f136a800640 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d70ca10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f13640cdca0 tx=0x7f136409cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be2223f26e67e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.488+0000 7f135aa00640 1 -- 172.21.15.52:0/3770972829 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640a08a0 con 0x564d8d6f43a0 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.488+0000 7f135aa00640 1 -- 172.21.15.52:0/3770972829 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13640a5160 con 0x564d8d6f43a0 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.488+0000 7f135aa00640 1 -- 172.21.15.52:0/3770972829 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640d00f0 con 0x564d8d6f43a0 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- 172.21.15.52:0/3770972829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 msgr2=0x564d8d70ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d70ca10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f13640cdca0 tx=0x7f136409cbf0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- 172.21.15.52:0/3770972829 shutdown_connections 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d70ca10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e9ae0 0x564d8d70c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 --2- 172.21.15.52:0/3770972829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e91c0 0x564d8d6ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- 172.21.15.52:0/3770972829 >> 172.21.15.52:0/3770972829 conn(0x564d8d6e56d0 msgr2=0x564d8d6ec5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- 172.21.15.52:0/3770972829 shutdown_connections 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- 172.21.15.52:0/3770972829 wait complete. 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 -- start start 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.489+0000 7f1373f7b740 1 Processor -- start 2024-04-26T01:56:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d768d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1a9e90 con 0x564d8d6e91c0 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1ab300 con 0x564d8d6f43a0 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1aa560 con 0x564d8d6e9ae0 2024-04-26T01:56:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1373f7b740 10 client.0 _ll_register_callbacks cb 0x7f13540218a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1369e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1369e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40290/0 (socket says 172.21.15.52:40290) 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1369e00640 1 -- 172.21.15.52:0/238759607 learned_addr learned my addr 172.21.15.52:0/238759607 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f1361400640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.490+0000 7f136a800640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d768d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 -- 172.21.15.52:0/238759607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 msgr2=0x564d8d6f9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 -- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 msgr2=0x564d8d768d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d768d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 -- 172.21.15.52:0/238759607 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d8d769280 con 0x564d8d6e91c0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f136a800640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d768d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1369e00640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f135c0052d0 tx=0x7f135c005260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f82248014879b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f134fe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c03f040 con 0x564d8d6e91c0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f135c04cdc0 con 0x564d8d6e91c0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.491+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13640a5160 con 0x564d8d6e91c0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f135c05b050 con 0x564d8d6e91c0 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 10 client.16380 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564d8d76a460 con 0x564d8d6e91c0 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13640a5160 con 0x564d8d6e91c0 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f135c04c740 con 0x564d8d6e91c0 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 10 client.16380 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 10 client.16380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 20 client.16380 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d76a8e0 0x564d8d76cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 4 client.16380 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.492+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d8d654320 con 0x564d8d76a8e0 2024-04-26T01:56:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.493+0000 7f1361400640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d76a8e0 0x564d8d76cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.493+0000 7f1361400640 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d76a8e0 0x564d8d76cd00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf45d544067b3da server_cookie=6dbf5a36247b6e6d in_seq=0 out_seq=0 2024-04-26T01:56:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.493+0000 7f134fe00640 10 client.16380 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.523+0000 7f134fe00640 1 -- 172.21.15.52:0/238759607 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d8d654320 con 0x564d8d76a8e0 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.523+0000 7f134fe00640 10 client.16380 handle_command_reply: tid=0 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.523+0000 7f1373f7b740 1 client.16380 shutdown 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.523+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 msgr2=0x564d8d6f9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.523+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f135c0052d0 tx=0x7f135c005260 comp rx=0 tx=0).stop 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 shutdown_connections 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564d8d76a8e0 0x564d8d76cd00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564d8d6f43a0 0x564d8d768d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564d8d6e9ae0 0x564d8d6f9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 --2- 172.21.15.52:0/238759607 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564d8d6e91c0 0x564d8d6f9340 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 >> 172.21.15.52:0/238759607 conn(0x564d8d6e56d0 msgr2=0x564d8d6f12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f134fe00640 0 client.16380 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 shutdown_connections 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 1 -- 172.21.15.52:0/238759607 wait complete. 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.524+0000 7f1373f7b740 20 client.16380 trim_cache size 0 max 16384 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:56:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:56:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 msgr2=0x7f13500a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 0x7f13500a61d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1354009710 tx=0x7f1354005850 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 msgr2=0x7f136c19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f136400dac0 tx=0x7f136400df80 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 shutdown_connections 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13500a3d10 0x7f13500a61d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f136c10e6f0 0x7f136c19a670 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f136c1014f0 0x7f136c1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 --2- 172.21.15.52:0/495623553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f136c100b40 0x7f136c1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 >> 172.21.15.52:0/495623553 conn(0x7f136c0fc790 msgr2=0x7f136c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 shutdown_connections 2024-04-26T01:56:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:40.525+0000 7f136b200640 1 -- 172.21.15.52:0/495623553 wait complete. 2024-04-26T01:56:41.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262651 2.35 MB/sec execute 480 sec latency 2828.399 ms 2024-04-26T01:56:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:41 smithi191 ceph-mon[32745]: pgmap v366: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:41.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:41 smithi183 ceph-mon[32752]: pgmap v366: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:41.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:41 smithi052 ceph-mon[27851]: pgmap v366: 97 pgs: 97 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 945 KiB/s wr, 80 op/s 2024-04-26T01:56:42.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262651 2.34 MB/sec execute 481 sec latency 3828.564 ms 2024-04-26T01:56:43.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262663 2.34 MB/sec execute 482 sec latency 4757.086 ms 2024-04-26T01:56:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:43 smithi191 ceph-mon[32745]: pgmap v367: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 994 KiB/s wr, 96 op/s 2024-04-26T01:56:43.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:43 smithi183 ceph-mon[32752]: pgmap v367: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 994 KiB/s wr, 96 op/s 2024-04-26T01:56:43.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:43 smithi052 ceph-mon[27851]: pgmap v367: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 994 KiB/s wr, 96 op/s 2024-04-26T01:56:44.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 262937 2.34 MB/sec execute 483 sec latency 28.798 ms 2024-04-26T01:56:45.138 INFO:tasks.workunit.client.0.smithi052.stdout: 1 263606 2.33 MB/sec execute 484 sec latency 29.268 ms 2024-04-26T01:56:45.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:45 smithi191 ceph-mon[32745]: pgmap v368: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 335 KiB/s wr, 56 op/s 2024-04-26T01:56:45.674 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:45 smithi183 ceph-mon[32752]: pgmap v368: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 335 KiB/s wr, 56 op/s 2024-04-26T01:56:45.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:45 smithi052 ceph-mon[27851]: pgmap v368: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 335 KiB/s wr, 56 op/s 2024-04-26T01:56:46.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 264204 2.34 MB/sec execute 485 sec latency 140.187 ms 2024-04-26T01:56:47.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 264204 2.33 MB/sec execute 486 sec latency 1140.322 ms 2024-04-26T01:56:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:47 smithi191 ceph-mon[32745]: pgmap v369: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 533 KiB/s wr, 81 op/s 2024-04-26T01:56:47.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:47 smithi183 ceph-mon[32752]: pgmap v369: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 533 KiB/s wr, 81 op/s 2024-04-26T01:56:47.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:47 smithi052 ceph-mon[27851]: pgmap v369: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 533 KiB/s wr, 81 op/s 2024-04-26T01:56:48.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 264287 2.33 MB/sec execute 487 sec latency 2067.931 ms 2024-04-26T01:56:49.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 265524 2.34 MB/sec execute 488 sec latency 34.711 ms 2024-04-26T01:56:49.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:49 smithi183 ceph-mon[32752]: pgmap v370: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:49 smithi052 ceph-mon[27851]: pgmap v370: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:49 smithi191 ceph-mon[32745]: pgmap v370: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:50.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266128 2.34 MB/sec execute 489 sec latency 512.988 ms 2024-04-26T01:56:51.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266128 2.34 MB/sec execute 490 sec latency 1513.134 ms 2024-04-26T01:56:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:51 smithi183 ceph-mon[32752]: pgmap v371: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:51 smithi052 ceph-mon[27851]: pgmap v371: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:51 smithi191 ceph-mon[32745]: pgmap v371: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 247 KiB/s wr, 42 op/s 2024-04-26T01:56:52.139 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266128 2.33 MB/sec execute 491 sec latency 2513.245 ms 2024-04-26T01:56:53.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266195 2.33 MB/sec execute 492 sec latency 3218.576 ms 2024-04-26T01:56:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 -- 172.21.15.52:0/341418801 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 msgr2=0x7f5f741032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 --2- 172.21.15.52:0/341418801 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741032f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009900 tx=0x7f5f6402a480 comp rx=0 tx=0).stop 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 -- 172.21.15.52:0/341418801 shutdown_connections 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 --2- 172.21.15.52:0/341418801 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 --2- 172.21.15.52:0/341418801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 0x7f5f7410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 --2- 172.21.15.52:0/341418801 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741032f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 -- 172.21.15.52:0/341418801 >> 172.21.15.52:0/341418801 conn(0x7f5f740fc770 msgr2=0x7f5f740feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 -- 172.21.15.52:0/341418801 shutdown_connections 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.674+0000 7f5f7a400640 1 -- 172.21.15.52:0/341418801 wait complete. 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.675+0000 7f5f7a400640 1 Processor -- start 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.675+0000 7f5f7a400640 1 -- start start 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 0x7f5f741a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5f74112010 con 0x7f5f74102f10 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5f74111e90 con 0x7f5f74103830 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f7a400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5f74112190 con 0x7f5f7410c330 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35666/0 (socket says 172.21.15.52:35666) 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 -- 172.21.15.52:0/3482866918 learned_addr learned my addr 172.21.15.52:0/3482866918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f6be00640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 0x7f5f741a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 -- 172.21.15.52:0/3482866918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 msgr2=0x7f5f7419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 -- 172.21.15.52:0/3482866918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 msgr2=0x7f5f741a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 0x7f5f741a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79000640 1 -- 172.21.15.52:0/3482866918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7419ad60 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.676+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f79000640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f64005ab0 tx=0x7f5f64003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f72000640 1 -- 172.21.15.52:0/3482866918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6403f070 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f72000640 1 -- 172.21.15.52:0/3482866918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f64031b60 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f72000640 1 -- 172.21.15.52:0/3482866918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64035030 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7419aff0 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.677+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f741a72f0 con 0x7f5f74102f10 2024-04-26T01:56:53.678 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:53 smithi183 ceph-mon[32752]: pgmap v372: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 378 KiB/s wr, 65 op/s 2024-04-26T01:56:53.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 Processor -- start 2024-04-26T01:56:53.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f72000640 1 -- 172.21.15.52:0/3482866918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5f64042070 con 0x7f5f74102f10 2024-04-26T01:56:53.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 -- start start 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b520977b90 0x55b520979f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b5208d2900 con 0x55b52096c9b0 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b520878ef0 con 0x55b52096d2d0 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.681+0000 7f5f7e803740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b5208ce8c0 con 0x55b520977b90 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35668/0 (socket says 172.21.15.52:35668) 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48732/0 (socket says 172.21.15.52:48732) 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79a00640 1 -- 172.21.15.52:0/3290869474 learned_addr learned my addr 172.21.15.52:0/3290869474 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:53.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f6be00640 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b520977b90 0x55b520979f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 -- 172.21.15.52:0/3290869474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b520977b90 msgr2=0x55b520979f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b520977b90 0x55b520979f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 -- 172.21.15.52:0/3290869474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 msgr2=0x55b52096cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79000640 1 -- 172.21.15.52:0/3290869474 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5205e5210 con 0x55b52096d2d0 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.682+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:53.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f79000640 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5f640ee590 tx=0x7f5f640f91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1e4f5b395fd4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f72000640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 0x7f5f580a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f6a000640 1 -- 172.21.15.52:0/3290869474 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f640b9130 con 0x55b52096d2d0 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f6a000640 1 -- 172.21.15.52:0/3290869474 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f640fe070 con 0x55b52096d2d0 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f6a000640 1 -- 172.21.15.52:0/3290869474 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f640bc610 con 0x55b52096d2d0 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 0x7f5f580a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.683+0000 7f5f72000640 1 -- 172.21.15.52:0/3482866918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f5f640f0e40 con 0x7f5f74102f10 2024-04-26T01:56:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 -- 172.21.15.52:0/3290869474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 msgr2=0x55b520977650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5f640ee590 tx=0x7f5f640f91a0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 0x7f5f580a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009fd0 tx=0x7f5f6c009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 -- 172.21.15.52:0/3290869474 shutdown_connections 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b520977b90 0x55b520979f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b520977650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3290869474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b52096cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 -- 172.21.15.52:0/3290869474 >> 172.21.15.52:0/3290869474 conn(0x55b520968ec0 msgr2=0x55b5209692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 -- 172.21.15.52:0/3290869474 shutdown_connections 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.684+0000 7f5f7e803740 1 -- 172.21.15.52:0/3290869474 wait complete. 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.685+0000 7f5f7e803740 1 -- start start 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.685+0000 7f5f7e803740 1 Processor -- start 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.685+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 0x55b520974ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.685+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b520975010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.685+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f7e803740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b52097a580 con 0x55b520977b90 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f7e803740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b52088c310 con 0x55b52096c9b0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f7e803740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b52097a700 con 0x55b52096d2d0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f7e803740 10 client.0 _ll_register_callbacks cb 0x55b52085acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 0x55b520974ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b520975010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35682/0 (socket says 172.21.15.52:35682) 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79a00640 1 -- 172.21.15.52:0/292714175 learned_addr learned my addr 172.21.15.52:0/292714175 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 -- 172.21.15.52:0/292714175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 msgr2=0x55b520975010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b520975010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 -- 172.21.15.52:0/292714175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 msgr2=0x55b520975550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79000640 1 -- 172.21.15.52:0/292714175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5206b1620 con 0x55b52096c9b0 2024-04-26T01:56:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.686+0000 7f5f79a00640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f79000640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 0x55b520974ad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f640a9510 tx=0x7f5f6402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e3d9d080550f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f62a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64049070 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f5f640f4b40 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f640fe070 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5f640bc7f0 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f7e803740 10 client.26525 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b52097a580 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.687+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f640fe070 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f5f640eece0 con 0x55b52096c9b0 2024-04-26T01:56:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 10 client.26525 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 10 client.26525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 20 client.26525 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b520987910 0x55b520989d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 4 client.26525 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:56:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5208e68b0 con 0x55b520987910 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.688+0000 7f5f6be00640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b520987910 0x55b520989d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.689+0000 7f5f6be00640 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b520987910 0x55b520989d40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbf6a341aad91723 server_cookie=d921ef4ea148090d in_seq=0 out_seq=0 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.689+0000 7f5f62a00640 10 client.26525 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f62a00640 1 -- 172.21.15.52:0/292714175 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5208e68b0 con 0x55b520987910 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f62a00640 10 client.26525 handle_command_reply: tid=0 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 client.26525 shutdown 2024-04-26T01:56:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 msgr2=0x55b520974ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 0x55b520974ad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f640a9510 tx=0x7f5f6402a480 comp rx=0 tx=0).stop 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 shutdown_connections 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b520987910 0x55b520989d40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b520975550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b520975010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 --2- 172.21.15.52:0/292714175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096c9b0 0x55b520974ad0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f62a00640 0 client.26525 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 >> 172.21.15.52:0/292714175 conn(0x55b520968ec0 msgr2=0x55b52096ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.690+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 shutdown_connections 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.691+0000 7f5f7e803740 1 -- 172.21.15.52:0/292714175 wait complete. 2024-04-26T01:56:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.691+0000 7f5f7e803740 20 client.26525 trim_cache size 0 max 16384 2024-04-26T01:56:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.707+0000 7f5f7e803740 1 Processor -- start 2024-04-26T01:56:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.707+0000 7f5f7e803740 1 -- start start 2024-04-26T01:56:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b5209833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b520977b90 0x55b52098f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741b2650 con 0x55b52096c9b0 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741afe90 con 0x55b520977b90 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f7e803740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741add00 con 0x55b52096d2d0 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f79000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b5209833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41500/0 (socket says 172.21.15.52:41500) 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f6be00640 1 -- 172.21.15.52:0/3675376270 learned_addr learned my addr 172.21.15.52:0/3675376270 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:53.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.708+0000 7f5f79a00640 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b520977b90 0x55b52098f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 -- 172.21.15.52:0/3675376270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b520977b90 msgr2=0x55b52098f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b520977b90 0x55b52098f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 -- 172.21.15.52:0/3675376270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 msgr2=0x55b5209833b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b5209833b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 -- 172.21.15.52:0/3675376270 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b520964820 con 0x55b52096d2d0 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f79000640 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b5209833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f6be00640 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f7419b720 tx=0x7f5f5c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160977f93721b2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f69600640 1 -- 172.21.15.52:0/3675376270 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c004230 con 0x55b52096d2d0 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f69600640 1 -- 172.21.15.52:0/3675376270 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f5c0043d0 con 0x55b52096d2d0 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.709+0000 7f5f69600640 1 -- 172.21.15.52:0/3675376270 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0201d0 con 0x55b52096d2d0 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- 172.21.15.52:0/3675376270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 msgr2=0x55b52096f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f7419b720 tx=0x7f5f5c005f90 comp rx=0 tx=0).stop 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- 172.21.15.52:0/3675376270 shutdown_connections 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b520977b90 0x55b52098f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096d2d0 0x55b52096f370 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 --2- 172.21.15.52:0/3675376270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b52096c9b0 0x55b5209833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- 172.21.15.52:0/3675376270 >> 172.21.15.52:0/3675376270 conn(0x55b520968ec0 msgr2=0x55b520975320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- 172.21.15.52:0/3675376270 shutdown_connections 2024-04-26T01:56:53.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- 172.21.15.52:0/3675376270 wait complete. 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.710+0000 7f5f7e803740 1 -- start start 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 Processor -- start 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b52097cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b52097d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741ae350 con 0x55b520977b90 2024-04-26T01:56:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741af7c0 con 0x55b52096d2d0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5f741aea20 con 0x55b52096c9b0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f7e803740 10 client.0 _ll_register_callbacks cb 0x7f5f640f4e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f79000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b52097cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f79a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b52097d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f79000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41512/0 (socket says 172.21.15.52:41512) 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.711+0000 7f5f79000640 1 -- 172.21.15.52:0/1318695777 learned_addr learned my addr 172.21.15.52:0/1318695777 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 -- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 msgr2=0x55b52097cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b52097cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 -- 172.21.15.52:0/1318695777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 msgr2=0x55b52097d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b52097d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 -- 172.21.15.52:0/1318695777 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b52097dd30 con 0x55b52096c9b0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f79000640 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000c00 tx=0x7f5f640f91a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9099774654a36c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f62a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f640b5080 con 0x55b52096c9b0 2024-04-26T01:56:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8521+0+0 (secure 0 0 0) 0x7f5f6403b460 con 0x55b52096c9b0 2024-04-26T01:56:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.712+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f5c0043d0 con 0x55b52096c9b0 2024-04-26T01:56:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5f640c0070 con 0x55b52096c9b0 2024-04-26T01:56:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 10 client.26536 fetch_fsmap learned FSMap version 47 2024-04-26T01:56:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b52097dd30 con 0x55b52096c9b0 2024-04-26T01:56:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f5c0043d0 con 0x55b52096c9b0 2024-04-26T01:56:53.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:53 smithi052 ceph-mon[27851]: pgmap v372: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 378 KiB/s wr, 65 op/s 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f5f640eece0 con 0x55b52096c9b0 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 10 client.26536 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 10 client.26536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 20 client.26536 populate_metadata read hostname 'smithi052' 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b5209ee230 0x55b5209f0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 4 client.26536 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.713+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5208e0320 con 0x55b5209ee230 2024-04-26T01:56:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.714+0000 7f5f6be00640 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b5209ee230 0x55b5209f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:56:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.714+0000 7f5f6be00640 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b5209ee230 0x55b5209f0650 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43fdfa549eb1c6c server_cookie=13ad556d62d20ffd in_seq=0 out_seq=0 2024-04-26T01:56:53.716 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.714+0000 7f5f62a00640 10 client.26536 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:53.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f62a00640 1 -- 172.21.15.52:0/1318695777 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5208e0320 con 0x55b5209ee230 2024-04-26T01:56:53.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f62a00640 10 client.26536 handle_command_reply: tid=0 2024-04-26T01:56:53.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 client.26536 shutdown 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 msgr2=0x55b52097fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000c00 tx=0x7f5f640f91a0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 shutdown_connections 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b5209ee230 0x55b5209f0650 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b520977b90 0x55b52097d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b52096d2d0 0x55b52097cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f62a00640 0 client.26536 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 --2- 172.21.15.52:0/1318695777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b52096c9b0 0x55b52097fc30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 >> 172.21.15.52:0/1318695777 conn(0x55b520968ec0 msgr2=0x55b520974a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.738+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 shutdown_connections 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7e803740 1 -- 172.21.15.52:0/1318695777 wait complete. 2024-04-26T01:56:53.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7e803740 20 client.26536 trim_cache size 0 max 16384 2024-04-26T01:56:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:56:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:56:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:56:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:56:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 msgr2=0x7f5f580a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 0x7f5f580a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009fd0 tx=0x7f5f6c009500 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 msgr2=0x7f5f741a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.739+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f64005ab0 tx=0x7f5f64003c10 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 shutdown_connections 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5f580a3d90 0x7f5f580a6250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f7410c330 0x7f5f7419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5f74103830 0x7f5f741a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 --2- 172.21.15.52:0/3482866918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f74102f10 0x7f5f741a0430 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 >> 172.21.15.52:0/3482866918 conn(0x7f5f740fc770 msgr2=0x7f5f74100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 shutdown_connections 2024-04-26T01:56:53.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:56:53.740+0000 7f5f7a400640 1 -- 172.21.15.52:0/3482866918 wait complete. 2024-04-26T01:56:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:53 smithi191 ceph-mon[32745]: pgmap v372: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 378 KiB/s wr, 65 op/s 2024-04-26T01:56:54.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266195 2.32 MB/sec execute 493 sec latency 1064.496 ms 2024-04-26T01:56:55.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266195 2.32 MB/sec execute 494 sec latency 2064.663 ms 2024-04-26T01:56:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:55 smithi183 ceph-mon[32752]: pgmap v373: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 329 KiB/s wr, 48 op/s 2024-04-26T01:56:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:55 smithi052 ceph-mon[27851]: pgmap v373: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 329 KiB/s wr, 48 op/s 2024-04-26T01:56:55.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:55 smithi191 ceph-mon[32745]: pgmap v373: 97 pgs: 97 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 329 KiB/s wr, 48 op/s 2024-04-26T01:56:56.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266332 2.32 MB/sec execute 495 sec latency 2634.877 ms 2024-04-26T01:56:57.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 266605 2.31 MB/sec execute 496 sec latency 25.078 ms 2024-04-26T01:56:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:57 smithi052 ceph-mon[27851]: pgmap v374: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 974 KiB/s wr, 79 op/s 2024-04-26T01:56:57.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:57 smithi191 ceph-mon[32745]: pgmap v374: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 974 KiB/s wr, 79 op/s 2024-04-26T01:56:57.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:57 smithi183 ceph-mon[32752]: pgmap v374: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 974 KiB/s wr, 79 op/s 2024-04-26T01:56:58.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 267298 2.31 MB/sec execute 497 sec latency 32.764 ms 2024-04-26T01:56:59.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 267752 2.31 MB/sec execute 498 sec latency 327.247 ms 2024-04-26T01:56:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:59 smithi052 ceph-mon[27851]: pgmap v375: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:56:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:56:59 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:59.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:59 smithi191 ceph-mon[32745]: pgmap v375: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:56:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:56:59 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:56:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:59 smithi183 ceph-mon[32752]: pgmap v375: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:56:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:56:59 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:00.140 INFO:tasks.workunit.client.0.smithi052.stdout: 1 267752 2.31 MB/sec execute 499 sec latency 1327.364 ms 2024-04-26T01:57:01.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 267752 2.30 MB/sec execute 500 sec latency 2327.497 ms 2024-04-26T01:57:01.704 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:01 smithi191 ceph-mon[32745]: pgmap v376: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:57:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:01 smithi052 ceph-mon[27851]: pgmap v376: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:57:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:01 smithi183 ceph-mon[32752]: pgmap v376: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 777 KiB/s wr, 53 op/s 2024-04-26T01:57:02.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 267752 2.30 MB/sec execute 501 sec latency 3327.658 ms 2024-04-26T01:57:03.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 268117 2.30 MB/sec execute 502 sec latency 4038.751 ms 2024-04-26T01:57:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:03 smithi052 ceph-mon[27851]: pgmap v377: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 979 KiB/s wr, 87 op/s 2024-04-26T01:57:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:03 smithi191 ceph-mon[32745]: pgmap v377: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 979 KiB/s wr, 87 op/s 2024-04-26T01:57:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:03 smithi183 ceph-mon[32752]: pgmap v377: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 979 KiB/s wr, 87 op/s 2024-04-26T01:57:04.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 268558 2.30 MB/sec execute 503 sec latency 61.165 ms 2024-04-26T01:57:04.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:04 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:04.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:04 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:04 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:05.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269409 2.30 MB/sec execute 504 sec latency 460.014 ms 2024-04-26T01:57:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:05 smithi191 ceph-mon[32745]: pgmap v378: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 847 KiB/s wr, 65 op/s 2024-04-26T01:57:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:05 smithi183 ceph-mon[32752]: pgmap v378: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 847 KiB/s wr, 65 op/s 2024-04-26T01:57:05.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:05 smithi052 ceph-mon[27851]: pgmap v378: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 847 KiB/s wr, 65 op/s 2024-04-26T01:57:06.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269435 2.30 MB/sec execute 505 sec latency 1281.080 ms 2024-04-26T01:57:06.431 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:57:06.431 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-04-26T01:57:06.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 -- 172.21.15.52:0/2643562305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 msgr2=0x7f4764112ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 --2- 172.21.15.52:0/2643562305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f4764112ad0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4750009930 tx=0x7f475002a4d0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 -- 172.21.15.52:0/2643562305 shutdown_connections 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 --2- 172.21.15.52:0/2643562305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764113010 0x7f4764115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 --2- 172.21.15.52:0/2643562305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f4764112ad0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 --2- 172.21.15.52:0/2643562305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764074250 0x7f4764074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 -- 172.21.15.52:0/2643562305 >> 172.21.15.52:0/2643562305 conn(0x7f47640793e0 msgr2=0x7f47640797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 -- 172.21.15.52:0/2643562305 shutdown_connections 2024-04-26T01:57:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.883+0000 7f4768c00640 1 -- 172.21.15.52:0/2643562305 wait complete. 2024-04-26T01:57:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.884+0000 7f4768c00640 1 Processor -- start 2024-04-26T01:57:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 -- start start 2024-04-26T01:57:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f47641a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 0x7f476419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4764118860 con 0x7f4764113010 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f47641186e0 con 0x7f4764074b70 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.885+0000 7f4768c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f47641189e0 con 0x7f4764074250 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f47641a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f47641a4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38816/0 (socket says 172.21.15.52:38816) 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53510/0 (socket says 172.21.15.52:53510) 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761400640 1 -- 172.21.15.52:0/1981982330 learned_addr learned my addr 172.21.15.52:0/1981982330 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4762800640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 0x7f476419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 -- 172.21.15.52:0/1981982330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 msgr2=0x7f47641a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f47641a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 -- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 msgr2=0x7f476419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 0x7f476419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 -- 172.21.15.52:0/1981982330 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476419f3e0 con 0x7f4764074250 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4762800640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 0x7f476419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.886+0000 7f4761e00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f475400e9f0 tx=0x7f475400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.887+0000 7f475aa00640 1 -- 172.21.15.52:0/1981982330 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475400cd80 con 0x7f4764074250 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.887+0000 7f475aa00640 1 -- 172.21.15.52:0/1981982330 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47540035f0 con 0x7f4764074250 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.887+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476419f6d0 con 0x7f4764074250 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.887+0000 7f475aa00640 1 -- 172.21.15.52:0/1981982330 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4754010670 con 0x7f4764074250 2024-04-26T01:57:06.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.887+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47641ab9e0 con 0x7f4764074250 2024-04-26T01:57:06.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.890+0000 7f475aa00640 1 -- 172.21.15.52:0/1981982330 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f47540037a0 con 0x7f4764074250 2024-04-26T01:57:06.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 Processor -- start 2024-04-26T01:57:06.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 -- start start 2024-04-26T01:57:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 0x55e768f94c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 0x55e768f9f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f9fa00 0x55e768fa1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e768efa900 con 0x55e768f9fa00 2024-04-26T01:57:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e768ea0ef0 con 0x55e768f95140 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.891+0000 7f476bb84740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e768ef68c0 con 0x55e768f94820 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4761e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 0x55e768f9f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4761400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f9fa00 0x55e768fa1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 0x55e768f94c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4761e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 0x55e768f9f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38828/0 (socket says 172.21.15.52:38828) 2024-04-26T01:57:06.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4761e00640 1 -- 172.21.15.52:0/2937606123 learned_addr learned my addr 172.21.15.52:0/2937606123 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 -- 172.21.15.52:0/2937606123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 msgr2=0x55e768f9f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 0x55e768f9f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 -- 172.21.15.52:0/2937606123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f9fa00 msgr2=0x55e768fa1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f9fa00 0x55e768fa1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f4762800640 1 -- 172.21.15.52:0/2937606123 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e768c0d210 con 0x55e768f94820 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.892+0000 7f475aa00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 0x7f47480a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f475aa00640 1 -- 172.21.15.52:0/1981982330 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8565+0+0 (secure 0 0 0) 0x7f47540d0bb0 con 0x7f4764074250 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f4762800640 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 0x55e768f94c00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f475c005fd0 tx=0x7f475c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e751f594ce5fb9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f4761e00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 0x7f47480a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f474f000640 1 -- 172.21.15.52:0/2937606123 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c012650 con 0x55e768f94820 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f474f000640 1 -- 172.21.15.52:0/2937606123 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f475c012ca0 con 0x55e768f94820 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.893+0000 7f474f000640 1 -- 172.21.15.52:0/2937606123 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c00fdb0 con 0x55e768f94820 2024-04-26T01:57:06.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- 172.21.15.52:0/2937606123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 msgr2=0x55e768f94c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f4761e00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 0x7f47480a6280 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f475400cfd0 tx=0x7f47540d4660 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 0x55e768f94c00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f475c005fd0 tx=0x7f475c0117b0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- 172.21.15.52:0/2937606123 shutdown_connections 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f9fa00 0x55e768fa1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f95140 0x55e768f9f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 --2- 172.21.15.52:0/2937606123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f94820 0x55e768f94c00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- 172.21.15.52:0/2937606123 >> 172.21.15.52:0/2937606123 conn(0x55e768f90d30 msgr2=0x55e768f91110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- 172.21.15.52:0/2937606123 shutdown_connections 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- 172.21.15.52:0/2937606123 wait complete. 2024-04-26T01:57:06.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.894+0000 7f476bb84740 1 -- start start 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 Processor -- start 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768f9c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768f9cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768f9d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e768fa2630 con 0x55e768f95140 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e768eb4310 con 0x55e768f94820 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e768fa27b0 con 0x55e768f9fa00 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.895+0000 7f476bb84740 10 client.0 _ll_register_callbacks cb 0x55e768e82cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4762800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768f9c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4762800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768f9c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38832/0 (socket says 172.21.15.52:38832) 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4762800640 1 -- 172.21.15.52:0/2113668393 learned_addr learned my addr 172.21.15.52:0/2113668393 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768f9cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4762800640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768f9d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 -- 172.21.15.52:0/2113668393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 msgr2=0x55e768f9d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768f9d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 -- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 msgr2=0x55e768f9c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768f9c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.896+0000 7f4761400640 1 -- 172.21.15.52:0/2113668393 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e768cd9620 con 0x55e768f95140 2024-04-26T01:57:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f4761400640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768f9cf00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4750004770 tx=0x7f47500043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d213c3c4d7bc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f4743400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:57:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750035040 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8565+0+0 (secure 0 0 0) 0x7f4750005220 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f475c012ca0 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4750005420 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f476bb84740 10 client.16410 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.897+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e768fae8c0 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f475c012ca0 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f475004c070 con 0x55e768f95140 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 10 client.16410 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 20 client.16410 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e768fafd80 0x55e768fb2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 4 client.16410 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e768f0e8b0 con 0x55e768fafd80 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.898+0000 7f4762800640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e768fafd80 0x55e768fb2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.899+0000 7f4762800640 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e768fafd80 0x55e768fb2170 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06f794a8b4cf134 server_cookie=43d38eb0dd2cdcb0 in_seq=0 out_seq=0 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.899+0000 7f4743400640 10 client.16410 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.899+0000 7f4743400640 1 -- 172.21.15.52:0/2113668393 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e768f0e8b0 con 0x55e768fafd80 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.899+0000 7f4743400640 10 client.16410 handle_command_reply: tid=0 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 client.16410 shutdown 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 msgr2=0x55e768f9cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768f9cf00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4750004770 tx=0x7f47500043b0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 shutdown_connections 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e768fafd80 0x55e768fb2170 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768f9d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768f9cf00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f4743400640 0 client.16410 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:06.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 --2- 172.21.15.52:0/2113668393 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768f9c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 >> 172.21.15.52:0/2113668393 conn(0x55e768f90d30 msgr2=0x55e768f96a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 shutdown_connections 2024-04-26T01:57:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 1 -- 172.21.15.52:0/2113668393 wait complete. 2024-04-26T01:57:06.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.900+0000 7f476bb84740 20 client.16410 trim_cache size 0 max 16384 2024-04-26T01:57:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 Processor -- start 2024-04-26T01:57:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 -- start start 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f94820 0x55e768fab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 0x55e768f9d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b6b40 con 0x55e768f94820 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b4380 con 0x55e768f9fa00 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.930+0000 7f476bb84740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b21f0 con 0x55e768f95140 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4762800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f94820 0x55e768fab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53524/0 (socket says 172.21.15.52:53524) 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 -- 172.21.15.52:0/3901947422 learned_addr learned my addr 172.21.15.52:0/3901947422 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4762800640 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 0x55e768f9d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 -- 172.21.15.52:0/3901947422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 msgr2=0x55e768f9d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 0x55e768f9d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 -- 172.21.15.52:0/3901947422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f94820 msgr2=0x55e768fab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f94820 0x55e768fab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 -- 172.21.15.52:0/3901947422 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e768fb27f0 con 0x55e768f95140 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4762800640 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 0x55e768f9d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.931+0000 7f4761400640 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4750009850 tx=0x7f475002a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c3633b14420df87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.932+0000 7f474e600640 1 -- 172.21.15.52:0/3901947422 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750035070 con 0x55e768f95140 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.932+0000 7f474e600640 1 -- 172.21.15.52:0/3901947422 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4750003a60 con 0x55e768f95140 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.932+0000 7f474e600640 1 -- 172.21.15.52:0/3901947422 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750052070 con 0x55e768f95140 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- 172.21.15.52:0/3901947422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 msgr2=0x55e768f90610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4750009850 tx=0x7f475002a4d0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- 172.21.15.52:0/3901947422 shutdown_connections 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f9fa00 0x55e768f9d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f95140 0x55e768f90610 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 --2- 172.21.15.52:0/3901947422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f94820 0x55e768fab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- 172.21.15.52:0/3901947422 >> 172.21.15.52:0/3901947422 conn(0x55e768f90d30 msgr2=0x55e768f9c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- 172.21.15.52:0/3901947422 shutdown_connections 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- 172.21.15.52:0/3901947422 wait complete. 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.933+0000 7f476bb84740 1 -- start start 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 Processor -- start 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768fa58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768fa5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b2840 con 0x55e768f95140 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b3cb0 con 0x55e768f94820 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f47641b2f10 con 0x55e768f9fa00 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.934+0000 7f476bb84740 10 client.0 _ll_register_callbacks cb 0x7f4750004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38836/0 (socket says 172.21.15.52:38836) 2024-04-26T01:57:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4761400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768fa58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 -- 172.21.15.52:0/2074400483 learned_addr learned my addr 172.21.15.52:0/2074400483 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768fa5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 -- 172.21.15.52:0/2074400483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 msgr2=0x55e768fa5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768fa5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 -- 172.21.15.52:0/2074400483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 msgr2=0x55e768fa58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768fa58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 -- 172.21.15.52:0/2074400483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e769015300 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4762800640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f475c014010 tx=0x7f475c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecc3a7808f815c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4743400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.935+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c00fea0 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8565+0+0 (secure 0 0 0) 0x7f475c0049c0 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4750003a60 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f475c004bc0 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 10 client.26546 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e7690164c0 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4750003a60 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f475c026040 con 0x55e768f94820 2024-04-26T01:57:06.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 10 client.26546 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 10 client.26546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 20 client.26546 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e769016a60 0x55e769018e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 4 client.26546 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.936+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e768f08320 con 0x55e769016a60 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.937+0000 7f4761400640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e769016a60 0x55e769018e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.937+0000 7f4761400640 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e769016a60 0x55e769018e80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454e77832fd7e28e server_cookie=70d6261e3369d91 in_seq=0 out_seq=0 2024-04-26T01:57:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.937+0000 7f4743400640 10 client.26546 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f4743400640 1 -- 172.21.15.52:0/2074400483 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e768f08320 con 0x55e769016a60 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f4743400640 10 client.26546 handle_command_reply: tid=0 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 client.26546 shutdown 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 msgr2=0x55e768fa53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f475c014010 tx=0x7f475c009500 comp rx=0 tx=0).stop 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 shutdown_connections 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e769016a60 0x55e769018e80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e768f9fa00 0x55e768fa5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e768f95140 0x55e768fa58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f4743400640 0 client.26546 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 --2- 172.21.15.52:0/2074400483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e768f94820 0x55e768fa53a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 >> 172.21.15.52:0/2074400483 conn(0x55e768f90d30 msgr2=0x55e768f9d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 shutdown_connections 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.972+0000 7f476bb84740 1 -- 172.21.15.52:0/2074400483 wait complete. 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f476bb84740 20 client.26546 trim_cache size 0 max 16384 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:57:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:57:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 msgr2=0x7f47480a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 0x7f47480a6280 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f475400cfd0 tx=0x7f47540d4660 comp rx=0 tx=0).stop 2024-04-26T01:57:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 msgr2=0x7f47641a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f475400e9f0 tx=0x7f475400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 shutdown_connections 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f47480a3dc0 0x7f47480a6280 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4764113010 0x7f476419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4764074b70 0x7f47641a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.973+0000 7f4768c00640 1 --2- 172.21.15.52:0/1981982330 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4764074250 0x7f47641a4a60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.974+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 >> 172.21.15.52:0/1981982330 conn(0x7f47640793e0 msgr2=0x7f47640fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.974+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 shutdown_connections 2024-04-26T01:57:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:06.974+0000 7f4768c00640 1 -- 172.21.15.52:0/1981982330 wait complete. 2024-04-26T01:57:06.974 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:07.141 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269675 2.30 MB/sec execute 506 sec latency 930.745 ms 2024-04-26T01:57:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:07 smithi191 ceph-mon[32745]: pgmap v379: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-26T01:57:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:07 smithi191 ceph-mon[32745]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:57:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:07 smithi183 ceph-mon[32752]: pgmap v379: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-26T01:57:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:07 smithi183 ceph-mon[32752]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:57:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:07 smithi052 ceph-mon[27851]: pgmap v379: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-26T01:57:07.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:07 smithi052 ceph-mon[27851]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T01:57:08.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269682 2.30 MB/sec execute 507 sec latency 1004.261 ms 2024-04-26T01:57:09.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269742 2.29 MB/sec execute 508 sec latency 789.991 ms 2024-04-26T01:57:09.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 -- 172.21.15.52:0/3185213367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4104ee0 msgr2=0x7f0fe41052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 --2- 172.21.15.52:0/3185213367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4104ee0 0x7f0fe41052c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40098f0 tx=0x7f0fd4009c00 comp rx=0 tx=0).stop 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 -- 172.21.15.52:0/3185213367 shutdown_connections 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 --2- 172.21.15.52:0/3185213367 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe4111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 --2- 172.21.15.52:0/3185213367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4104ee0 0x7f0fe41052c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 --2- 172.21.15.52:0/3185213367 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4105890 0x7f0fe41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.607+0000 7f0fea400700 1 -- 172.21.15.52:0/3185213367 >> 172.21.15.52:0/3185213367 conn(0x7f0fe4100770 msgr2=0x7f0fe4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.608+0000 7f0fea400700 1 -- 172.21.15.52:0/3185213367 shutdown_connections 2024-04-26T01:57:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.608+0000 7f0fea400700 1 -- 172.21.15.52:0/3185213367 wait complete. 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 Processor -- start 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 -- start start 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4104ee0 0x7f0fe419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4105890 0x7f0fe419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe4114eb0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe4114d30 con 0x7f0fe4105890 2024-04-26T01:57:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.609+0000 7f0fea400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe4115030 con 0x7f0fe4104ee0 2024-04-26T01:57:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe2a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4105890 0x7f0fe419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40240/0 (socket says 172.21.15.52:40240) 2024-04-26T01:57:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 -- 172.21.15.52:0/2502131240 learned_addr learned my addr 172.21.15.52:0/2502131240 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4104ee0 0x7f0fe419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 -- 172.21.15.52:0/2502131240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4104ee0 msgr2=0x7f0fe419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4104ee0 0x7f0fe419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 -- 172.21.15.52:0/2502131240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4105890 msgr2=0x7f0fe419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4105890 0x7f0fe419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.610+0000 7f0fe3e00700 1 -- 172.21.15.52:0/2502131240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe41a03d0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fe3e00700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00c370 tx=0x7f0fdc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc0092e0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fdc003a00 con 0x7f0fe4109fc0 2024-04-26T01:57:09.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe41a0660 con 0x7f0fe4109fc0 2024-04-26T01:57:09.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc0046f0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.611+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe41a7f40 con 0x7f0fe4109fc0 2024-04-26T01:57:09.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.612+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe4069730 con 0x7f0fe4109fc0 2024-04-26T01:57:09.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.615+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0fdc01a020 con 0x7f0fe4109fc0 2024-04-26T01:57:09.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.616+0000 7f0fdbe00700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 0x7f0fc80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.616+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8565+0+0 (secure 0 0 0) 0x7f0fdc0d06c0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.617+0000 7f0fe3400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 0x7f0fc80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:09.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.617+0000 7f0fe3400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 0x7f0fc80a6270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40098c0 tx=0x7f0fd4019040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:09.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.619+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fdc010040 con 0x7f0fe4109fc0 2024-04-26T01:57:09.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.901+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fe402df80 con 0x7f0fe4109fc0 2024-04-26T01:57:09.900 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:09 smithi052 ceph-mon[27851]: pgmap v381: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:09.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.903+0000 7f0fdbe00700 1 -- 172.21.15.52:0/2502131240 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11889 (secure 0 0 0) 0x7f0fdc09ccd0 con 0x7f0fe4109fc0 2024-04-26T01:57:09.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:57:09.902 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:57:02.826434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:57:09.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:09 smithi191 ceph-mon[32745]: pgmap v381: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 msgr2=0x7f0fc80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 0x7f0fc80a6270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40098c0 tx=0x7f0fd4019040 comp rx=0 tx=0).stop 2024-04-26T01:57:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 msgr2=0x7f0fe419fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:09.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00c370 tx=0x7f0fdc00c680 comp rx=0 tx=0).stop 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 shutdown_connections 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0fc80a3db0 0x7f0fc80a6270 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0fe4109fc0 0x7f0fe419fc90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fe4104ee0 0x7f0fe419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 --2- 172.21.15.52:0/2502131240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe4105890 0x7f0fe419b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 >> 172.21.15.52:0/2502131240 conn(0x7f0fe4100770 msgr2=0x7f0fe4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 shutdown_connections 2024-04-26T01:57:09.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:09.906+0000 7f0fea400700 1 -- 172.21.15.52:0/2502131240 wait complete. 2024-04-26T01:57:09.906 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 49 2024-04-26T01:57:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:09 smithi183 ceph-mon[32752]: pgmap v381: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:10.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269742 2.29 MB/sec execute 509 sec latency 1790.145 ms 2024-04-26T01:57:10.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:10 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2502131240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:10.741 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0' 2024-04-26T01:57:10.741 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0 2024-04-26T01:57:10.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:10 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2502131240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:10.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:10 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2502131240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:11.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269742 2.28 MB/sec execute 510 sec latency 2790.234 ms 2024-04-26T01:57:11.219 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:11.633 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:11 smithi052 ceph-mon[27851]: pgmap v382: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:11.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:11 smithi191 ceph-mon[32745]: pgmap v382: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:11 smithi183 ceph-mon[32752]: pgmap v382: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 546 KiB/s wr, 106 op/s 2024-04-26T01:57:12.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269742 2.28 MB/sec execute 511 sec latency 3790.318 ms 2024-04-26T01:57:13.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269754 2.28 MB/sec execute 512 sec latency 4720.304 ms 2024-04-26T01:57:13.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 -- 172.21.15.52:0/817062302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 msgr2=0x7fdd4c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 --2- 172.21.15.52:0/817062302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c075050 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009990 tx=0x7fdd3c009ca0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 -- 172.21.15.52:0/817062302 shutdown_connections 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 --2- 172.21.15.52:0/817062302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 --2- 172.21.15.52:0/817062302 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c075050 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 --2- 172.21.15.52:0/817062302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 -- 172.21.15.52:0/817062302 >> 172.21.15.52:0/817062302 conn(0x7fdd4c079ba0 msgr2=0x7fdd4c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 -- 172.21.15.52:0/817062302 shutdown_connections 2024-04-26T01:57:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.474+0000 7fdd50e00700 1 -- 172.21.15.52:0/817062302 wait complete. 2024-04-26T01:57:13.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 Processor -- start 2024-04-26T01:57:13.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 -- start start 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c114e80 con 0x7fdd4c10f4a0 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c114d00 con 0x7fdd4c075590 2024-04-26T01:57:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.477+0000 7fdd50e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c115000 con 0x7fdd4c074c70 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53554/0 (socket says 172.21.15.52:53554) 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 -- 172.21.15.52:0/3006632425 learned_addr learned my addr 172.21.15.52:0/3006632425 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 -- 172.21.15.52:0/3006632425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 msgr2=0x7fdd4c19b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd4a800700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c19b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 -- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 msgr2=0x7fdd4c19fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c19fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49e00700 1 -- 172.21.15.52:0/3006632425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c1a0240 con 0x7fdd4c074c70 2024-04-26T01:57:13.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd4a800700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:13.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.478+0000 7fdd49400700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:13.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.479+0000 7fdd49e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000c00 tx=0x7fdd3c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.479+0000 7fdd42a00700 1 -- 172.21.15.52:0/3006632425 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c027070 con 0x7fdd4c074c70 2024-04-26T01:57:13.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.479+0000 7fdd42a00700 1 -- 172.21.15.52:0/3006632425 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c022470 con 0x7fdd4c074c70 2024-04-26T01:57:13.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.479+0000 7fdd42a00700 1 -- 172.21.15.52:0/3006632425 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c017650 con 0x7fdd4c074c70 2024-04-26T01:57:13.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.479+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4c1a04d0 con 0x7fdd4c074c70 2024-04-26T01:57:13.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.480+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4c1a7db0 con 0x7fdd4c074c70 2024-04-26T01:57:13.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.485+0000 7fdd42a00700 1 -- 172.21.15.52:0/3006632425 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdd3c02a070 con 0x7fdd4c074c70 2024-04-26T01:57:13.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.487+0000 7fdd42a00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 0x7fdd300a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.488+0000 7fdd42a00700 1 -- 172.21.15.52:0/3006632425 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fdd3c0d15f0 con 0x7fdd4c074c70 2024-04-26T01:57:13.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.488+0000 7fdd49400700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 0x7fdd300a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.489+0000 7fdd5b827b80 1 Processor -- start 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.489+0000 7fdd49400700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 0x7fdd300a6410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009680 tx=0x7fdd38009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.489+0000 7fdd5b827b80 1 -- start start 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6e3560 0x559f4e6e5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559f4e62b020 con 0x559f4e6d86f0 2024-04-26T01:57:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559f4e5d18c0 con 0x559f4e6e3560 2024-04-26T01:57:13.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559f4e626fe0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd49400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53562/0 (socket says 172.21.15.52:53562) 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd49e00700 1 -- 172.21.15.52:0/1779227095 learned_addr learned my addr 172.21.15.52:0/1779227095 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.490+0000 7fdd4a800700 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6e3560 0x559f4e6e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 -- 172.21.15.52:0/1779227095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6e3560 msgr2=0x559f4e6e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6e3560 0x559f4e6e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 -- 172.21.15.52:0/1779227095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 msgr2=0x559f4e6e3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 -- 172.21.15.52:0/1779227095 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4e5c7810 con 0x559f4e6d7dd0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49400700 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd49e00700 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c024000 tx=0x7fdd3c09d100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372b04b0f5d6aa0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd37400700 1 -- 172.21.15.52:0/1779227095 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c09d5c0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd37400700 1 -- 172.21.15.52:0/1779227095 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c0a1970 con 0x559f4e6d7dd0 2024-04-26T01:57:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.491+0000 7fdd37400700 1 -- 172.21.15.52:0/1779227095 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c09d820 con 0x559f4e6d7dd0 2024-04-26T01:57:13.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1779227095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 msgr2=0x559f4e6d81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c024000 tx=0x7fdd3c09d100 comp rx=0 tx=0).stop 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1779227095 shutdown_connections 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6e3560 0x559f4e6e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6d81b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1779227095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1779227095 >> 172.21.15.52:0/1779227095 conn(0x559f4e6070e0 msgr2=0x559f4e6d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1779227095 shutdown_connections 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.492+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1779227095 wait complete. 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.493+0000 7fdd5b827b80 1 -- start start 2024-04-26T01:57:13.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.493+0000 7fdd5b827b80 1 Processor -- start 2024-04-26T01:57:13.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6e0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559f4e6e5e90 con 0x559f4e6d86f0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559f4e5e49f0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559f4e6e6010 con 0x559f4e6e3560 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd5b827b80 10 client.0 _ll_register_callbacks cb 0x559f4e5c38b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38906/0 (socket says 172.21.15.52:38906) 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd49e00700 1 -- 172.21.15.52:0/583234116 learned_addr learned my addr 172.21.15.52:0/583234116 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.494+0000 7fdd49400700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 -- 172.21.15.52:0/583234116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 msgr2=0x559f4e6e0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd4a800700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6e0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6e0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 -- 172.21.15.52:0/583234116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 msgr2=0x559f4e6e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 -- 172.21.15.52:0/583234116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4e578a60 con 0x559f4e6d7dd0 2024-04-26T01:57:13.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49400700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd4a800700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6e0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:13.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd49e00700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c099f90 tx=0x7fdd3c09dd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bdb9c63ee83c5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd2be00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:13.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.495+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c09d5c0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fdd3c09da30 con 0x559f4e6d7dd0 2024-04-26T01:57:13.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3c0a1970 con 0x559f4e6d7dd0 2024-04-26T01:57:13.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdd3c0bc2a0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd5b827b80 10 client.26564 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559f4e5c1c80 con 0x559f4e6d7dd0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c0a1970 con 0x559f4e6d7dd0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.496+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fdd3c09ca10 con 0x559f4e6d7dd0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 10 client.26564 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 10 client.26564 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 20 client.26564 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e6f2990 0x559f4e6f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 4 client.26564 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f4e638810 con 0x559f4e6f2990 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.497+0000 7fdd49400700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e6f2990 0x559f4e6f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.498+0000 7fdd49400700 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e6f2990 0x559f4e6f4d80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edb5d75ac3c755a0 server_cookie=11d465835757f674 in_seq=0 out_seq=0 2024-04-26T01:57:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.498+0000 7fdd2be00700 10 client.26564 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:13.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd2be00700 1 -- 172.21.15.52:0/583234116 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f4e638810 con 0x559f4e6f2990 2024-04-26T01:57:13.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd2be00700 10 client.26564 handle_command_reply: tid=0 2024-04-26T01:57:13.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 client.26564 shutdown 2024-04-26T01:57:13.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 msgr2=0x559f4e6e2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c099f90 tx=0x7fdd3c09dd80 comp rx=0 tx=0).stop 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 shutdown_connections 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e6f2990 0x559f4e6f4d80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6e0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e2d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/583234116 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d86f0 0x559f4e6e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 >> 172.21.15.52:0/583234116 conn(0x559f4e6070e0 msgr2=0x559f4e6d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd2be00700 0 client.26564 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 shutdown_connections 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 1 -- 172.21.15.52:0/583234116 wait complete. 2024-04-26T01:57:13.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.499+0000 7fdd5b827b80 20 client.26564 trim_cache size 0 max 16384 2024-04-26T01:57:13.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.525+0000 7fdd5b827b80 1 Processor -- start 2024-04-26T01:57:13.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.525+0000 7fdd5b827b80 1 -- start start 2024-04-26T01:57:13.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:13 smithi052 ceph-mon[27851]: pgmap v383: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 388 KiB/s wr, 89 op/s 2024-04-26T01:57:13.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:13 smithi052 ceph-mon[27851]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1b1cc0 con 0x559f4e6d7dd0 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1af4c0 con 0x559f4e6d86f0 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1ad330 con 0x559f4e6e3560 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd4a800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.526+0000 7fdd49400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53582/0 (socket says 172.21.15.52:53582) 2024-04-26T01:57:13.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 -- 172.21.15.52:0/1340254732 learned_addr learned my addr 172.21.15.52:0/1340254732 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd49e00700 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 -- 172.21.15.52:0/1340254732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 msgr2=0x559f4e6eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 -- 172.21.15.52:0/1340254732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 msgr2=0x559f4e6e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd4a800700 1 -- 172.21.15.52:0/1340254732 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4e6e04f0 con 0x559f4e6e3560 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.527+0000 7fdd49e00700 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.528+0000 7fdd4a800700 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009cb0 tx=0x7fdd44009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ef687c87ad952a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.528+0000 7fdd36a00700 1 -- 172.21.15.52:0/1340254732 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4401a020 con 0x559f4e6e3560 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.528+0000 7fdd36a00700 1 -- 172.21.15.52:0/1340254732 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd44016400 con 0x559f4e6e3560 2024-04-26T01:57:13.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.528+0000 7fdd36a00700 1 -- 172.21.15.52:0/1340254732 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd440155c0 con 0x559f4e6e3560 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1340254732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 msgr2=0x559f4e6eb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009cb0 tx=0x7fdd44009040 comp rx=0 tx=0).stop 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1340254732 shutdown_connections 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6e3560 0x559f4e6eb0d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/1340254732 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1340254732 >> 172.21.15.52:0/1340254732 conn(0x559f4e6070e0 msgr2=0x559f4e6e1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.529+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1340254732 shutdown_connections 2024-04-26T01:57:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.530+0000 7fdd5b827b80 1 -- 172.21.15.52:0/1340254732 wait complete. 2024-04-26T01:57:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.530+0000 7fdd5b827b80 1 -- start start 2024-04-26T01:57:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.530+0000 7fdd5b827b80 1 Processor -- start 2024-04-26T01:57:13.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6ea480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6e3560 0x559f4e758d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1ad980 con 0x559f4e6e3560 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1aedf0 con 0x559f4e6d86f0 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c1ae050 con 0x559f4e6d7dd0 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd5b827b80 10 client.0 _ll_register_callbacks cb 0x559f4e5c38b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:13.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd49e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53588/0 (socket says 172.21.15.52:53588) 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.531+0000 7fdd49e00700 1 -- 172.21.15.52:0/86381872 learned_addr learned my addr 172.21.15.52:0/86381872 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd4a800700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6e3560 0x559f4e758d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6ea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 -- 172.21.15.52:0/86381872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 msgr2=0x559f4e6e9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 -- 172.21.15.52:0/86381872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6e3560 msgr2=0x559f4e758d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6e3560 0x559f4e758d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.532+0000 7fdd49400700 1 -- 172.21.15.52:0/86381872 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4e7592c0 con 0x559f4e6d86f0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.533+0000 7fdd49400700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6ea480 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x559f4e6f5460 tx=0x7fdd380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2abbc9d0728eb6e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.533+0000 7fdd2be00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.533+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38032070 con 0x559f4e6d86f0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.533+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fdd3802b320 con 0x559f4e6d86f0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.533+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd44016400 con 0x559f4e6d86f0 2024-04-26T01:57:13.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.534+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdd3803c050 con 0x559f4e6d86f0 2024-04-26T01:57:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.534+0000 7fdd5b827b80 10 client.26570 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.534+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559f4e75a4f0 con 0x559f4e6d86f0 2024-04-26T01:57:13.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd44016400 con 0x559f4e6d86f0 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fdd38037b50 con 0x559f4e6d86f0 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 10 client.26570 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 10 client.26570 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 20 client.26570 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e75acd0 0x559f4e75d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:13.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 4 client.26570 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}] 2024-04-26T01:57:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.535+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}) -- 0x559f4e6328b0 con 0x559f4e75acd0 2024-04-26T01:57:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.536+0000 7fdd49e00700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e75acd0 0x559f4e75d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.537+0000 7fdd49e00700 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e75acd0 0x559f4e75d0f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad42012ddbb1a035 server_cookie=30139ec26fea2056 in_seq=0 out_seq=0 2024-04-26T01:57:13.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.537+0000 7fdd2be00700 10 client.26570 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:13.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.615+0000 7fdd2be00700 1 -- 172.21.15.52:0/86381872 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559f4e6328b0 con 0x559f4e75acd0 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.615+0000 7fdd2be00700 10 client.26570 handle_command_reply: tid=0 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.615+0000 7fdd5b827b80 1 client.26570 shutdown 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 msgr2=0x559f4e6ea480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6ea480 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x559f4e6f5460 tx=0x7fdd380047c0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 shutdown_connections 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559f4e75acd0 0x559f4e75d0f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559f4e6e3560 0x559f4e758d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559f4e6d7dd0 0x559f4e6e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 --2- 172.21.15.52:0/86381872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559f4e6d86f0 0x559f4e6ea480 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd2be00700 0 client.26570 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 >> 172.21.15.52:0/86381872 conn(0x559f4e6070e0 msgr2=0x559f4e6e2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 shutdown_connections 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 1 -- 172.21.15.52:0/86381872 wait complete. 2024-04-26T01:57:13.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.616+0000 7fdd5b827b80 20 client.26570 trim_cache size 0 max 16384 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 49, 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 34, 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "c7b75781": { 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 34, 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:57:13.616 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 79.2, 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:13.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 msgr2=0x7fdd300a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 0x7fdd300a6410 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009680 tx=0x7fdd38009260 comp rx=0 tx=0).stop 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 msgr2=0x7fdd4c19b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000c00 tx=0x7fdd3c0057c0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 shutdown_connections 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd300a3f50 0x7fdd300a6410 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd4c10f4a0 0x7fdd4c19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c074c70 0x7fdd4c19b230 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 --2- 172.21.15.52:0/3006632425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd4c075590 0x7fdd4c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 >> 172.21.15.52:0/3006632425 conn(0x7fdd4c079ba0 msgr2=0x7fdd4c0ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 shutdown_connections 2024-04-26T01:57:13.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:13.617+0000 7fdd50e00700 1 -- 172.21.15.52:0/3006632425 wait complete. 2024-04-26T01:57:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:13 smithi191 ceph-mon[32745]: pgmap v383: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 388 KiB/s wr, 89 op/s 2024-04-26T01:57:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:13 smithi191 ceph-mon[32745]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:57:13.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:13 smithi183 ceph-mon[32752]: pgmap v383: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 388 KiB/s wr, 89 op/s 2024-04-26T01:57:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:13 smithi183 ceph-mon[32752]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T01:57:14.142 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.27 MB/sec execute 513 sec latency 592.512 ms 2024-04-26T01:57:14.406 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c7b75781', quiesce took 8.0 seconds. Will release after: 19.25 2024-04-26T01:57:15.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.27 MB/sec execute 514 sec latency 1592.627 ms 2024-04-26T01:57:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:15 smithi191 ceph-mon[32745]: pgmap v385: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 105 KiB/s wr, 29 op/s 2024-04-26T01:57:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:15 smithi183 ceph-mon[32752]: pgmap v385: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 105 KiB/s wr, 29 op/s 2024-04-26T01:57:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:15 smithi052 ceph-mon[27851]: pgmap v385: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 105 KiB/s wr, 29 op/s 2024-04-26T01:57:16.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.26 MB/sec execute 515 sec latency 2592.763 ms 2024-04-26T01:57:17.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.26 MB/sec execute 516 sec latency 3592.935 ms 2024-04-26T01:57:17.800 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:17 smithi052 ceph-mon[27851]: pgmap v386: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 399 KiB/s wr, 44 op/s 2024-04-26T01:57:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:17 smithi191 ceph-mon[32745]: pgmap v386: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 399 KiB/s wr, 44 op/s 2024-04-26T01:57:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:17 smithi183 ceph-mon[32752]: pgmap v386: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 399 KiB/s wr, 44 op/s 2024-04-26T01:57:18.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.26 MB/sec execute 517 sec latency 4593.064 ms 2024-04-26T01:57:19.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.25 MB/sec execute 518 sec latency 5593.193 ms 2024-04-26T01:57:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:19 smithi052 ceph-mon[27851]: pgmap v387: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:19 smithi191 ceph-mon[32745]: pgmap v387: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:19 smithi183 ceph-mon[32752]: pgmap v387: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 -- 172.21.15.52:0/4236279709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00104d50 msgr2=0x7f6e00108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 --2- 172.21.15.52:0/4236279709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00104d50 0x7f6e00108d90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009ca0 tx=0x7f6de802a8c0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 -- 172.21.15.52:0/4236279709 shutdown_connections 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 --2- 172.21.15.52:0/4236279709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e00109470 0x7f6e00111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 --2- 172.21.15.52:0/4236279709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00104d50 0x7f6e00108d90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 --2- 172.21.15.52:0/4236279709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e001043a0 0x7f6e00104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 -- 172.21.15.52:0/4236279709 >> 172.21.15.52:0/4236279709 conn(0x7f6e000ffc40 msgr2=0x7f6e00102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 -- 172.21.15.52:0/4236279709 shutdown_connections 2024-04-26T01:57:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.121+0000 7f6e05a00640 1 -- 172.21.15.52:0/4236279709 wait complete. 2024-04-26T01:57:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 Processor -- start 2024-04-26T01:57:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 -- start start 2024-04-26T01:57:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00109470 0x7f6e001a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001143b0 con 0x7f6e0019a730 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6e00114230 con 0x7f6e00109470 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.123+0000 7f6e05a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6e00114530 con 0x7f6e001043a0 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dff400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dff400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44162/0 (socket says 172.21.15.52:44162) 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34970/0 (socket says 172.21.15.52:34970) 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dff400640 1 -- 172.21.15.52:0/773014092 learned_addr learned my addr 172.21.15.52:0/773014092 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00109470 0x7f6e001a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 -- 172.21.15.52:0/773014092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00109470 msgr2=0x7f6e001a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00109470 0x7f6e001a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 -- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 msgr2=0x7f6e0019abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dfea00640 1 -- 172.21.15.52:0/773014092 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0019b330 con 0x7f6e001043a0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.124+0000 7f6dff400640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6dfea00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6df000e9f0 tx=0x7f6df000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6df7400640 1 -- 172.21.15.52:0/773014092 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df000ccc0 con 0x7f6e001043a0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e001a73c0 con 0x7f6e001043a0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6df7400640 1 -- 172.21.15.52:0/773014092 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df00108e0 con 0x7f6e001043a0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6df7400640 1 -- 172.21.15.52:0/773014092 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df0018880 con 0x7f6e001043a0 2024-04-26T01:57:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.125+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e001a77c0 con 0x7f6e001043a0 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.128+0000 7f6e08829740 1 Processor -- start 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.128+0000 7f6e08829740 1 -- start start 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6df7400640 1 -- 172.21.15.52:0/773014092 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6df0010430 con 0x7f6e001043a0 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143ab94c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab95140 0x56143ab9f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56143aafa900 con 0x56143ab9fa00 2024-04-26T01:57:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56143aaa0ef0 con 0x56143ab94820 2024-04-26T01:57:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6e08829740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56143aaf68c0 con 0x56143ab95140 2024-04-26T01:57:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab95140 0x56143ab9f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44166/0 (socket says 172.21.15.52:44166) 2024-04-26T01:57:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6dfe000640 1 -- 172.21.15.52:0/1324520881 learned_addr learned my addr 172.21.15.52:0/1324520881 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.129+0000 7f6dff400640 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143ab94c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 -- 172.21.15.52:0/1324520881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab95140 msgr2=0x56143ab9f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab95140 0x56143ab9f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 -- 172.21.15.52:0/1324520881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 msgr2=0x56143ab94c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143ab94c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 -- 172.21.15.52:0/1324520881 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56143a80d210 con 0x56143ab9fa00 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009ca0 tx=0x7f6de80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b02d821636d5432 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.130+0000 7f6df7400640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 0x7f6de00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6defe00640 1 -- 172.21.15.52:0/1324520881 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8031cf0 con 0x56143ab9fa00 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6df7400640 1 -- 172.21.15.52:0/773014092 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f6df00d0af0 con 0x7f6e001043a0 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6dff400640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 0x7f6de00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6defe00640 1 -- 172.21.15.52:0/1324520881 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de8036030 con 0x56143ab9fa00 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6defe00640 1 -- 172.21.15.52:0/1324520881 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8004240 con 0x56143ab9fa00 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6e08829740 1 -- 172.21.15.52:0/1324520881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 msgr2=0x56143aba1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6dff400640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 0x7f6de00a6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009640 tx=0x7f6df8005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.131+0000 7f6e08829740 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009ca0 tx=0x7f6de80093f0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 -- 172.21.15.52:0/1324520881 shutdown_connections 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab9fa00 0x56143aba1df0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab95140 0x56143ab9f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 --2- 172.21.15.52:0/1324520881 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143ab94c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 -- 172.21.15.52:0/1324520881 >> 172.21.15.52:0/1324520881 conn(0x56143ab90d30 msgr2=0x56143ab91110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 -- 172.21.15.52:0/1324520881 shutdown_connections 2024-04-26T01:57:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 -- 172.21.15.52:0/1324520881 wait complete. 2024-04-26T01:57:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 -- start start 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.132+0000 7f6e08829740 1 Processor -- start 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab94820 0x56143ab9c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ab9d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56143aba2450 con 0x56143ab94820 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56143aab4310 con 0x56143ab95140 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56143aba25d0 con 0x56143ab9fa00 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6e08829740 10 client.0 _ll_register_callbacks cb 0x56143aa82cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab94820 0x56143ab9c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35532/0 (socket says 172.21.15.52:35532) 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6dfe000640 1 -- 172.21.15.52:0/1043679305 learned_addr learned my addr 172.21.15.52:0/1043679305 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.133+0000 7f6dff400640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ab9d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 -- 172.21.15.52:0/1043679305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 msgr2=0x56143ab9d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ab9d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 -- 172.21.15.52:0/1043679305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab94820 msgr2=0x56143ab9c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab94820 0x56143ab9c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 -- 172.21.15.52:0/1043679305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56143a8d9620 con 0x56143ab95140 2024-04-26T01:57:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6dfe000640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009620 tx=0x7f6de8003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e8dde716dd6c24d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6decc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8052020 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f6de802f660 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.134+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de8036030 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f6de805d050 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 10 client.26582 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56143aba2450 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de8036030 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f6de804d030 con 0x56143ab95140 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 10 client.26582 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 10 client.26582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 20 client.26582 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143abaf850 0x56143abb1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 4 client.26582 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.135+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56143ab0e8b0 con 0x56143abaf850 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.136+0000 7f6dfea00640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143abaf850 0x56143abb1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.136+0000 7f6dfea00640 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143abaf850 0x56143abb1cc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbcb41164b1221d3 server_cookie=e16121bc5a5aada9 in_seq=0 out_seq=0 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.136+0000 7f6decc00640 10 client.26582 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.137+0000 7f6decc00640 1 -- 172.21.15.52:0/1043679305 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56143ab0e8b0 con 0x56143abaf850 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.137+0000 7f6decc00640 10 client.26582 handle_command_reply: tid=0 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.137+0000 7f6e08829740 1 client.26582 shutdown 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 msgr2=0x56143ab9ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009620 tx=0x7f6de8003c00 comp rx=0 tx=0).stop 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 shutdown_connections 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143abaf850 0x56143abb1cc0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ab9d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab95140 0x56143ab9ce50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 --2- 172.21.15.52:0/1043679305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab94820 0x56143ab9c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6decc00640 0 client.26582 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 >> 172.21.15.52:0/1043679305 conn(0x56143ab90d30 msgr2=0x56143ab96a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 shutdown_connections 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 1 -- 172.21.15.52:0/1043679305 wait complete. 2024-04-26T01:57:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.138+0000 7f6e08829740 20 client.26582 trim_cache size 0 max 16384 2024-04-26T01:57:20.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.25 MB/sec execute 519 sec latency 6593.281 ms 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.154+0000 7f6e08829740 1 Processor -- start 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.154+0000 7f6e08829740 1 -- start start 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 0x56143abab310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 0x56143abb75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001b2560 con 0x56143ab95140 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001afda0 con 0x56143ab9fa00 2024-04-26T01:57:20.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6e08829740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001adc10 con 0x56143ab94820 2024-04-26T01:57:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44180/0 (socket says 172.21.15.52:44180) 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6dfe000640 1 -- 172.21.15.52:0/2552785798 learned_addr learned my addr 172.21.15.52:0/2552785798 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 0x56143abab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.155+0000 7f6dff400640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 0x56143abb75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 -- 172.21.15.52:0/2552785798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 msgr2=0x56143abb75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 0x56143abb75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 -- 172.21.15.52:0/2552785798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 msgr2=0x56143ab9cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 -- 172.21.15.52:0/2552785798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56143ab958b0 con 0x56143ab94820 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dff400640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 0x56143abb75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfe000640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 0x56143abab310 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6df00ce1e0 tx=0x7f6df00d2f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f8f271b607d60c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6def400640 1 -- 172.21.15.52:0/2552785798 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df009c440 con 0x56143ab94820 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6def400640 1 -- 172.21.15.52:0/2552785798 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6df0010720 con 0x56143ab94820 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.156+0000 7f6def400640 1 -- 172.21.15.52:0/2552785798 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df00bd0b0 con 0x56143ab94820 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- 172.21.15.52:0/2552785798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 msgr2=0x56143abab310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 0x56143abab310 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6df00ce1e0 tx=0x7f6df00d2f40 comp rx=0 tx=0).stop 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- 172.21.15.52:0/2552785798 shutdown_connections 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab9fa00 0x56143abb75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143ab9cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 --2- 172.21.15.52:0/2552785798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab94820 0x56143abab310 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- 172.21.15.52:0/2552785798 >> 172.21.15.52:0/2552785798 conn(0x56143ab90d30 msgr2=0x56143ab9c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- 172.21.15.52:0/2552785798 shutdown_connections 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- 172.21.15.52:0/2552785798 wait complete. 2024-04-26T01:57:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.157+0000 7f6e08829740 1 -- start start 2024-04-26T01:57:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 Processor -- start 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143aba5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ac14e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001ae260 con 0x56143ab95140 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001af6d0 con 0x56143ab94820 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6e001ae930 con 0x56143ab9fa00 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6e08829740 10 client.0 _ll_register_callbacks cb 0x7f6de8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.158+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfe000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44184/0 (socket says 172.21.15.52:44184) 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfe000640 1 -- 172.21.15.52:0/2417806833 learned_addr learned my addr 172.21.15.52:0/2417806833 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143aba5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dff400640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ac14e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 -- 172.21.15.52:0/2417806833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 msgr2=0x56143ac14e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ac14e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 -- 172.21.15.52:0/2417806833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 msgr2=0x56143aba5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 -- 172.21.15.52:0/2417806833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56143aba6340 con 0x56143ab94820 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfe000640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6dfea00640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143aba5570 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6df00a0fa0 tx=0x7f6df000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1fe6c47e52117c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6decc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df009cae0 con 0x56143ab94820 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f6df009ade0 con 0x56143ab94820 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.159+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56143ab08320 con 0x56143ab94820 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f6df00a0de0 con 0x56143ab94820 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 10 client.26588 fetch_fsmap learned FSMap version 49 2024-04-26T01:57:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56143ac16470 con 0x56143ab94820 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56143ab08320 con 0x56143ab94820 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f6df00ba070 con 0x56143ab94820 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 10 client.26588 fetch_fsmap finished waiting for FSMap version 49 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 10 client.26588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 20 client.26588 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143ac16bb0 0x56143ac18fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 4 client.26588 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.160+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56143ab02190 con 0x56143ac16bb0 2024-04-26T01:57:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.161+0000 7f6dfe000640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143ac16bb0 0x56143ac18fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.161+0000 7f6dfe000640 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143ac16bb0 0x56143ac18fd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c075e0d3f432b57 server_cookie=a9a26189a0dc4970 in_seq=0 out_seq=0 2024-04-26T01:57:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.161+0000 7f6decc00640 10 client.26588 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6decc00640 1 -- 172.21.15.52:0/2417806833 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56143ab02190 con 0x56143ac16bb0 2024-04-26T01:57:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6decc00640 10 client.26588 handle_command_reply: tid=0 2024-04-26T01:57:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 client.26588 shutdown 2024-04-26T01:57:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 msgr2=0x56143aba5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143aba5570 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6df00a0fa0 tx=0x7f6df000cef0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 shutdown_connections 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56143ac16bb0 0x56143ac18fd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56143ab9fa00 0x56143ac14e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56143ab95140 0x56143aba5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 --2- 172.21.15.52:0/2417806833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56143ab94820 0x56143aba5570 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6decc00640 0 client.26588 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.192+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 >> 172.21.15.52:0/2417806833 conn(0x56143ab90d30 msgr2=0x56143ab9c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.193+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 shutdown_connections 2024-04-26T01:57:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.193+0000 7f6e08829740 1 -- 172.21.15.52:0/2417806833 wait complete. 2024-04-26T01:57:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.193+0000 7f6e08829740 20 client.26588 trim_cache size 0 max 16384 2024-04-26T01:57:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 msgr2=0x7f6de00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 0x7f6de00a6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009640 tx=0x7f6df8005f90 comp rx=0 tx=0).stop 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 msgr2=0x7f6e001a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6df000e9f0 tx=0x7f6df000eeb0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 shutdown_connections 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6de00a3e40 0x7f6de00a6300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e0019a730 0x7f6e0019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e00109470 0x7f6e001a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 --2- 172.21.15.52:0/773014092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e001043a0 0x7f6e001a0620 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 >> 172.21.15.52:0/773014092 conn(0x7f6e000ffc40 msgr2=0x7f6e00101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 shutdown_connections 2024-04-26T01:57:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:20.194+0000 7f6e05a00640 1 -- 172.21.15.52:0/773014092 wait complete. 2024-04-26T01:57:21.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.24 MB/sec execute 520 sec latency 7593.353 ms 2024-04-26T01:57:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:21 smithi191 ceph-mon[32745]: pgmap v388: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:21 smithi183 ceph-mon[32752]: pgmap v388: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:21 smithi052 ceph-mon[27851]: pgmap v388: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 386 KiB/s wr, 43 op/s 2024-04-26T01:57:22.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.24 MB/sec execute 521 sec latency 8593.515 ms 2024-04-26T01:57:23.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.23 MB/sec execute 522 sec latency 9593.659 ms 2024-04-26T01:57:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:23 smithi191 ceph-mon[32745]: pgmap v389: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 353 KiB/s wr, 35 op/s 2024-04-26T01:57:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:23 smithi183 ceph-mon[32752]: pgmap v389: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 353 KiB/s wr, 35 op/s 2024-04-26T01:57:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:23 smithi052 ceph-mon[27851]: pgmap v389: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 353 KiB/s wr, 35 op/s 2024-04-26T01:57:24.143 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.23 MB/sec execute 523 sec latency 10593.756 ms 2024-04-26T01:57:25.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.23 MB/sec execute 524 sec latency 11593.901 ms 2024-04-26T01:57:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:25 smithi191 ceph-mon[32745]: pgmap v390: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 324 KiB/s wr, 32 op/s 2024-04-26T01:57:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:25 smithi183 ceph-mon[32752]: pgmap v390: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 324 KiB/s wr, 32 op/s 2024-04-26T01:57:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:25 smithi052 ceph-mon[27851]: pgmap v390: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 324 KiB/s wr, 32 op/s 2024-04-26T01:57:26.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.22 MB/sec execute 525 sec latency 12594.041 ms 2024-04-26T01:57:27.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.22 MB/sec execute 526 sec latency 13594.177 ms 2024-04-26T01:57:27.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:27 smithi191 ceph-mon[32745]: pgmap v391: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 324 KiB/s wr, 33 op/s 2024-04-26T01:57:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:27 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:27 smithi183 ceph-mon[32752]: pgmap v391: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 324 KiB/s wr, 33 op/s 2024-04-26T01:57:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:27 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:27 smithi052 ceph-mon[27851]: pgmap v391: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 324 KiB/s wr, 33 op/s 2024-04-26T01:57:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:27 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:28.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.21 MB/sec execute 527 sec latency 14594.311 ms 2024-04-26T01:57:29.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.21 MB/sec execute 528 sec latency 15594.469 ms 2024-04-26T01:57:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:29 smithi191 ceph-mon[32745]: pgmap v392: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:29 smithi183 ceph-mon[32752]: pgmap v392: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:29 smithi052 ceph-mon[27851]: pgmap v392: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:30.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.20 MB/sec execute 529 sec latency 16594.611 ms 2024-04-26T01:57:31.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.20 MB/sec execute 530 sec latency 17594.749 ms 2024-04-26T01:57:31.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:31 smithi191 ceph-mon[32745]: pgmap v393: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:31.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:31 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:31 smithi183 ceph-mon[32752]: pgmap v393: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:31 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:31 smithi052 ceph-mon[27851]: pgmap v393: 97 pgs: 97 active+clean; 104 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-26T01:57:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:31 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:57:32.144 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.20 MB/sec execute 531 sec latency 18594.839 ms 2024-04-26T01:57:32.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:32 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:32.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:32 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:32.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:32 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:57:33.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.19 MB/sec execute 532 sec latency 19594.971 ms 2024-04-26T01:57:33.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 -- 172.21.15.52:0/1208169332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a81000d0 msgr2=0x7fd8a81004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/1208169332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a81000d0 0x7fd8a81004b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd898009930 tx=0x7fd89802a420 comp rx=0 tx=0).stop 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 -- 172.21.15.52:0/1208169332 shutdown_connections 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/1208169332 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a810a5e0 0x7fd8a8110f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/1208169332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a8100a80 0x7fd8a8109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/1208169332 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a81000d0 0x7fd8a81004b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 -- 172.21.15.52:0/1208169332 >> 172.21.15.52:0/1208169332 conn(0x7fd8a8076db0 msgr2=0x7fd8a80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 -- 172.21.15.52:0/1208169332 shutdown_connections 2024-04-26T01:57:33.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.351+0000 7fd8a7e00640 1 -- 172.21.15.52:0/1208169332 wait complete. 2024-04-26T01:57:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.352+0000 7fd8a7e00640 1 Processor -- start 2024-04-26T01:57:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.352+0000 7fd8a7e00640 1 -- start start 2024-04-26T01:57:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.352+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a810a5e0 0x7fd8a81a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a819a620 0x7fd8a819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8114280 con 0x7fd8a810a5e0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8114100 con 0x7fd8a81000d0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8114400 con 0x7fd8a819a620 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35466/0 (socket says 172.21.15.52:35466) 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 -- 172.21.15.52:0/465445088 learned_addr learned my addr 172.21.15.52:0/465445088 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd89fe00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a810a5e0 0x7fd8a81a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a7400640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a819a620 0x7fd8a819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 -- 172.21.15.52:0/465445088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a819a620 msgr2=0x7fd8a819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a819a620 0x7fd8a819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 -- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a810a5e0 msgr2=0x7fd8a81a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a810a5e0 0x7fd8a81a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.353+0000 7fd8a6a00640 1 -- 172.21.15.52:0/465445088 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a819b140 con 0x7fd8a81000d0 2024-04-26T01:57:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd898005ab0 tx=0x7fd898003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd89f400640 1 -- 172.21.15.52:0/465445088 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89803f070 con 0x7fd8a81000d0 2024-04-26T01:57:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd89f400640 1 -- 172.21.15.52:0/465445088 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd898031b50 con 0x7fd8a81000d0 2024-04-26T01:57:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a819b3d0 con 0x7fd8a81000d0 2024-04-26T01:57:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd89f400640 1 -- 172.21.15.52:0/465445088 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898035030 con 0x7fd8a81000d0 2024-04-26T01:57:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.354+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a81a75d0 con 0x7fd8a81000d0 2024-04-26T01:57:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.357+0000 7fd89f400640 1 -- 172.21.15.52:0/465445088 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd898042070 con 0x7fd8a81000d0 2024-04-26T01:57:33.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.357+0000 7fd8b07b6740 1 Processor -- start 2024-04-26T01:57:33.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 -- start start 2024-04-26T01:57:33.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be96369b0 0x563be9636d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be9641650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 0x563be9643f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563be959c900 con 0x563be96372d0 2024-04-26T01:57:33.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563be9542ef0 con 0x563be9641b90 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563be95988c0 con 0x563be96369b0 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.358+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be9641650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd8a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be96369b0 0x563be9636d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be9641650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49820/0 (socket says 172.21.15.52:49820) 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 0x563be9643f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd8a6a00640 1 -- 172.21.15.52:0/1423583356 learned_addr learned my addr 172.21.15.52:0/1423583356 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89f400640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 0x7fd88c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 -- 172.21.15.52:0/1423583356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be96369b0 msgr2=0x563be9636d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be96369b0 0x563be9636d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 -- 172.21.15.52:0/1423583356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 msgr2=0x563be9641650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be9641650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 -- 172.21.15.52:0/1423583356 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563be92af210 con 0x563be9641b90 2024-04-26T01:57:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89f400640 1 -- 172.21.15.52:0/465445088 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fd8980f0e50 con 0x7fd8a81000d0 2024-04-26T01:57:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd8a7400640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 0x7fd88c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.359+0000 7fd89fe00640 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 0x563be9643f80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd8a81034b0 tx=0x7fd890011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a7bfb15196461b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd897e00640 1 -- 172.21.15.52:0/1423583356 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890004050 con 0x563be9641b90 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd897e00640 1 -- 172.21.15.52:0/1423583356 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8900054d0 con 0x563be9641b90 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd897e00640 1 -- 172.21.15.52:0/1423583356 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890004ca0 con 0x563be9641b90 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 -- 172.21.15.52:0/1423583356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 msgr2=0x563be9643f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8a7400640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 0x7fd88c0a6250 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005fd0 tx=0x7fd8a0004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 0x563be9643f80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd8a81034b0 tx=0x7fd890011810 comp rx=0 tx=0).stop 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 -- 172.21.15.52:0/1423583356 shutdown_connections 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be9641b90 0x563be9643f80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be9641650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/1423583356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be96369b0 0x563be9636d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.360+0000 7fd8b07b6740 1 -- 172.21.15.52:0/1423583356 >> 172.21.15.52:0/1423583356 conn(0x563be9632ec0 msgr2=0x563be96332a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.361+0000 7fd8b07b6740 1 -- 172.21.15.52:0/1423583356 shutdown_connections 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.361+0000 7fd8b07b6740 1 -- 172.21.15.52:0/1423583356 wait complete. 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.361+0000 7fd8b07b6740 1 -- start start 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.361+0000 7fd8b07b6740 1 Processor -- start 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be963f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be963f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563be96447c0 con 0x563be96372d0 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563be9556310 con 0x563be96369b0 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563be9644940 con 0x563be9641b90 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8b07b6740 10 client.0 _ll_register_callbacks cb 0x563be9524cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be963f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35476/0 (socket says 172.21.15.52:35476) 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8a6a00640 1 -- 172.21.15.52:0/2298254495 learned_addr learned my addr 172.21.15.52:0/2298254495 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd8a7400640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be963f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd89fe00640 1 -- 172.21.15.52:0/2298254495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 msgr2=0x563be963f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd89fe00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be963f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd89fe00640 1 -- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 msgr2=0x563be963eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.362+0000 7fd89fe00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd89fe00640 1 -- 172.21.15.52:0/2298254495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563be937b620 con 0x563be96372d0 2024-04-26T01:57:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd89fe00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be963f090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd89001b040 tx=0x7fd8900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84de67d8376fe045 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd894c00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890028070 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8900054d0 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.363+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fd8900037f0 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd890034050 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 10 client.16467 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x563be9650a50 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8900054d0 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fd890004c40 con 0x563be96372d0 2024-04-26T01:57:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 10 client.16467 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 10 client.16467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 20 client.16467 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be9651e60 0x563be9654250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 4 client.16467 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.364+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563be95b08b0 con 0x563be9651e60 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.365+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be9651e60 0x563be9654250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.365+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be9651e60 0x563be9654250 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa87d46accf8a265 server_cookie=14a7578a1a2b42ba in_seq=0 out_seq=0 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.365+0000 7fd894c00640 10 client.16467 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.366+0000 7fd894c00640 1 -- 172.21.15.52:0/2298254495 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563be95b08b0 con 0x563be9651e60 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.366+0000 7fd894c00640 10 client.16467 handle_command_reply: tid=0 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.366+0000 7fd8b07b6740 1 client.16467 shutdown 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.366+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 msgr2=0x563be963f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.366+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be963f090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd89001b040 tx=0x7fd8900044c0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 shutdown_connections 2024-04-26T01:57:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be9651e60 0x563be9654250 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be963f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96372d0 0x563be963f090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd894c00640 0 client.16467 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/2298254495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96369b0 0x563be963eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 >> 172.21.15.52:0/2298254495 conn(0x563be9632ec0 msgr2=0x563be9638c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 shutdown_connections 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 1 -- 172.21.15.52:0/2298254495 wait complete. 2024-04-26T01:57:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.367+0000 7fd8b07b6740 20 client.16467 trim_cache size 0 max 16384 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.383+0000 7fd8b07b6740 1 Processor -- start 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.383+0000 7fd8b07b6740 1 -- start start 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be964d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9640a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b24c0 con 0x563be96369b0 2024-04-26T01:57:33.383 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81afd60 con 0x563be96372d0 2024-04-26T01:57:33.383 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81adbd0 con 0x563be9641b90 2024-04-26T01:57:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be964d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35480/0 (socket says 172.21.15.52:35480) 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd89fe00640 1 -- 172.21.15.52:0/4061795952 learned_addr learned my addr 172.21.15.52:0/4061795952 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.384+0000 7fd8a7400640 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9640a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 -- 172.21.15.52:0/4061795952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 msgr2=0x563be9640a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9640a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 -- 172.21.15.52:0/4061795952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 msgr2=0x563be964d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be964d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 -- 172.21.15.52:0/4061795952 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563be96441b0 con 0x563be96372d0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd89fe00640 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd890004690 tx=0x7fd890011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c06ed2b555dedd32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd897400640 1 -- 172.21.15.52:0/4061795952 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890028070 con 0x563be96372d0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd897400640 1 -- 172.21.15.52:0/4061795952 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd890004e00 con 0x563be96372d0 2024-04-26T01:57:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.385+0000 7fd897400640 1 -- 172.21.15.52:0/4061795952 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89001d040 con 0x563be96372d0 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 -- 172.21.15.52:0/4061795952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 msgr2=0x563be96327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd890004690 tx=0x7fd890011810 comp rx=0 tx=0).stop 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 -- 172.21.15.52:0/4061795952 shutdown_connections 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9640a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be96327a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/4061795952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be964d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 -- 172.21.15.52:0/4061795952 >> 172.21.15.52:0/4061795952 conn(0x563be9632ec0 msgr2=0x563be9640380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 -- 172.21.15.52:0/4061795952 shutdown_connections 2024-04-26T01:57:33.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.386+0000 7fd8b07b6740 1 -- 172.21.15.52:0/4061795952 wait complete. 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 -- start start 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 Processor -- start 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be9649ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9647760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81ae220 con 0x563be96369b0 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81af690 con 0x563be96372d0 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81ae8f0 con 0x563be9641b90 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.387+0000 7fd8b07b6740 10 client.0 _ll_register_callbacks cb 0x7fd890009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be9649ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35492/0 (socket says 172.21.15.52:35492) 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 -- 172.21.15.52:0/3580632390 learned_addr learned my addr 172.21.15.52:0/3580632390 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd8a6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be9649ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49856/0 (socket says 172.21.15.52:49856) 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd8a7400640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9647760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 -- 172.21.15.52:0/3580632390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 msgr2=0x563be9647760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9647760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 -- 172.21.15.52:0/3580632390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 msgr2=0x563be9649ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be9649ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd89fe00640 1 -- 172.21.15.52:0/3580632390 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563be9647f90 con 0x563be96372d0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.388+0000 7fd8a7400640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9647760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd89fe00640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd890009b40 tx=0x7fd890004620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=688591811012c5bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd894c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89001d070 con 0x563be96372d0 2024-04-26T01:57:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fd8900046c0 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd890004e00 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd890028040 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd8b07b6740 10 client.26606 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x563be96b8130 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.389+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd890004e00 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fd89003f020 con 0x563be96372d0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 10 client.26606 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 10 client.26606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 20 client.26606 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be96b86f0 0x563be96bab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 4 client.26606 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563be95aa320 con 0x563be96b86f0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be96b86f0 0x563be96bab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.391+0000 7fd894c00640 10 client.26606 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.390+0000 7fd8a6a00640 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be96b86f0 0x563be96bab10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4947a1f8d6984d18 server_cookie=3311e200e9d28ef9 in_seq=0 out_seq=0 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd894c00640 1 -- 172.21.15.52:0/3580632390 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563be95aa320 con 0x563be96b86f0 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd894c00640 10 client.26606 handle_command_reply: tid=0 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 client.26606 shutdown 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 msgr2=0x563be9647220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd890009b40 tx=0x7fd890004620 comp rx=0 tx=0).stop 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 shutdown_connections 2024-04-26T01:57:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563be96b86f0 0x563be96bab10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563be9641b90 0x563be9647760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563be96372d0 0x563be9647220 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 --2- 172.21.15.52:0/3580632390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563be96369b0 0x563be9649ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd894c00640 0 client.26606 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.418+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 >> 172.21.15.52:0/3580632390 conn(0x563be9632ec0 msgr2=0x563be9640c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.419+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 shutdown_connections 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.419+0000 7fd8b07b6740 1 -- 172.21.15.52:0/3580632390 wait complete. 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.419+0000 7fd8b07b6740 20 client.26606 trim_cache size 0 max 16384 2024-04-26T01:57:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 msgr2=0x7fd88c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 0x7fd88c0a6250 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005fd0 tx=0x7fd8a0004a40 comp rx=0 tx=0).stop 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 msgr2=0x7fd8a81a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd898005ab0 tx=0x7fd898003b90 comp rx=0 tx=0).stop 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 shutdown_connections 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fd88c0a3d90 0x7fd88c0a6250 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd8a819a620 0x7fd8a819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a810a5e0 0x7fd8a81a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 --2- 172.21.15.52:0/465445088 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd8a81000d0 0x7fd8a81a0420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 >> 172.21.15.52:0/465445088 conn(0x7fd8a8076db0 msgr2=0x7fd8a80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 shutdown_connections 2024-04-26T01:57:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:33.420+0000 7fd8a7e00640 1 -- 172.21.15.52:0/465445088 wait complete. 2024-04-26T01:57:33.658 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:33 smithi191 ceph-mon[32745]: pgmap v394: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 135 KiB/s wr, 21 op/s 2024-04-26T01:57:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:33 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:33 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:33 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:57:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:33 smithi183 ceph-mon[32752]: pgmap v394: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 135 KiB/s wr, 21 op/s 2024-04-26T01:57:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:33 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:33 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:33 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:57:33.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:33 smithi052 ceph-mon[27851]: pgmap v394: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 135 KiB/s wr, 21 op/s 2024-04-26T01:57:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:33 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:57:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:33 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:57:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:33 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:57:34.143 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:34.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.19 MB/sec execute 533 sec latency 20595.063 ms 2024-04-26T01:57:35.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.18 MB/sec execute 534 sec latency 21595.145 ms 2024-04-26T01:57:35.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:35 smithi191 ceph-mon[32745]: pgmap v395: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-04-26T01:57:35.909 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:35 smithi052 ceph-mon[27851]: pgmap v395: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-04-26T01:57:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:35 smithi183 ceph-mon[32752]: pgmap v395: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-04-26T01:57:36.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.18 MB/sec execute 535 sec latency 22595.251 ms 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.446+0000 7f1bdea00700 1 -- 172.21.15.52:0/2206907646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 msgr2=0x7f1bd8111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.446+0000 7f1bdea00700 1 --2- 172.21.15.52:0/2206907646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd8111b90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009fc0 tx=0x7f1bd000b330 comp rx=0 tx=0).stop 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 -- 172.21.15.52:0/2206907646 shutdown_connections 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 --2- 172.21.15.52:0/2206907646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd8111b90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 --2- 172.21.15.52:0/2206907646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd81052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 --2- 172.21.15.52:0/2206907646 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 0x7f1bd81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 -- 172.21.15.52:0/2206907646 >> 172.21.15.52:0/2206907646 conn(0x7f1bd8100770 msgr2=0x7f1bd8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 -- 172.21.15.52:0/2206907646 shutdown_connections 2024-04-26T01:57:36.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.447+0000 7f1bdea00700 1 -- 172.21.15.52:0/2206907646 wait complete. 2024-04-26T01:57:36.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.448+0000 7f1bdea00700 1 Processor -- start 2024-04-26T01:57:36.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.448+0000 7f1bdea00700 1 -- start start 2024-04-26T01:57:36.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:36.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 0x7f1bd8077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:36.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd80779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:36.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8114ea0 con 0x7f1bd8104ee0 2024-04-26T01:57:36.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8114d20 con 0x7f1bd8105890 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdea00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8115020 con 0x7f1bd8109fc0 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bd7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bd7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55738/0 (socket says 172.21.15.52:55738) 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bd7e00700 1 -- 172.21.15.52:0/3670896164 learned_addr learned my addr 172.21.15.52:0/3670896164 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bdca00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd80779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.449+0000 7f1bd7400700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 0x7f1bd8077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 -- 172.21.15.52:0/3670896164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 msgr2=0x7f1bd8077490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 0x7f1bd8077490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 -- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 msgr2=0x7f1bd8076f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 -- 172.21.15.52:0/3670896164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd81aba20 con 0x7f1bd8109fc0 2024-04-26T01:57:36.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bd7e00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:36.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.450+0000 7f1bdca00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd80779d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0015000 tx=0x7f1bd000cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:36.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.451+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0021070 con 0x7f1bd8109fc0 2024-04-26T01:57:36.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.451+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd000ce90 con 0x7f1bd8109fc0 2024-04-26T01:57:36.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.451+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0005040 con 0x7f1bd8109fc0 2024-04-26T01:57:36.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.451+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd81abcb0 con 0x7f1bd8109fc0 2024-04-26T01:57:36.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.451+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd81ac110 con 0x7f1bd8109fc0 2024-04-26T01:57:36.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.452+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd81063c0 con 0x7f1bd8109fc0 2024-04-26T01:57:36.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.454+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1bd0024070 con 0x7f1bd8109fc0 2024-04-26T01:57:36.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.455+0000 7f1bd4c00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 0x7f1bbc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:36.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.456+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f1bd00d1ad0 con 0x7f1bd8109fc0 2024-04-26T01:57:36.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.456+0000 7f1bd7e00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 0x7f1bbc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:36.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.457+0000 7f1bd7e00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 0x7f1bbc0a62f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009750 tx=0x7f1bc8009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:36.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.458+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bd009e1a0 con 0x7f1bd8109fc0 2024-04-26T01:57:36.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.741+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bd802df80 con 0x7f1bd8109fc0 2024-04-26T01:57:36.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.743+0000 7f1bd4c00700 1 -- 172.21.15.52:0/3670896164 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11886 (secure 0 0 0) 0x7f1bd009daa0 con 0x7f1bd8109fc0 2024-04-26T01:57:36.742 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:57:36.742 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:57:30.955584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.745+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 msgr2=0x7f1bbc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.745+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 0x7f1bbc0a62f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009750 tx=0x7f1bc8009310 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 msgr2=0x7f1bd80779d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd80779d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0015000 tx=0x7f1bd000cc30 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 shutdown_connections 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1bbc0a3e30 0x7f1bbc0a62f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bd8109fc0 0x7f1bd80779d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1bd8104ee0 0x7f1bd8076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 --2- 172.21.15.52:0/3670896164 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1bd8105890 0x7f1bd8077490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 >> 172.21.15.52:0/3670896164 conn(0x7f1bd8100770 msgr2=0x7f1bd8102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 shutdown_connections 2024-04-26T01:57:36.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:36.746+0000 7f1bdea00700 1 -- 172.21.15.52:0/3670896164 wait complete. 2024-04-26T01:57:36.746 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 51 2024-04-26T01:57:36.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:36 smithi191 ceph-mon[32745]: pgmap v396: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 170 KiB/s wr, 24 op/s 2024-04-26T01:57:36.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:36 smithi183 ceph-mon[32752]: pgmap v396: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 170 KiB/s wr, 24 op/s 2024-04-26T01:57:36.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:36 smithi052 ceph-mon[27851]: pgmap v396: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 170 KiB/s wr, 24 op/s 2024-04-26T01:57:37.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.18 MB/sec execute 536 sec latency 23595.405 ms 2024-04-26T01:57:37.742 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id c7b75781 --release --await' 2024-04-26T01:57:37.742 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id c7b75781 --release --await 2024-04-26T01:57:37.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:37 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3670896164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:37 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3670896164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:37 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3670896164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:38.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.17 MB/sec execute 537 sec latency 24595.492 ms 2024-04-26T01:57:38.208 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:38.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:38 smithi191 ceph-mon[32745]: pgmap v397: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:38.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:38 smithi183 ceph-mon[32752]: pgmap v397: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:38.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:38 smithi052 ceph-mon[27851]: pgmap v397: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:39.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.17 MB/sec execute 538 sec latency 25595.570 ms 2024-04-26T01:57:40.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 269910 2.16 MB/sec execute 539 sec latency 26595.669 ms 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 -- 172.21.15.52:0/567919975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 msgr2=0x7f10dc108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 --2- 172.21.15.52:0/567919975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc108e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009990 tx=0x7f10d0009ca0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 -- 172.21.15.52:0/567919975 shutdown_connections 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 --2- 172.21.15.52:0/567919975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 0x7f10dc10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 --2- 172.21.15.52:0/567919975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc108e10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 --2- 172.21.15.52:0/567919975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 -- 172.21.15.52:0/567919975 >> 172.21.15.52:0/567919975 conn(0x7f10dc0788f0 msgr2=0x7f10dc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 -- 172.21.15.52:0/567919975 shutdown_connections 2024-04-26T01:57:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.521+0000 7f10e3c00700 1 -- 172.21.15.52:0/567919975 wait complete. 2024-04-26T01:57:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.522+0000 7f10e3c00700 1 Processor -- start 2024-04-26T01:57:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.522+0000 7f10e3c00700 1 -- start start 2024-04-26T01:57:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 0x7f10dc19b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc112c00 con 0x7f10dc108a30 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc112a80 con 0x7f10dc06b0f0 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e3c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc112d80 con 0x7f10dc06a750 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e1200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e1200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44702/0 (socket says 172.21.15.52:44702) 2024-04-26T01:57:40.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e1200700 1 -- 172.21.15.52:0/337666650 learned_addr learned my addr 172.21.15.52:0/337666650 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10dbe00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 0x7f10dc19b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.523+0000 7f10e1c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 -- 172.21.15.52:0/337666650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 msgr2=0x7f10dc19b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 0x7f10dc19b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 -- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 msgr2=0x7f10dc19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 -- 172.21.15.52:0/337666650 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10dc1a0360 con 0x7f10dc06a750 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e1200700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10d9600700 1 -- 172.21.15.52:0/337666650 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0027070 con 0x7f10dc06a750 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10dc1a05f0 con 0x7f10dc06a750 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.524+0000 7f10d9600700 1 -- 172.21.15.52:0/337666650 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10d0022470 con 0x7f10dc06a750 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.525+0000 7f10d9600700 1 -- 172.21.15.52:0/337666650 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0017650 con 0x7f10dc06a750 2024-04-26T01:57:40.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.525+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10dc1a7ed0 con 0x7f10dc06a750 2024-04-26T01:57:40.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.528+0000 7f10d9600700 1 -- 172.21.15.52:0/337666650 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f10d002a070 con 0x7f10dc06a750 2024-04-26T01:57:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.529+0000 7f10d9600700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 0x7f10c40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.530+0000 7f10d9600700 1 -- 172.21.15.52:0/337666650 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f10d00d14d0 con 0x7f10dc06a750 2024-04-26T01:57:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.530+0000 7f10ee66ab80 1 Processor -- start 2024-04-26T01:57:40.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.530+0000 7f10dbe00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 0x7f10c40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.530+0000 7f10ee66ab80 1 -- start start 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.530+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 0x559317c14f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c178c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559317b5cf90 con 0x559317c09d40 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559317b03830 con 0x559317c154d0 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559317b58f50 con 0x559317c0a660 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10dbe00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 0x7f10c40a62f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10dc10fd40 tx=0x7f10c8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10e1200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 0x559317c14f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10e1200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55764/0 (socket says 172.21.15.52:55764) 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10e1200700 1 -- 172.21.15.52:0/2284674897 learned_addr learned my addr 172.21.15.52:0/2284674897 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.531+0000 7f10e1c00700 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c178c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 -- 172.21.15.52:0/2284674897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 msgr2=0x559317c14f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 0x559317c14f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 -- 172.21.15.52:0/2284674897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 msgr2=0x559317c178c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c178c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 -- 172.21.15.52:0/2284674897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559317af9780 con 0x559317c09d40 2024-04-26T01:57:40.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10dbe00700 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 0x559317c14f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10e1200700 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f10d00a5b50 tx=0x7f10d009d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e10950d8d125513 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10ce000700 1 -- 172.21.15.52:0/2284674897 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d009dba0 con 0x559317c09d40 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.532+0000 7f10ce000700 1 -- 172.21.15.52:0/2284674897 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10d009d4a0 con 0x559317c09d40 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ce000700 1 -- 172.21.15.52:0/2284674897 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d009d700 con 0x559317c09d40 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2284674897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 msgr2=0x559317c0a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f10d00a5b50 tx=0x7f10d009d340 comp rx=0 tx=0).stop 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2284674897 shutdown_connections 2024-04-26T01:57:40.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c178c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c0a120 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2284674897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c0a660 0x559317c14f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.533+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2284674897 >> 172.21.15.52:0/2284674897 conn(0x559317b39050 msgr2=0x559317c06250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.534+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2284674897 shutdown_connections 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.534+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2284674897 wait complete. 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.534+0000 7f10ee66ab80 1 -- start start 2024-04-26T01:57:40.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.535+0000 7f10ee66ab80 1 Processor -- start 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.535+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c14cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559317b5cf90 con 0x559317c09d40 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x559317b16960 con 0x559317c0a660 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559317c17e00 con 0x559317c154d0 2024-04-26T01:57:40.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10ee66ab80 10 client.0 _ll_register_callbacks cb 0x559317af5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:40.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10e1200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c14cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46520/0 (socket says 172.21.15.52:46520) 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10e1c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 -- 172.21.15.52:0/2723253444 learned_addr learned my addr 172.21.15.52:0/2723253444 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 -- 172.21.15.52:0/2723253444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 msgr2=0x559317c124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 -- 172.21.15.52:0/2723253444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 msgr2=0x559317c14cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c14cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.536+0000 7f10dbe00700 1 -- 172.21.15.52:0/2723253444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559317aaa9d0 con 0x559317c0a660 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10dbe00700 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f10c801d8f0 tx=0x7f10c80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b07377c017ce868 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8004020 con 0x559317c0a660 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10d009d4a0 con 0x559317c0a660 2024-04-26T01:57:40.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f10c80041c0 con 0x559317c0a660 2024-04-26T01:57:40.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f10c80043c0 con 0x559317c0a660 2024-04-26T01:57:40.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10ee66ab80 10 client.26624 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.537+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559317af3bf0 con 0x559317c0a660 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10d009d4a0 con 0x559317c0a660 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7f10c801cb60 con 0x559317c0a660 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 10 client.26624 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 10 client.26624 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 20 client.26624 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c24a50 0x559317c26ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 4 client.26624 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.538+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559317b6a780 con 0x559317c24a50 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.539+0000 7f10e1200700 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c24a50 0x559317c26ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.539+0000 7f10e1200700 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c24a50 0x559317c26ea0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5875e9bfea39fd server_cookie=5d3b89226a63b1b4 in_seq=0 out_seq=0 2024-04-26T01:57:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.539+0000 7f10c2a00700 10 client.26624 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10c2a00700 1 -- 172.21.15.52:0/2723253444 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559317b6a780 con 0x559317c24a50 2024-04-26T01:57:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10c2a00700 10 client.26624 handle_command_reply: tid=0 2024-04-26T01:57:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 client.26624 shutdown 2024-04-26T01:57:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 msgr2=0x559317c11fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f10c801d8f0 tx=0x7f10c80047c0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 shutdown_connections 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c24a50 0x559317c26ea0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c09d40 0x559317c14cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10c2a00700 0 client.26624 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/2723253444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c0a660 0x559317c11fb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 >> 172.21.15.52:0/2723253444 conn(0x559317b39050 msgr2=0x559317c05e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.540+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 shutdown_connections 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.541+0000 7f10ee66ab80 1 -- 172.21.15.52:0/2723253444 wait complete. 2024-04-26T01:57:40.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.541+0000 7f10ee66ab80 20 client.26624 trim_cache size 0 max 16384 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.562+0000 7f10ee66ab80 1 Processor -- start 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.562+0000 7f10ee66ab80 1 -- start start 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 0x559317c13e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c12640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1b1d50 con 0x559317c0a660 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1af5b0 con 0x559317c09d40 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1ad420 con 0x559317c154d0 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10e1c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c12640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55780/0 (socket says 172.21.15.52:55780) 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10dbe00700 1 -- 172.21.15.52:0/1943666008 learned_addr learned my addr 172.21.15.52:0/1943666008 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:40.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.563+0000 7f10e1200700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 0x559317c13e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 -- 172.21.15.52:0/1943666008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 msgr2=0x559317c12640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c12640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 -- 172.21.15.52:0/1943666008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 msgr2=0x559317c13e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 0x559317c13e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 -- 172.21.15.52:0/1943666008 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559317c06d20 con 0x559317c0a660 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10e1200700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 0x559317c13e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10e1c00700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c12640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.564+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f10c8012fd0 tx=0x7f10c80117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d4d00a41fcfcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.565+0000 7f10cd600700 1 -- 172.21.15.52:0/1943666008 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c803c070 con 0x559317c0a660 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.565+0000 7f10cd600700 1 -- 172.21.15.52:0/1943666008 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10c80045e0 con 0x559317c0a660 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.565+0000 7f10cd600700 1 -- 172.21.15.52:0/1943666008 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8004a90 con 0x559317c0a660 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1943666008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 msgr2=0x559317c063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f10c8012fd0 tx=0x7f10c80117d0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1943666008 shutdown_connections 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c154d0 0x559317c12640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c09d40 0x559317c13e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1943666008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c063a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1943666008 >> 172.21.15.52:0/1943666008 conn(0x559317b39050 msgr2=0x559317c05c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1943666008 shutdown_connections 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1943666008 wait complete. 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.566+0000 7f10ee66ab80 1 -- start start 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 Processor -- start 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 0x559317c1bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c1c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1ada70 con 0x559317c0a660 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1aeee0 con 0x559317c154d0 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1ae140 con 0x559317c09d40 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10ee66ab80 10 client.0 _ll_register_callbacks cb 0x559317af5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.567+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55786/0 (socket says 172.21.15.52:55786) 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10dbe00700 1 -- 172.21.15.52:0/1365583822 learned_addr learned my addr 172.21.15.52:0/1365583822 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 0x559317c1bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1c00700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c1c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 -- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 msgr2=0x559317c1c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c1c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 -- 172.21.15.52:0/1365583822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 msgr2=0x559317c1c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1200700 1 -- 172.21.15.52:0/1365583822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559317c0a370 con 0x559317c09d40 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:40.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.568+0000 7f10e1c00700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c1c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10e1200700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 0x559317c1bc00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f10d00a1a10 tx=0x7f10d0099080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=249e7bbfaf236cb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d009d4a0 con 0x559317c09d40 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f10d0022920 con 0x559317c09d40 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10c80045e0 con 0x559317c09d40 2024-04-26T01:57:40.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f10d009dba0 con 0x559317c09d40 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10ee66ab80 10 client.26635 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559317c0ace0 con 0x559317c09d40 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.569+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10c80045e0 con 0x559317c09d40 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7f10d00a5bf0 con 0x559317c09d40 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 10 client.26635 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 10 client.26635 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 20 client.26635 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c8c650 0x559317c8ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 4 client.26635 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c7b75781", "release": true, "await": true}] 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c7b75781", "release": true, "await": true}) -- 0x559317b64820 con 0x559317c8c650 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.570+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c8c650 0x559317c8ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.571+0000 7f10dbe00700 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c8c650 0x559317c8ea40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da49c6a6da06867 server_cookie=7d674c949c0e2af3 in_seq=0 out_seq=0 2024-04-26T01:57:40.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.571+0000 7f10c2a00700 10 client.26635 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:40.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.575+0000 7f10c2a00700 1 -- 172.21.15.52:0/1365583822 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559317b64820 con 0x559317c8c650 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.575+0000 7f10c2a00700 10 client.26635 handle_command_reply: tid=0 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.575+0000 7f10ee66ab80 1 client.26635 shutdown 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 msgr2=0x559317c1bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 0x559317c1bc00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f10d00a1a10 tx=0x7f10d0099080 comp rx=0 tx=0).stop 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 shutdown_connections 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x559317c8c650 0x559317c8ea40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x559317c154d0 0x559317c1c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10c2a00700 0 client.26635 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559317c09d40 0x559317c1bc00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 --2- 172.21.15.52:0/1365583822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559317c0a660 0x559317c1c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 >> 172.21.15.52:0/1365583822 conn(0x559317b39050 msgr2=0x559317c1fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.576+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 shutdown_connections 2024-04-26T01:57:40.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.577+0000 7f10ee66ab80 1 -- 172.21.15.52:0/1365583822 wait complete. 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.577+0000 7f10ee66ab80 20 client.26635 trim_cache size 0 max 16384 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 51, 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 36, 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "c7b75781": { 2024-04-26T01:57:40.577 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 36, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 79.2, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:40.578 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:40.579 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:57:40.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 msgr2=0x7f10c40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 0x7f10c40a62f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10dc10fd40 tx=0x7f10c8009260 comp rx=0 tx=0).stop 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 msgr2=0x7f10dc19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d00057c0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 shutdown_connections 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f10c40a3e30 0x7f10c40a62f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10dc108a30 0x7f10dc19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10dc06a750 0x7f10dc19b350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 --2- 172.21.15.52:0/337666650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10dc06b0f0 0x7f10dc19b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.578+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 >> 172.21.15.52:0/337666650 conn(0x7f10dc0788f0 msgr2=0x7f10dc0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.579+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 shutdown_connections 2024-04-26T01:57:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:40.579+0000 7f10e3c00700 1 -- 172.21.15.52:0/337666650 wait complete. 2024-04-26T01:57:41.145 INFO:tasks.workunit.client.0.smithi052.stdout: 1 270089 2.16 MB/sec execute 540 sec latency 27029.001 ms 2024-04-26T01:57:41.512 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:41 smithi052 ceph-mon[27851]: pgmap v398: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:41.516 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c7b75781', seconds elapsed: 35.1 2024-04-26T01:57:41.516 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.25 2024-04-26T01:57:41.517 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:57:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:41 smithi191 ceph-mon[32745]: pgmap v398: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:41.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:41 smithi183 ceph-mon[32752]: pgmap v398: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 139 KiB/s wr, 20 op/s 2024-04-26T01:57:42.010 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:42.146 INFO:tasks.workunit.client.0.smithi052.stdout: 1 271074 2.17 MB/sec execute 541 sec latency 23.342 ms 2024-04-26T01:57:43.146 INFO:tasks.workunit.client.0.smithi052.stdout: 1 271377 2.16 MB/sec execute 542 sec latency 625.595 ms 2024-04-26T01:57:43.557 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:43 smithi052 ceph-mon[27851]: pgmap v399: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 165 KiB/s wr, 26 op/s 2024-04-26T01:57:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:43 smithi191 ceph-mon[32745]: pgmap v399: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 165 KiB/s wr, 26 op/s 2024-04-26T01:57:43.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:43 smithi183 ceph-mon[32752]: pgmap v399: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 165 KiB/s wr, 26 op/s 2024-04-26T01:57:44.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 -- 172.21.15.52:0/329369701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec102a90 msgr2=0x7f6eec102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:44.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/329369701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec102a90 0x7f6eec102f10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400b0a0 tx=0x7f6ed400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 -- 172.21.15.52:0/329369701 shutdown_connections 2024-04-26T01:57:44.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/329369701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec103450 0x7f6eec10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/329369701 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec108a50 0x7f6eec108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/329369701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec102a90 0x7f6eec102f10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 -- 172.21.15.52:0/329369701 >> 172.21.15.52:0/329369701 conn(0x7f6eec079ba0 msgr2=0x7f6eec079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 -- 172.21.15.52:0/329369701 shutdown_connections 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.124+0000 7f6ef2a00700 1 -- 172.21.15.52:0/329369701 wait complete. 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 Processor -- start 2024-04-26T01:57:44.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 -- start start 2024-04-26T01:57:44.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec102a90 0x7f6eec199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:44.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:44.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec108a50 0x7f6eec19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:44.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec112c50 con 0x7f6eec102a90 2024-04-26T01:57:44.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec112ad0 con 0x7f6eec108a50 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.126+0000 7f6ef2a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec112dd0 con 0x7f6eec103450 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6eebe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec102a90 0x7f6eec199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44742/0 (socket says 172.21.15.52:44742) 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 -- 172.21.15.52:0/2642050999 learned_addr learned my addr 172.21.15.52:0/2642050999 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:44.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ef0a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec108a50 0x7f6eec19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 -- 172.21.15.52:0/2642050999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec108a50 msgr2=0x7f6eec19da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec108a50 0x7f6eec19da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 -- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec102a90 msgr2=0x7f6eec199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec102a90 0x7f6eec199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.127+0000 7f6ee3400700 1 -- 172.21.15.52:0/2642050999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec19e190 con 0x7f6eec103450 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.128+0000 7f6ee3400700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400d010 tx=0x7f6ed4004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:44.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.128+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4021070 con 0x7f6eec103450 2024-04-26T01:57:44.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.128+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed4007e30 con 0x7f6eec103450 2024-04-26T01:57:44.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.128+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eec19e420 con 0x7f6eec103450 2024-04-26T01:57:44.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.128+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eec074c70 con 0x7f6eec103450 2024-04-26T01:57:44.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.129+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4004640 con 0x7f6eec103450 2024-04-26T01:57:44.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.129+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eec069730 con 0x7f6eec103450 2024-04-26T01:57:44.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.131+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6ed4007980 con 0x7f6eec103450 2024-04-26T01:57:44.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.133+0000 7f6ee9600700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 0x7f6ed00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:44.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.133+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f6ed400f020 con 0x7f6eec103450 2024-04-26T01:57:44.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.133+0000 7f6eebe00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 0x7f6ed00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:44.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.134+0000 7f6eebe00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 0x7f6ed00a62f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6edc004470 tx=0x7f6edc014040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:44.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.137+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ed402a020 con 0x7f6eec103450 2024-04-26T01:57:44.146 INFO:tasks.workunit.client.0.smithi052.stdout: 1 272776 2.17 MB/sec execute 543 sec latency 25.975 ms 2024-04-26T01:57:44.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.462+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eec19a4e0 con 0x7f6eec103450 2024-04-26T01:57:44.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.463+0000 7f6ee9600700 1 -- 172.21.15.52:0/2642050999 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11886 (secure 0 0 0) 0x7f6ed4004000 con 0x7f6eec103450 2024-04-26T01:57:44.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:57:44.462 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:57:30.955584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:57:44.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 msgr2=0x7f6ed00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:44.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 0x7f6ed00a62f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6edc004470 tx=0x7f6edc014040 comp rx=0 tx=0).stop 2024-04-26T01:57:44.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 msgr2=0x7f6eec1996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:44.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400d010 tx=0x7f6ed4004d00 comp rx=0 tx=0).stop 2024-04-26T01:57:44.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 shutdown_connections 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f6ed00a3e30 0x7f6ed00a62f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6eec108a50 0x7f6eec19da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6eec102a90 0x7f6eec199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 --2- 172.21.15.52:0/2642050999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eec103450 0x7f6eec1996c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.467+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 >> 172.21.15.52:0/2642050999 conn(0x7f6eec079ba0 msgr2=0x7f6eec101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.468+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 shutdown_connections 2024-04-26T01:57:44.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:44.468+0000 7f6ef2a00700 1 -- 172.21.15.52:0/2642050999 wait complete. 2024-04-26T01:57:44.468 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 51 2024-04-26T01:57:45.146 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273219 2.18 MB/sec execute 544 sec latency 642.583 ms 2024-04-26T01:57:45.288 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0' 2024-04-26T01:57:45.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 79.25 --await-for 42.0 2024-04-26T01:57:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:45 smithi191 ceph-mon[32745]: pgmap v400: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 103 KiB/s wr, 21 op/s 2024-04-26T01:57:45.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:45 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2642050999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:45.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:45 smithi183 ceph-mon[32752]: pgmap v400: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 103 KiB/s wr, 21 op/s 2024-04-26T01:57:45.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:45 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2642050999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:45.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:45 smithi052 ceph-mon[27851]: pgmap v400: 97 pgs: 97 active+clean; 101 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 103 KiB/s wr, 21 op/s 2024-04-26T01:57:45.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:45 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2642050999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:57:45.786 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:57:46.146 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273286 2.17 MB/sec execute 545 sec latency 1349.324 ms 2024-04-26T01:57:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.567+0000 7fb068a00640 1 -- 172.21.15.52:0/1258395144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0641092c0 msgr2=0x7fb064110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.567+0000 7fb068a00640 1 --2- 172.21.15.52:0/1258395144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0641092c0 0x7fb064110e90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009ca0 tx=0x7fb05c02a8c0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 -- 172.21.15.52:0/1258395144 shutdown_connections 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 --2- 172.21.15.52:0/1258395144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb0641092c0 0x7fb064110e90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 --2- 172.21.15.52:0/1258395144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb064104ba0 0x7fb064108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 --2- 172.21.15.52:0/1258395144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 -- 172.21.15.52:0/1258395144 >> 172.21.15.52:0/1258395144 conn(0x7fb0640ffa30 msgr2=0x7fb064101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 -- 172.21.15.52:0/1258395144 shutdown_connections 2024-04-26T01:57:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.568+0000 7fb068a00640 1 -- 172.21.15.52:0/1258395144 wait complete. 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 Processor -- start 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 -- start start 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0641092c0 0x7fb0641a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb06419a5b0 0x7fb06419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb064114230 con 0x7fb06419a5b0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641140b0 con 0x7fb0641041f0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.569+0000 7fb068a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641143b0 con 0x7fb0641092c0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43458/0 (socket says 172.21.15.52:43458) 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 -- 172.21.15.52:0/3536612951 learned_addr learned my addr 172.21.15.52:0/3536612951 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb063200640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb06419a5b0 0x7fb06419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb061e00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0641092c0 0x7fb0641a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 -- 172.21.15.52:0/3536612951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0641092c0 msgr2=0x7fb0641a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0641092c0 0x7fb0641a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 -- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb06419a5b0 msgr2=0x7fb06419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb06419a5b0 0x7fb06419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 -- 172.21.15.52:0/3536612951 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06419b0d0 con 0x7fb0641041f0 2024-04-26T01:57:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.570+0000 7fb062800640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb05400ea60 tx=0x7fb05400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.571+0000 7fb05b400640 1 -- 172.21.15.52:0/3536612951 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05400cde0 con 0x7fb0641041f0 2024-04-26T01:57:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.571+0000 7fb05b400640 1 -- 172.21.15.52:0/3536612951 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0540035f0 con 0x7fb0641041f0 2024-04-26T01:57:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.571+0000 7fb05b400640 1 -- 172.21.15.52:0/3536612951 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054010690 con 0x7fb0641041f0 2024-04-26T01:57:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.571+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06419b3c0 con 0x7fb0641041f0 2024-04-26T01:57:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.571+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0641a7610 con 0x7fb0641041f0 2024-04-26T01:57:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.574+0000 7fb05b400640 1 -- 172.21.15.52:0/3536612951 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb0540037a0 con 0x7fb0641041f0 2024-04-26T01:57:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.574+0000 7fb06c237740 1 Processor -- start 2024-04-26T01:57:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.574+0000 7fb06c237740 1 -- start start 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae8fb90 0x55d42ae91f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d42adea900 con 0x55d42ae852d0 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d42ad90ef0 con 0x55d42ae849b0 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb06c237740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d42ade68c0 con 0x55d42ae8fb90 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb063200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb063200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43460/0 (socket says 172.21.15.52:43460) 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb063200640 1 -- 172.21.15.52:0/272134588 learned_addr learned my addr 172.21.15.52:0/272134588 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb062800640 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.575+0000 7fb061e00640 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae8fb90 0x55d42ae91f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 -- 172.21.15.52:0/272134588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae8fb90 msgr2=0x55d42ae91f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae8fb90 0x55d42ae91f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 -- 172.21.15.52:0/272134588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 msgr2=0x55d42ae8f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 -- 172.21.15.52:0/272134588 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d42aafd210 con 0x55d42ae849b0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb063200640 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009410 tx=0x7fb05c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f7b79aaa918267 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb05b400640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 0x7fb0480a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb04f400640 1 -- 172.21.15.52:0/272134588 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c031cf0 con 0x55d42ae849b0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb04f400640 1 -- 172.21.15.52:0/272134588 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb05c036030 con 0x55d42ae849b0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb04f400640 1 -- 172.21.15.52:0/272134588 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0042a0 con 0x55d42ae849b0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb05b400640 1 -- 172.21.15.52:0/3536612951 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fb0540d0b00 con 0x7fb0641041f0 2024-04-26T01:57:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.576+0000 7fb062800640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 0x7fb0480a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 -- 172.21.15.52:0/272134588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 msgr2=0x55d42ae84d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009410 tx=0x7fb05c005ef0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb062800640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 0x7fb0480a61d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d42ae85f50 tx=0x7fb0540d45b0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 -- 172.21.15.52:0/272134588 shutdown_connections 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae8fb90 0x55d42ae91f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 --2- 172.21.15.52:0/272134588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae84d90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 -- 172.21.15.52:0/272134588 >> 172.21.15.52:0/272134588 conn(0x55d42ae80ec0 msgr2=0x55d42ae812a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 -- 172.21.15.52:0/272134588 shutdown_connections 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.577+0000 7fb06c237740 1 -- 172.21.15.52:0/272134588 wait complete. 2024-04-26T01:57:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.578+0000 7fb06c237740 1 -- start start 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.578+0000 7fb06c237740 1 Processor -- start 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 0x55d42ae8cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae8d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d42ae925e0 con 0x55d42ae852d0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d42ada4310 con 0x55d42ae8fb90 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d42ae92760 con 0x55d42ae849b0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb06c237740 10 client.0 _ll_register_callbacks cb 0x55d42ad72cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb063200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 0x55d42ae8cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb063200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 0x55d42ae8cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57074/0 (socket says 172.21.15.52:57074) 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb063200640 1 -- 172.21.15.52:0/479497538 learned_addr learned my addr 172.21.15.52:0/479497538 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37464/0 (socket says 172.21.15.52:37464) 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb063200640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae8d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 -- 172.21.15.52:0/479497538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 msgr2=0x55d42ae8cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 0x55d42ae8cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 -- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 msgr2=0x55d42ae8d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae8d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.579+0000 7fb061e00640 1 -- 172.21.15.52:0/479497538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d42abc9620 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb061e00640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55d42ae822e0 tx=0x7fb050012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b1f9e042518c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb043e00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05000f8f0 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fb05000fa90 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb05c036030 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb05000fc90 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb06c237740 10 client.16515 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.580+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d42ae9e8f0 con 0x55d42ae852d0 2024-04-26T01:57:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb05c036030 con 0x55d42ae852d0 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fb050010430 con 0x55d42ae852d0 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 10 client.16515 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 20 client.16515 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42ae9fdb0 0x55d42aea21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 4 client.16515 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.581+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d42adfe8b0 con 0x55d42ae9fdb0 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.582+0000 7fb063200640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42ae9fdb0 0x55d42aea21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.582+0000 7fb063200640 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42ae9fdb0 0x55d42aea21a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cfeeb84f435d4e6 server_cookie=2313a2e3b008278a in_seq=0 out_seq=0 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.582+0000 7fb043e00640 10 client.16515 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb043e00640 1 -- 172.21.15.52:0/479497538 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d42adfe8b0 con 0x55d42ae9fdb0 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb043e00640 10 client.16515 handle_command_reply: tid=0 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 client.16515 shutdown 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 msgr2=0x55d42ae8d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55d42ae822e0 tx=0x7fb050012b60 comp rx=0 tx=0).stop 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 shutdown_connections 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42ae9fdb0 0x55d42aea21a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae8d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae852d0 0x55d42ae8d0b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb043e00640 0 client.16515 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 --2- 172.21.15.52:0/479497538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae849b0 0x55d42ae8cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 >> 172.21.15.52:0/479497538 conn(0x55d42ae80ec0 msgr2=0x55d42ae86c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 shutdown_connections 2024-04-26T01:57:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.583+0000 7fb06c237740 1 -- 172.21.15.52:0/479497538 wait complete. 2024-04-26T01:57:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.584+0000 7fb06c237740 20 client.16515 trim_cache size 0 max 16384 2024-04-26T01:57:46.598 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.600+0000 7fb06c237740 1 Processor -- start 2024-04-26T01:57:46.599 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.600+0000 7fb06c237740 1 -- start start 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae9b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae8d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641b2520 con 0x55d42ae8fb90 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641afdc0 con 0x55d42ae849b0 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb06c237740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641adc30 con 0x55d42ae852d0 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb063200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb063200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37480/0 (socket says 172.21.15.52:37480) 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb063200640 1 -- 172.21.15.52:0/2391739895 learned_addr learned my addr 172.21.15.52:0/2391739895 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb063200640 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae9b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.601+0000 7fb061e00640 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae8d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 -- 172.21.15.52:0/2391739895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 msgr2=0x55d42ae8d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae8d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 -- 172.21.15.52:0/2391739895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 msgr2=0x55d42ae9b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae9b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 -- 172.21.15.52:0/2391739895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d42ae85c60 con 0x55d42ae8fb90 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb063200640 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005cc0 tx=0x7fb05c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b800da19a32957 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb04ea00640 1 -- 172.21.15.52:0/2391739895 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c031af0 con 0x55d42ae8fb90 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb04ea00640 1 -- 172.21.15.52:0/2391739895 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb05c036040 con 0x55d42ae8fb90 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.602+0000 7fb04ea00640 1 -- 172.21.15.52:0/2391739895 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c004260 con 0x55d42ae8fb90 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 -- 172.21.15.52:0/2391739895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 msgr2=0x55d42aea7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005cc0 tx=0x7fb05c005ef0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 -- 172.21.15.52:0/2391739895 shutdown_connections 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae8fb90 0x55d42aea7a70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae8d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 --2- 172.21.15.52:0/2391739895 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae849b0 0x55d42ae9b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 -- 172.21.15.52:0/2391739895 >> 172.21.15.52:0/2391739895 conn(0x55d42ae80ec0 msgr2=0x55d42ae8ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 -- 172.21.15.52:0/2391739895 shutdown_connections 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.603+0000 7fb06c237740 1 -- 172.21.15.52:0/2391739895 wait complete. 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 -- start start 2024-04-26T01:57:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 Processor -- start 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae97ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae94ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641ae280 con 0x55d42ae849b0 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641af6f0 con 0x55d42ae8fb90 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0641ae950 con 0x55d42ae852d0 2024-04-26T01:57:46.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.604+0000 7fb06c237740 10 client.0 _ll_register_callbacks cb 0x7fb050009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37484/0 (socket says 172.21.15.52:37484) 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 -- 172.21.15.52:0/1506189149 learned_addr learned my addr 172.21.15.52:0/1506189149 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb061e00640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae97ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae94ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 -- 172.21.15.52:0/1506189149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 msgr2=0x55d42ae97ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae97ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 -- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 msgr2=0x55d42ae94ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae94ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 -- 172.21.15.52:0/1506189149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d42ae956d0 con 0x55d42ae849b0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb063200640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005c90 tx=0x7fb05c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b78b071f37b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.605+0000 7fb043e00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:57:46.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c031af0 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fb05c04a040 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb05c036040 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb05c031c90 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb06c237740 10 client.16521 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d42af05ee0 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.606+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb05c036040 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fb05c060020 con 0x55d42ae849b0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 10 client.16521 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 20 client.16521 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42af062c0 0x55d42af086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 4 client.16521 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d42adf8320 con 0x55d42af062c0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.607+0000 7fb061e00640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42af062c0 0x55d42af086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:46.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.608+0000 7fb061e00640 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42af062c0 0x55d42af086b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c372b76e330d3fe server_cookie=e8ef2945b7443fa8 in_seq=0 out_seq=0 2024-04-26T01:57:46.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.608+0000 7fb043e00640 10 client.16521 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb043e00640 1 -- 172.21.15.52:0/1506189149 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d42adf8320 con 0x55d42af062c0 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb043e00640 10 client.16521 handle_command_reply: tid=0 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 client.16521 shutdown 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 msgr2=0x55d42ae979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005c90 tx=0x7fb05c0095c0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 shutdown_connections 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d42af062c0 0x55d42af086b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d42ae8fb90 0x55d42ae94ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d42ae852d0 0x55d42ae97ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb043e00640 0 client.16521 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 --2- 172.21.15.52:0/1506189149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d42ae849b0 0x55d42ae979a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 >> 172.21.15.52:0/1506189149 conn(0x55d42ae80ec0 msgr2=0x55d42ae8e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.641+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 shutdown_connections 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb06c237740 1 -- 172.21.15.52:0/1506189149 wait complete. 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb06c237740 20 client.16521 trim_cache size 0 max 16384 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:57:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:57:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:57:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:57:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 msgr2=0x7fb0480a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 0x7fb0480a61d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d42ae85f50 tx=0x7fb0540d45b0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 msgr2=0x7fb0641a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.642+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb05400ea60 tx=0x7fb05400ef20 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 shutdown_connections 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb0480a3d10 0x7fb0480a61d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb06419a5b0 0x7fb06419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0641092c0 0x7fb0641a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 --2- 172.21.15.52:0/3536612951 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0641041f0 0x7fb0641a0400 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 >> 172.21.15.52:0/3536612951 conn(0x7fb0640ffa30 msgr2=0x7fb064100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 shutdown_connections 2024-04-26T01:57:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:46.643+0000 7fb068a00640 1 -- 172.21.15.52:0/3536612951 wait complete. 2024-04-26T01:57:47.147 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273286 2.17 MB/sec execute 546 sec latency 1158.315 ms 2024-04-26T01:57:47.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:47 smithi052 ceph-mon[27851]: pgmap v401: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 718 KiB/s wr, 69 op/s 2024-04-26T01:57:47.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:47 smithi191 ceph-mon[32745]: pgmap v401: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 718 KiB/s wr, 69 op/s 2024-04-26T01:57:47.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:47 smithi183 ceph-mon[32752]: pgmap v401: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 718 KiB/s wr, 69 op/s 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.086+0000 7f2002400700 1 -- 172.21.15.52:0/1532927617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 msgr2=0x7f1ffc10f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.086+0000 7f2002400700 1 --2- 172.21.15.52:0/1532927617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc10f8c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009f90 tx=0x7f1ff400b330 comp rx=0 tx=0).stop 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 -- 172.21.15.52:0/1532927617 shutdown_connections 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 --2- 172.21.15.52:0/1532927617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc10f8c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 --2- 172.21.15.52:0/1532927617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 0x7f1ffc108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 --2- 172.21.15.52:0/1532927617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc06abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 -- 172.21.15.52:0/1532927617 >> 172.21.15.52:0/1532927617 conn(0x7f1ffc0788e0 msgr2=0x7f1ffc078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 -- 172.21.15.52:0/1532927617 shutdown_connections 2024-04-26T01:57:48.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.087+0000 7f2002400700 1 -- 172.21.15.52:0/1532927617 wait complete. 2024-04-26T01:57:48.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.088+0000 7f2002400700 1 Processor -- start 2024-04-26T01:57:48.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.088+0000 7f2002400700 1 -- start start 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc19b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 0x7f1ffc19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc102c40 con 0x7f1ffc108a60 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc102ac0 con 0x7f1ffc06a740 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f2002400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc102dc0 con 0x7f1ffc06b100 2024-04-26T01:57:48.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.089+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43508/0 (socket says 172.21.15.52:43508) 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 -- 172.21.15.52:0/999237868 learned_addr learned my addr 172.21.15.52:0/999237868 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc19b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 0x7f1ffc19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 -- 172.21.15.52:0/999237868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 msgr2=0x7f1ffc19b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc19b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 -- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 msgr2=0x7f1ffc19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 0x7f1ffc19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffb400700 1 -- 172.21.15.52:0/999237868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1a0220 con 0x7f1ffc06a740 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.090+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.091+0000 7f1ffb400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00d980 tx=0x7f1fec00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.091+0000 7f1ff3e00700 1 -- 172.21.15.52:0/999237868 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec016070 con 0x7f1ffc06a740 2024-04-26T01:57:48.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.091+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1a0510 con 0x7f1ffc06a740 2024-04-26T01:57:48.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.091+0000 7f1ff3e00700 1 -- 172.21.15.52:0/999237868 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fec004830 con 0x7f1ffc06a740 2024-04-26T01:57:48.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.091+0000 7f1ff3e00700 1 -- 172.21.15.52:0/999237868 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec005540 con 0x7f1ffc06a740 2024-04-26T01:57:48.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.092+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1a7d90 con 0x7f1ffc06a740 2024-04-26T01:57:48.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.095+0000 7f1ff3e00700 1 -- 172.21.15.52:0/999237868 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1fec004220 con 0x7f1ffc06a740 2024-04-26T01:57:48.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.097+0000 7f1ff3e00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.097+0000 7f200cd19b80 1 Processor -- start 2024-04-26T01:57:48.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.097+0000 7f1ff3e00700 1 -- 172.21.15.52:0/999237868 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f1fec0d04e0 con 0x7f1ffc06a740 2024-04-26T01:57:48.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.097+0000 7f200cd19b80 1 -- start start 2024-04-26T01:57:48.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2ba020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 0x55ec8d2bc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d202020 con 0x55ec8d2aedd0 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d1a88c0 con 0x55ec8d2ba560 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f200cd19b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d1fdfe0 con 0x55ec8d2af6f0 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2ba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f1ffbe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 0x55ec8d2bc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2ba020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57134/0 (socket says 172.21.15.52:57134) 2024-04-26T01:57:48.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.098+0000 7f1ffaa00700 1 -- 172.21.15.52:0/3759422879 learned_addr learned my addr 172.21.15.52:0/3759422879 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:48.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 -- 172.21.15.52:0/3759422879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 msgr2=0x55ec8d2ba020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2ba020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 -- 172.21.15.52:0/3759422879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2aedd0 msgr2=0x55ec8d2af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 -- 172.21.15.52:0/3759422879 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec8d19e810 con 0x55ec8d2ba560 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8014a10 tx=0x7f1fe8014dd0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.099+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 0x55ec8d2bc950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4006070 tx=0x7f1ff4009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d46a91bb6f647d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.100+0000 7f1ff0c00700 1 -- 172.21.15.52:0/3759422879 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4020070 con 0x55ec8d2ba560 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.100+0000 7f1ff0c00700 1 -- 172.21.15.52:0/3759422879 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff400bd60 con 0x55ec8d2ba560 2024-04-26T01:57:48.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.100+0000 7f1ff0c00700 1 -- 172.21.15.52:0/3759422879 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4007c10 con 0x55ec8d2ba560 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 -- 172.21.15.52:0/3759422879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 msgr2=0x55ec8d2bc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 0x55ec8d2bc950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4006070 tx=0x7f1ff4009390 comp rx=0 tx=0).stop 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 -- 172.21.15.52:0/3759422879 shutdown_connections 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2ba560 0x55ec8d2bc950 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3759422879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2ba020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 -- 172.21.15.52:0/3759422879 >> 172.21.15.52:0/3759422879 conn(0x55ec8d1de0e0 msgr2=0x55ec8d2ab2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 -- 172.21.15.52:0/3759422879 shutdown_connections 2024-04-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.101+0000 7f200cd19b80 1 -- 172.21.15.52:0/3759422879 wait complete. 2024-04-26T01:57:48.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.102+0000 7f200cd19b80 1 -- start start 2024-04-26T01:57:48.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.102+0000 7f200cd19b80 1 Processor -- start 2024-04-26T01:57:48.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d2bcf30 con 0x55ec8d2ba560 2024-04-26T01:57:48.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d1bb9f0 con 0x55ec8d2af6f0 2024-04-26T01:57:48.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ec8d2bd0b0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f200cd19b80 10 client.0 _ll_register_callbacks cb 0x55ec8d19a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f1ffbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57148/0 (socket says 172.21.15.52:57148) 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.103+0000 7f1ffb400700 1 -- 172.21.15.52:0/248521575 learned_addr learned my addr 172.21.15.52:0/248521575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:48.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 -- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2af6f0 msgr2=0x55ec8d2b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 -- 172.21.15.52:0/248521575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2ba560 msgr2=0x55ec8d2b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 -- 172.21.15.52:0/248521575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec8d14fa60 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1ffb400700 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fec09c3b0 tx=0x7f1fec004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73720d4cbbc7fdec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1fe5600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0b5060 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f1fec098a90 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.104+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff400bd60 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1fec09c420 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 10 client.26671 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ec8d198c80 con 0x55ec8d2aedd0 2024-04-26T01:57:48.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff400bd60 con 0x55ec8d2aedd0 2024-04-26T01:57:48.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7f1fec09c680 con 0x55ec8d2aedd0 2024-04-26T01:57:48.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 10 client.26671 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:48.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 10 client.26671 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 20 client.26671 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d2c9c00 0x55ec8d2cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 4 client.26671 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.105+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec8d20f810 con 0x55ec8d2c9c00 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.106+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d2c9c00 0x55ec8d2cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.106+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d2c9c00 0x55ec8d2cbff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb781c90275900c2 server_cookie=3717040433b6329c in_seq=0 out_seq=0 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.106+0000 7f1fe5600700 10 client.26671 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.107+0000 7f1fe5600700 1 -- 172.21.15.52:0/248521575 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec8d20f810 con 0x55ec8d2c9c00 2024-04-26T01:57:48.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.107+0000 7f1fe5600700 10 client.26671 handle_command_reply: tid=0 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.107+0000 7f200cd19b80 1 client.26671 shutdown 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 msgr2=0x55ec8d2b9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fec09c3b0 tx=0x7f1fec004580 comp rx=0 tx=0).stop 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 shutdown_connections 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d2c9c00 0x55ec8d2cbff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2b9c40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 --2- 172.21.15.52:0/248521575 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 >> 172.21.15.52:0/248521575 conn(0x55ec8d1de0e0 msgr2=0x55ec8d2aaed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f1fe5600700 0 client.26671 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 shutdown_connections 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 1 -- 172.21.15.52:0/248521575 wait complete. 2024-04-26T01:57:48.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.108+0000 7f200cd19b80 20 client.26671 trim_cache size 0 max 16384 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.135+0000 7f200cd19b80 1 Processor -- start 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.135+0000 7f200cd19b80 1 -- start start 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1b1c20 con 0x55ec8d2af6f0 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1af480 con 0x55ec8d2aedd0 2024-04-26T01:57:48.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.136+0000 7f200cd19b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1ad2f0 con 0x55ec8d2ba560 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43542/0 (socket says 172.21.15.52:43542) 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 -- 172.21.15.52:0/3484074999 learned_addr learned my addr 172.21.15.52:0/3484074999 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffaa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2d1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37532/0 (socket says 172.21.15.52:37532) 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:57164/0 (socket says 172.21.15.52:57164) 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 -- 172.21.15.52:0/3484074999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 msgr2=0x55ec8d2b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 -- 172.21.15.52:0/3484074999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 msgr2=0x55ec8d2d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 -- 172.21.15.52:0/3484074999 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec8d2afd70 con 0x55ec8d2aedd0 2024-04-26T01:57:48.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.137+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1fec09c0d0 tx=0x7f1fec09bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b943aad72e1753f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.138+0000 7f1fe7e00700 1 -- 172.21.15.52:0/3484074999 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec014070 con 0x55ec8d2aedd0 2024-04-26T01:57:48.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.138+0000 7f1fe7e00700 1 -- 172.21.15.52:0/3484074999 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fec0a07d0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.138+0000 7f1fe7e00700 1 -- 172.21.15.52:0/3484074999 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0a4e90 con 0x55ec8d2aedd0 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 -- 172.21.15.52:0/3484074999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 msgr2=0x55ec8d2d1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1fec09c0d0 tx=0x7f1fec09bf60 comp rx=0 tx=0).stop 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 -- 172.21.15.52:0/3484074999 shutdown_connections 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2d1880 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3484074999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2d1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 -- 172.21.15.52:0/3484074999 >> 172.21.15.52:0/3484074999 conn(0x55ec8d1de0e0 msgr2=0x55ec8d2b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 -- 172.21.15.52:0/3484074999 shutdown_connections 2024-04-26T01:57:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.139+0000 7f200cd19b80 1 -- 172.21.15.52:0/3484074999 wait complete. 2024-04-26T01:57:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.140+0000 7f200cd19b80 1 -- start start 2024-04-26T01:57:48.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.140+0000 7f200cd19b80 1 Processor -- start 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2bfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1ad940 con 0x55ec8d2af6f0 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1aedb0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1ae010 con 0x55ec8d2ba560 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.141+0000 7f200cd19b80 10 client.0 _ll_register_callbacks cb 0x55ec8d19a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43544/0 (socket says 172.21.15.52:43544) 2024-04-26T01:57:48.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 -- 172.21.15.52:0/3993023642 learned_addr learned my addr 172.21.15.52:0/3993023642 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2bfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 -- 172.21.15.52:0/3993023642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 msgr2=0x55ec8d2c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 -- 172.21.15.52:0/3993023642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 msgr2=0x55ec8d2bfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2bfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffb400700 1 -- 172.21.15.52:0/3993023642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec8d2b0590 con 0x55ec8d2aedd0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.142+0000 7f1ffbe00700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f1ffb400700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1fec000c00 tx=0x7f1fec005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9a81a50b0146a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f1fe5600700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec083070 con 0x55ec8d2aedd0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fec0a07d0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7f1fec098a90 con 0x55ec8d2aedd0 2024-04-26T01:57:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1fec0ac050 con 0x55ec8d2aedd0 2024-04-26T01:57:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f200cd19b80 10 client.26669 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.143+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ec8d33a8e0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fec0a07d0 con 0x55ec8d2aedd0 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7f1fec098170 con 0x55ec8d2aedd0 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 10 client.26669 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 10 client.26669 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 20 client.26669 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d33ad10 0x55ec8d33d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 4 client.26669 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}] 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.25, "await_for": 42.0}) -- 0x55ec8d2098b0 con 0x55ec8d33ad10 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.144+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d33ad10 0x55ec8d33d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.145+0000 7f1ffaa00700 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d33ad10 0x55ec8d33d100 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=971373f29ff808c server_cookie=42c4eaf79129dd21 in_seq=0 out_seq=0 2024-04-26T01:57:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.145+0000 7f1fe5600700 10 client.26669 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:48.147 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273307 2.16 MB/sec execute 547 sec latency 2079.505 ms 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.209+0000 7f1fe5600700 1 -- 172.21.15.52:0/3993023642 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ec8d2098b0 con 0x55ec8d33ad10 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.209+0000 7f1fe5600700 10 client.26669 handle_command_reply: tid=0 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 client.26669 shutdown 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 msgr2=0x55ec8d2c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1fec000c00 tx=0x7f1fec005bb0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 shutdown_connections 2024-04-26T01:57:48.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ec8d33ad10 0x55ec8d33d100 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ec8d2ba560 0x55ec8d2c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ec8d2aedd0 0x55ec8d2c2920 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 --2- 172.21.15.52:0/3993023642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ec8d2af6f0 0x55ec8d2bfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f1fe5600700 0 client.26669 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 >> 172.21.15.52:0/3993023642 conn(0x55ec8d1de0e0 msgr2=0x55ec8d2b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 shutdown_connections 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 1 -- 172.21.15.52:0/3993023642 wait complete. 2024-04-26T01:57:48.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.210+0000 7f200cd19b80 20 client.26669 trim_cache size 0 max 16384 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 51, 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 38, 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:57:48.211 INFO:teuthology.orchestra.run.smithi052.stdout: "bde8d999": { 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 38, 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 79.2, 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:57:48.212 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:57:48.213 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:57:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 msgr2=0x7f1fe00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8014a10 tx=0x7f1fe8014dd0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 msgr2=0x7f1ffc19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00d980 tx=0x7f1fec00dc90 comp rx=0 tx=0).stop 2024-04-26T01:57:48.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 shutdown_connections 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1fe00a3ed0 0x7f1fe00a6390 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1ffc108a60 0x7f1ffc19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ffc06a740 0x7f1ffc19b210 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 --2- 172.21.15.52:0/999237868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ffc06b100 0x7f1ffc19b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.211+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 >> 172.21.15.52:0/999237868 conn(0x7f1ffc0788e0 msgr2=0x7f1ffc106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.212+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 shutdown_connections 2024-04-26T01:57:48.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:57:48.212+0000 7f2002400700 1 -- 172.21.15.52:0/999237868 wait complete. 2024-04-26T01:57:49.012 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bde8d999', quiesce took 7.5 seconds. Will release after: 19.25 2024-04-26T01:57:49.147 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.16 MB/sec execute 548 sec latency 953.988 ms 2024-04-26T01:57:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:49 smithi191 ceph-mon[32745]: pgmap v402: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:49.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:49 smithi183 ceph-mon[32752]: pgmap v402: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:49 smithi052 ceph-mon[27851]: pgmap v402: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:50.147 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.16 MB/sec execute 549 sec latency 1954.126 ms 2024-04-26T01:57:51.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.15 MB/sec execute 550 sec latency 2954.287 ms 2024-04-26T01:57:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:51 smithi191 ceph-mon[32745]: pgmap v403: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:51 smithi183 ceph-mon[32752]: pgmap v403: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:51 smithi052 ceph-mon[27851]: pgmap v403: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 642 KiB/s wr, 53 op/s 2024-04-26T01:57:52.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.15 MB/sec execute 551 sec latency 3954.434 ms 2024-04-26T01:57:53.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.15 MB/sec execute 552 sec latency 4954.631 ms 2024-04-26T01:57:53.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:53 smithi191 ceph-mon[32745]: pgmap v404: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 696 KiB/s wr, 77 op/s 2024-04-26T01:57:53.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:53 smithi183 ceph-mon[32752]: pgmap v404: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 696 KiB/s wr, 77 op/s 2024-04-26T01:57:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:53 smithi052 ceph-mon[27851]: pgmap v404: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 696 KiB/s wr, 77 op/s 2024-04-26T01:57:54.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.14 MB/sec execute 553 sec latency 5954.786 ms 2024-04-26T01:57:55.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.14 MB/sec execute 554 sec latency 6954.873 ms 2024-04-26T01:57:55.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:55 smithi191 ceph-mon[32745]: pgmap v405: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 670 KiB/s wr, 71 op/s 2024-04-26T01:57:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:55 smithi183 ceph-mon[32752]: pgmap v405: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 670 KiB/s wr, 71 op/s 2024-04-26T01:57:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:55 smithi052 ceph-mon[27851]: pgmap v405: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 670 KiB/s wr, 71 op/s 2024-04-26T01:57:56.148 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.14 MB/sec execute 555 sec latency 7955.038 ms 2024-04-26T01:57:57.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.13 MB/sec execute 556 sec latency 8955.174 ms 2024-04-26T01:57:57.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:57 smithi191 ceph-mon[32745]: pgmap v406: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 671 KiB/s wr, 75 op/s 2024-04-26T01:57:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:57 smithi183 ceph-mon[32752]: pgmap v406: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 671 KiB/s wr, 75 op/s 2024-04-26T01:57:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:57 smithi052 ceph-mon[27851]: pgmap v406: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 671 KiB/s wr, 75 op/s 2024-04-26T01:57:58.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.13 MB/sec execute 557 sec latency 9955.309 ms 2024-04-26T01:57:59.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.12 MB/sec execute 558 sec latency 10955.448 ms 2024-04-26T01:57:59.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:57:59 smithi191 ceph-mon[32745]: pgmap v407: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:57:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:57:59 smithi183 ceph-mon[32752]: pgmap v407: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:57:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:57:59 smithi052 ceph-mon[27851]: pgmap v407: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:57:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 -- 172.21.15.52:0/3013295768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8102990 msgr2=0x7fcbd81069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 --2- 172.21.15.52:0/3013295768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8102990 0x7fcbd81069d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009900 tx=0x7fcbc402a3f0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 -- 172.21.15.52:0/3013295768 shutdown_connections 2024-04-26T01:57:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 --2- 172.21.15.52:0/3013295768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd81070b0 0x7fcbd810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 --2- 172.21.15.52:0/3013295768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8102990 0x7fcbd81069d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.791+0000 7fcbdd200640 1 --2- 172.21.15.52:0/3013295768 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.792+0000 7fcbdd200640 1 -- 172.21.15.52:0/3013295768 >> 172.21.15.52:0/3013295768 conn(0x7fcbd80fd8a0 msgr2=0x7fcbd80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.792+0000 7fcbdd200640 1 -- 172.21.15.52:0/3013295768 shutdown_connections 2024-04-26T01:57:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.792+0000 7fcbdd200640 1 -- 172.21.15.52:0/3013295768 wait complete. 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 Processor -- start 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 -- start start 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd81070b0 0x7fcbd81a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 0x7fcbd819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd8112040 con 0x7fcbd8101fe0 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd8111ec0 con 0x7fcbd81070b0 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.793+0000 7fcbdd200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81121c0 con 0x7fcbd819a5e0 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42920/0 (socket says 172.21.15.52:42920) 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 -- 172.21.15.52:0/771661410 learned_addr learned my addr 172.21.15.52:0/771661410 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 0x7fcbd819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd81070b0 0x7fcbd81a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 -- 172.21.15.52:0/771661410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 msgr2=0x7fcbd819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 0x7fcbd819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 -- 172.21.15.52:0/771661410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd81070b0 msgr2=0x7fcbd81a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd81070b0 0x7fcbd81a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 -- 172.21.15.52:0/771661410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd819b100 con 0x7fcbd8101fe0 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 0x7fcbd819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:57:59.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.794+0000 7fcbd7400640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800ea60 tx=0x7fcbc800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.795+0000 7fcbcfe00640 1 -- 172.21.15.52:0/771661410 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc800cde0 con 0x7fcbd8101fe0 2024-04-26T01:57:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.795+0000 7fcbcfe00640 1 -- 172.21.15.52:0/771661410 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbc80035f0 con 0x7fcbd8101fe0 2024-04-26T01:57:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.795+0000 7fcbcfe00640 1 -- 172.21.15.52:0/771661410 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8010690 con 0x7fcbd8101fe0 2024-04-26T01:57:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.795+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd819b3f0 con 0x7fcbd8101fe0 2024-04-26T01:57:59.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.795+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd81a75f0 con 0x7fcbd8101fe0 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.798+0000 7fcbe1636740 1 Processor -- start 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.798+0000 7fcbe1636740 1 -- start start 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.798+0000 7fcbcfe00640 1 -- 172.21.15.52:0/771661410 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fcbc80037a0 con 0x7fcbd8101fe0 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ebf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ec1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a061e1a900 con 0x55a061ebfb90 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a061dc0ef0 con 0x55a061eb49b0 2024-04-26T01:57:59.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbe1636740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a061e168c0 con 0x55a061eb52d0 2024-04-26T01:57:59.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ec1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ebf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33082/0 (socket says 172.21.15.52:33082) 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 -- 172.21.15.52:0/1624204609 learned_addr learned my addr 172.21.15.52:0/1624204609 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ec1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42936/0 (socket says 172.21.15.52:42936) 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 -- 172.21.15.52:0/1624204609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 msgr2=0x55a061ebf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ebf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 -- 172.21.15.52:0/1624204609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 msgr2=0x55a061ec1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.799+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ec1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbd7e00640 1 -- 172.21.15.52:0/1624204609 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a061b2d210 con 0x55a061eb49b0 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011790 tx=0x7fcbd0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b807bc9127bf08d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbccc00640 1 -- 172.21.15.52:0/1624204609 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0003c30 con 0x55a061eb49b0 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbccc00640 1 -- 172.21.15.52:0/1624204609 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd000fe50 con 0x55a061eb49b0 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbccc00640 1 -- 172.21.15.52:0/1624204609 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd00193e0 con 0x55a061eb49b0 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.800+0000 7fcbcfe00640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 0x7fcbbc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbcfe00640 1 -- 172.21.15.52:0/771661410 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fcbc80d0a80 con 0x7fcbd8101fe0 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbd7400640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 0x7fcbbc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 -- 172.21.15.52:0/1624204609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 msgr2=0x55a061eb4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0011790 tx=0x7fcbd0011c50 comp rx=0 tx=0).stop 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 -- 172.21.15.52:0/1624204609 shutdown_connections 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ec1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ebf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1624204609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb49b0 0x55a061eb4d90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 -- 172.21.15.52:0/1624204609 >> 172.21.15.52:0/1624204609 conn(0x55a061eb0ec0 msgr2=0x55a061eb12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 -- 172.21.15.52:0/1624204609 shutdown_connections 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbe1636740 1 -- 172.21.15.52:0/1624204609 wait complete. 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.801+0000 7fcbd7400640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 0x7fcbbc0a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55a061eb5f50 tx=0x7fcbc80d4530 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.802+0000 7fcbe1636740 1 -- start start 2024-04-26T01:57:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.802+0000 7fcbe1636740 1 Processor -- start 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.802+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ebcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.802+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.802+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ebd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbe1636740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a061ec27c0 con 0x55a061ebfb90 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbe1636740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a061dd4310 con 0x55a061eb52d0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbe1636740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a061ec2940 con 0x55a061eb49b0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbe1636740 10 client.0 _ll_register_callbacks cb 0x55a061da2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33084/0 (socket says 172.21.15.52:33084) 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ebd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 -- 172.21.15.52:0/1154345316 learned_addr learned my addr 172.21.15.52:0/1154345316 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ebcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 -- 172.21.15.52:0/1154345316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 msgr2=0x55a061ebcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ebcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 -- 172.21.15.52:0/1154345316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 msgr2=0x55a061ebd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ebd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd6a00640 1 -- 172.21.15.52:0/1154345316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a061bf9620 con 0x55a061eb52d0 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.803+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ebcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000c00 tx=0x7fcbc40043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4cee69e03d54082 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbc1600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4035040 con 0x55a061eb52d0 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fcbc4005450 con 0x55a061eb52d0 2024-04-26T01:57:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbd000fe50 con 0x55a061eb52d0 2024-04-26T01:57:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fcbc4005650 con 0x55a061eb52d0 2024-04-26T01:57:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbe1636740 10 client.26681 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.804+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a061ecea40 con 0x55a061eb52d0 2024-04-26T01:57:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd000fe50 con 0x55a061eb52d0 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fcbc4003ab0 con 0x55a061eb52d0 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 10 client.26681 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 10 client.26681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 20 client.26681 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061ecff50 0x55a061ed2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 4 client.26681 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.805+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a061e2e8b0 con 0x55a061ecff50 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.806+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061ecff50 0x55a061ed2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.806+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061ecff50 0x55a061ed2340 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9080527d6f864ac server_cookie=2198cf69e84f96c1 in_seq=0 out_seq=0 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.806+0000 7fcbc1600640 10 client.26681 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbc1600640 1 -- 172.21.15.52:0/1154345316 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a061e2e8b0 con 0x55a061ecff50 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbc1600640 10 client.26681 handle_command_reply: tid=0 2024-04-26T01:57:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 client.26681 shutdown 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 msgr2=0x55a061ebd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000c00 tx=0x7fcbc40043b0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 shutdown_connections 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061ecff50 0x55a061ed2340 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061ebfb90 0x55a061ebd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061eb52d0 0x55a061ebd070 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 --2- 172.21.15.52:0/1154345316 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ebcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 >> 172.21.15.52:0/1154345316 conn(0x55a061eb0ec0 msgr2=0x55a061eb6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.807+0000 7fcbc1600640 0 client.26681 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.808+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 shutdown_connections 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.808+0000 7fcbe1636740 1 -- 172.21.15.52:0/1154345316 wait complete. 2024-04-26T01:57:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.808+0000 7fcbe1636740 20 client.26681 trim_cache size 0 max 16384 2024-04-26T01:57:59.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 Processor -- start 2024-04-26T01:57:59.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 -- start start 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb52d0 0x55a061ebd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061ed7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81b2530 con 0x55a061eb52d0 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81afdd0 con 0x55a061ebfb90 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.824+0000 7fcbe1636740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81adc40 con 0x55a061eb49b0 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36964/0 (socket says 172.21.15.52:36964) 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 -- 172.21.15.52:0/3675600986 learned_addr learned my addr 172.21.15.52:0/3675600986 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb52d0 0x55a061ebd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061ed7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 -- 172.21.15.52:0/3675600986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 msgr2=0x55a061ed7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061ed7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 -- 172.21.15.52:0/3675600986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb52d0 msgr2=0x55a061ebd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb52d0 0x55a061ebd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.825+0000 7fcbd7e00640 1 -- 172.21.15.52:0/3675600986 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a061ec22f0 con 0x55a061eb49b0 2024-04-26T01:57:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.826+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006010 tx=0x7fcbd001c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1315697d796476b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.826+0000 7fcbc3e00640 1 -- 172.21.15.52:0/3675600986 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0004050 con 0x55a061eb49b0 2024-04-26T01:57:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.826+0000 7fcbc3e00640 1 -- 172.21.15.52:0/3675600986 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd00041f0 con 0x55a061eb49b0 2024-04-26T01:57:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.826+0000 7fcbc3e00640 1 -- 172.21.15.52:0/3675600986 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000fc60 con 0x55a061eb49b0 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- 172.21.15.52:0/3675600986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 msgr2=0x55a061ecb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006010 tx=0x7fcbd001c040 comp rx=0 tx=0).stop 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- 172.21.15.52:0/3675600986 shutdown_connections 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061ed7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb52d0 0x55a061ebd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 --2- 172.21.15.52:0/3675600986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb49b0 0x55a061ecb680 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- 172.21.15.52:0/3675600986 >> 172.21.15.52:0/3675600986 conn(0x55a061eb0ec0 msgr2=0x55a061ebcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- 172.21.15.52:0/3675600986 shutdown_connections 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- 172.21.15.52:0/3675600986 wait complete. 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.827+0000 7fcbe1636740 1 -- start start 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 Processor -- start 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ec6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061f2f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81ae290 con 0x55a061eb49b0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81af700 con 0x55a061ebfb90 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd81ae960 con 0x55a061eb52d0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.828+0000 7fcbe1636740 10 client.0 _ll_register_callbacks cb 0x7fcbc4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42964/0 (socket says 172.21.15.52:42964) 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 -- 172.21.15.52:0/519839742 learned_addr learned my addr 172.21.15.52:0/519839742 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061f2f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ec6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 -- 172.21.15.52:0/519839742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 msgr2=0x55a061ec6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ec6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 -- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 msgr2=0x55a061f2f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061f2f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 -- 172.21.15.52:0/519839742 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a061f2fa90 con 0x55a061eb49b0 2024-04-26T01:57:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbd7e00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000f810 tx=0x7fcbd002d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974a36cdb581a847 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:57:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.829+0000 7fcbc1600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0003a30 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8521+0+0 (secure 0 0 0) 0x7fcbd0003bd0 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbd00041f0 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fcbd0003dd0 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbe1636740 10 client.16557 fetch_fsmap learned FSMap version 51 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a061f36e20 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd00041f0 con 0x55a061eb49b0 2024-04-26T01:57:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.830+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7fcbd0018070 con 0x55a061eb49b0 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 10 client.16557 fetch_fsmap finished waiting for FSMap version 51 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 10 client.16557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 20 client.16557 populate_metadata read hostname 'smithi052' 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061f37200 0x55a061f395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 4 client.16557 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a061e28320 con 0x55a061f37200 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061f37200 0x55a061f395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbd6a00640 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061f37200 0x55a061f395f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adf9edf3af052288 server_cookie=7467b0a9b561ceca in_seq=0 out_seq=0 2024-04-26T01:57:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.831+0000 7fcbc1600640 10 client.16557 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.858+0000 7fcbc1600640 1 -- 172.21.15.52:0/519839742 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a061e28320 con 0x55a061f37200 2024-04-26T01:57:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.858+0000 7fcbc1600640 10 client.16557 handle_command_reply: tid=0 2024-04-26T01:57:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.858+0000 7fcbe1636740 1 client.16557 shutdown 2024-04-26T01:57:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.858+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 msgr2=0x55a061ec5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.858+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000f810 tx=0x7fcbd002d040 comp rx=0 tx=0).stop 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 shutdown_connections 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a061f37200 0x55a061f395f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a061ebfb90 0x55a061f2f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a061eb52d0 0x55a061ec6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 --2- 172.21.15.52:0/519839742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a061eb49b0 0x55a061ec5df0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 >> 172.21.15.52:0/519839742 conn(0x55a061eb0ec0 msgr2=0x55a061eb6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbc1600640 0 client.16557 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 shutdown_connections 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 1 -- 172.21.15.52:0/519839742 wait complete. 2024-04-26T01:57:59.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.859+0000 7fcbe1636740 20 client.16557 trim_cache size 0 max 16384 2024-04-26T01:57:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:57:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:57:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:57:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 msgr2=0x7fcbbc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 0x7fcbbc0a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55a061eb5f50 tx=0x7fcbc80d4530 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 msgr2=0x7fcbd81a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800ea60 tx=0x7fcbc800ef20 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 shutdown_connections 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcbbc0a3d90 0x7fcbbc0a6250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcbd819a5e0 0x7fcbd819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcbd81070b0 0x7fcbd81a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 --2- 172.21.15.52:0/771661410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcbd8101fe0 0x7fcbd81a0430 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 >> 172.21.15.52:0/771661410 conn(0x7fcbd80fd8a0 msgr2=0x7fcbd80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 shutdown_connections 2024-04-26T01:57:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:57:59.860+0000 7fcbdd200640 1 -- 172.21.15.52:0/771661410 wait complete. 2024-04-26T01:58:00.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.12 MB/sec execute 559 sec latency 11955.555 ms 2024-04-26T01:58:01.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.12 MB/sec execute 560 sec latency 12955.791 ms 2024-04-26T01:58:01.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:01 smithi191 ceph-mon[32745]: pgmap v408: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:58:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:01 smithi183 ceph-mon[32752]: pgmap v408: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:58:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:01 smithi052 ceph-mon[27851]: pgmap v408: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 28 op/s 2024-04-26T01:58:02.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.11 MB/sec execute 561 sec latency 13955.936 ms 2024-04-26T01:58:03.149 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.11 MB/sec execute 562 sec latency 14956.085 ms 2024-04-26T01:58:03.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:03 smithi191 ceph-mon[32745]: pgmap v409: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 132 KiB/s wr, 57 op/s 2024-04-26T01:58:03.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:03 smithi183 ceph-mon[32752]: pgmap v409: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 132 KiB/s wr, 57 op/s 2024-04-26T01:58:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:03 smithi052 ceph-mon[27851]: pgmap v409: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 132 KiB/s wr, 57 op/s 2024-04-26T01:58:04.150 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.11 MB/sec execute 563 sec latency 15956.252 ms 2024-04-26T01:58:05.150 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.10 MB/sec execute 564 sec latency 16956.419 ms 2024-04-26T01:58:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:05 smithi191 ceph-mon[32745]: pgmap v410: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-26T01:58:05.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:05 smithi183 ceph-mon[32752]: pgmap v410: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-26T01:58:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:05 smithi052 ceph-mon[27851]: pgmap v410: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-26T01:58:06.150 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.10 MB/sec execute 565 sec latency 17956.619 ms 2024-04-26T01:58:07.150 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.09 MB/sec execute 566 sec latency 18956.795 ms 2024-04-26T01:58:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:07 smithi191 ceph-mon[32745]: pgmap v411: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 174 KiB/s wr, 37 op/s 2024-04-26T01:58:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:07 smithi191 ceph-mon[32745]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:58:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:07 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:07 smithi183 ceph-mon[32752]: pgmap v411: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 174 KiB/s wr, 37 op/s 2024-04-26T01:58:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:07 smithi183 ceph-mon[32752]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:58:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:07 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:07 smithi052 ceph-mon[27851]: pgmap v411: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 174 KiB/s wr, 37 op/s 2024-04-26T01:58:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:07 smithi052 ceph-mon[27851]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T01:58:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:07 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:08.150 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.09 MB/sec execute 567 sec latency 19956.934 ms 2024-04-26T01:58:08.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:58:08.740 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:09.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.09 MB/sec execute 568 sec latency 20957.030 ms 2024-04-26T01:58:09.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:09 smithi052 ceph-mon[27851]: pgmap v413: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:09.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:09 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:09.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:09 smithi191 ceph-mon[32745]: pgmap v413: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:09.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:09 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:09 smithi183 ceph-mon[32752]: pgmap v413: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:09 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:10.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.08 MB/sec execute 569 sec latency 21957.182 ms 2024-04-26T01:58:11.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 -- 172.21.15.52:0/568409140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c108a30 msgr2=0x7f470c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:11.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 --2- 172.21.15.52:0/568409140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c108a30 0x7f470c108e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009990 tx=0x7f46fc009ca0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 -- 172.21.15.52:0/568409140 shutdown_connections 2024-04-26T01:58:11.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 --2- 172.21.15.52:0/568409140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c06b0f0 0x7f470c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 --2- 172.21.15.52:0/568409140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c108a30 0x7f470c108e10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 --2- 172.21.15.52:0/568409140 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06a750 0x7f470c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 -- 172.21.15.52:0/568409140 >> 172.21.15.52:0/568409140 conn(0x7f470c0788f0 msgr2=0x7f470c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 -- 172.21.15.52:0/568409140 shutdown_connections 2024-04-26T01:58:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.078+0000 7f4711200700 1 -- 172.21.15.52:0/568409140 wait complete. 2024-04-26T01:58:11.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.079+0000 7f4711200700 1 Processor -- start 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.079+0000 7f4711200700 1 -- start start 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 0x7f470c101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 0x7f470c101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c108a30 0x7f470c102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f470c112bd0 con 0x7f470c06a750 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f470c112a50 con 0x7f470c108a30 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4711200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f470c112d50 con 0x7f470c06b0f0 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4701400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 0x7f470c101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4709e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 0x7f470c101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:11.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4701400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 0x7f470c101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43792/0 (socket says 172.21.15.52:43792) 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f4701400700 1 -- 172.21.15.52:0/2957487252 learned_addr learned my addr 172.21.15.52:0/2957487252 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.080+0000 7f470a800700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c108a30 0x7f470c102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 -- 172.21.15.52:0/2957487252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 msgr2=0x7f470c101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 0x7f470c101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 -- 172.21.15.52:0/2957487252 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c108a30 msgr2=0x7f470c102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c108a30 0x7f470c102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 -- 172.21.15.52:0/2957487252 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1a7610 con 0x7f470c06a750 2024-04-26T01:58:11.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.081+0000 7f4709e00700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 0x7f470c101a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f46fc006010 tx=0x7f46fc0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.082+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc027070 con 0x7f470c06a750 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.082+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46fc022470 con 0x7f470c06a750 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.082+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc017650 con 0x7f470c06a750 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.082+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1a78a0 con 0x7f470c06a750 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.082+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1a7cf0 con 0x7f470c06a750 2024-04-26T01:58:11.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.083+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c069730 con 0x7f470c06a750 2024-04-26T01:58:11.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.085+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f46fc02a070 con 0x7f470c06a750 2024-04-26T01:58:11.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.087+0000 7f4703400700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:11.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.087+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8565+0+0 (secure 0 0 0) 0x7f46fc0d1440 con 0x7f470c06a750 2024-04-26T01:58:11.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.087+0000 7f4701400700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:11.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.088+0000 7f4701400700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 0x7f46f00a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f470c078ef0 tx=0x7f46f400b040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:11.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.092+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46fc013070 con 0x7f470c06a750 2024-04-26T01:58:11.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.08 MB/sec execute 570 sec latency 22957.292 ms 2024-04-26T01:58:11.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.380+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f470c02e140 con 0x7f470c06a750 2024-04-26T01:58:11.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.382+0000 7f4703400700 1 -- 172.21.15.52:0/2957487252 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11886 (secure 0 0 0) 0x7f46fc09dae0 con 0x7f470c06a750 2024-04-26T01:58:11.381 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:58:11.381 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:58:08.052654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26243":{"gid":26243,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/4039499547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":4039499547},{"type":"v1","addr":"172.21.15.183:6837","nonce":4039499547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 msgr2=0x7f46f00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 0x7f46f00a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f470c078ef0 tx=0x7f46f400b040 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 msgr2=0x7f470c101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 0x7f470c101a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f46fc006010 tx=0x7f46fc0057c0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 shutdown_connections 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f470c108a30 0x7f470c102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f470c06a750 0x7f470c101a10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 --2- 172.21.15.52:0/2957487252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f470c06b0f0 0x7f470c101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 >> 172.21.15.52:0/2957487252 conn(0x7f470c0788f0 msgr2=0x7f470c0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 shutdown_connections 2024-04-26T01:58:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:11.385+0000 7f4711200700 1 -- 172.21.15.52:0/2957487252 wait complete. 2024-04-26T01:58:11.385 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 53 2024-04-26T01:58:11.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:11 smithi191 ceph-mon[32745]: pgmap v414: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:11.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:11 smithi183 ceph-mon[32752]: pgmap v414: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:11.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:11 smithi052 ceph-mon[27851]: pgmap v414: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 208 KiB/s wr, 40 op/s 2024-04-26T01:58:12.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.08 MB/sec execute 571 sec latency 23957.382 ms 2024-04-26T01:58:12.269 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id bde8d999 --release --await' 2024-04-26T01:58:12.269 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id bde8d999 --release --await 2024-04-26T01:58:12.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:12 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2957487252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:12.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:12 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2957487252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:12.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:12 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2957487252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:58:12.747 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 -- 172.21.15.52:0/3549321076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a8105f80 msgr2=0x7fb4a810db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 --2- 172.21.15.52:0/3549321076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a8105f80 0x7fb4a810db50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009ca0 tx=0x7fb4a002a8c0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 -- 172.21.15.52:0/3549321076 shutdown_connections 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 --2- 172.21.15.52:0/3549321076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a8105f80 0x7fb4a810db50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 --2- 172.21.15.52:0/3549321076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8101860 0x7fb4a81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 --2- 172.21.15.52:0/3549321076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 -- 172.21.15.52:0/3549321076 >> 172.21.15.52:0/3549321076 conn(0x7fb4a80fc770 msgr2=0x7fb4a80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 -- 172.21.15.52:0/3549321076 shutdown_connections 2024-04-26T01:58:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.004+0000 7fb4ad400640 1 -- 172.21.15.52:0/3549321076 wait complete. 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.005+0000 7fb4ad400640 1 Processor -- start 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.005+0000 7fb4ad400640 1 -- start start 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 0x7fb4a8102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a81033c0 0x7fb4a81a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a8110f30 con 0x7fb4a81033c0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a8110db0 con 0x7fb4a8105f80 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4ad400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81110b0 con 0x7fb4a8100eb0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43810/0 (socket says 172.21.15.52:43810) 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 0x7fb4a8102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4a7e00640 1 -- 172.21.15.52:0/2394265729 learned_addr learned my addr 172.21.15.52:0/2394265729 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.006+0000 7fb4aca00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a81033c0 0x7fb4a81a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 -- 172.21.15.52:0/2394265729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 msgr2=0x7fb4a8102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 0x7fb4a8102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 -- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a81033c0 msgr2=0x7fb4a81a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a81033c0 0x7fb4a81a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 -- 172.21.15.52:0/2394265729 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a81a6f30 con 0x7fb4a8100eb0 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 0x7fb4a8102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb49800e9f0 tx=0x7fb49800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a4c00640 1 -- 172.21.15.52:0/2394265729 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49800cd80 con 0x7fb4a8100eb0 2024-04-26T01:58:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a81a7220 con 0x7fb4a8100eb0 2024-04-26T01:58:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a4c00640 1 -- 172.21.15.52:0/2394265729 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4980108e0 con 0x7fb4a8100eb0 2024-04-26T01:58:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4a4c00640 1 -- 172.21.15.52:0/2394265729 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498018980 con 0x7fb4a8100eb0 2024-04-26T01:58:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.007+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a81a7770 con 0x7fb4a8100eb0 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.010+0000 7fb4b178a740 1 Processor -- start 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 -- start start 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4a4c00640 1 -- 172.21.15.52:0/2394265729 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb498010430 con 0x7fb4a8100eb0 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 0x5647f403cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4047650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5647f3fa2900 con 0x5647f403c9b0 2024-04-26T01:58:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5647f3f48ef0 con 0x5647f403d2d0 2024-04-26T01:58:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4b178a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5647f3f9e8c0 con 0x5647f4047b90 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4aca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 0x5647f403cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4047650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43822/0 (socket says 172.21.15.52:43822) 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.011+0000 7fb4a7400640 1 -- 172.21.15.52:0/1907062610 learned_addr learned my addr 172.21.15.52:0/1907062610 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 -- 172.21.15.52:0/1907062610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 msgr2=0x5647f4047650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4047650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 -- 172.21.15.52:0/1907062610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 msgr2=0x5647f403cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 0x5647f403cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 -- 172.21.15.52:0/1907062610 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647f3cb5210 con 0x5647f4047b90 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4aca00640 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 0x5647f403cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb494009fd0 tx=0x7fb4940118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7052d2fad9e92904 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb49d200640 1 -- 172.21.15.52:0/1907062610 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494004310 con 0x5647f4047b90 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb49d200640 1 -- 172.21.15.52:0/1907062610 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4940044b0 con 0x5647f4047b90 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb49d200640 1 -- 172.21.15.52:0/1907062610 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4940035f0 con 0x5647f4047b90 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.012+0000 7fb4a4c00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 0x7fb48c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4a4c00640 1 -- 172.21.15.52:0/2394265729 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8565+0+0 (secure 0 0 0) 0x7fb4980d0b70 con 0x7fb4a8100eb0 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4aca00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 0x7fb48c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 -- 172.21.15.52:0/1907062610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 msgr2=0x5647f4049f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb494009fd0 tx=0x7fb4940118d0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 -- 172.21.15.52:0/1907062610 shutdown_connections 2024-04-26T01:58:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f4047b90 0x5647f4049f80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4047650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1907062610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403c9b0 0x5647f403cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 -- 172.21.15.52:0/1907062610 >> 172.21.15.52:0/1907062610 conn(0x5647f4038ec0 msgr2=0x5647f40392a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 -- 172.21.15.52:0/1907062610 shutdown_connections 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.013+0000 7fb4b178a740 1 -- 172.21.15.52:0/1907062610 wait complete. 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.014+0000 7fb4aca00640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 0x7fb48c0a61d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0006010 tx=0x7fb4a0005d80 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.014+0000 7fb4b178a740 1 -- start start 2024-04-26T01:58:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.014+0000 7fb4b178a740 1 Processor -- start 2024-04-26T01:58:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4044fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f4045520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5647f404a7c0 con 0x5647f4047b90 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5647f3f5c310 con 0x5647f403d2d0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5647f404a940 con 0x5647f403c9b0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4b178a740 10 client.0 _ll_register_callbacks cb 0x5647f3f2acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4044fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4aca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f4045520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4044fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:47216/0 (socket says 172.21.15.52:47216) 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43834/0 (socket says 172.21.15.52:43834) 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7400640 1 -- 172.21.15.52:0/3405723824 learned_addr learned my addr 172.21.15.52:0/3405723824 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 msgr2=0x5647f4044fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4044fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3405723824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 msgr2=0x5647f4045520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f4045520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.015+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3405723824 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647f3d81620 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb4980ce230 tx=0x7fb498018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb6f02411c414af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49809cb30 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8565+0+0 (secure 0 0 0) 0x7fb4980ce4b0 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4940044b0 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb4980ce6b0 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb4b178a740 10 client.26713 fetch_fsmap learned FSMap version 53 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5647f40569a0 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4940044b0 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.016+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7fb498018c40 con 0x5647f403c9b0 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 10 client.26713 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 10 client.26713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 20 client.26713 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f4057e60 0x5647f405a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 4 client.26713 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647f3fb68b0 con 0x5647f4057e60 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.017+0000 7fb4a7400640 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f4057e60 0x5647f405a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.018+0000 7fb4a7400640 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f4057e60 0x5647f405a2b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9c63b9763d0fbc server_cookie=a9dcf624abd68f7a in_seq=0 out_seq=0 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.018+0000 7fb491600640 10 client.26713 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.018+0000 7fb491600640 1 -- 172.21.15.52:0/3405723824 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647f3fb68b0 con 0x5647f4057e60 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.018+0000 7fb491600640 10 client.26713 handle_command_reply: tid=0 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.018+0000 7fb4b178a740 1 client.26713 shutdown 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 msgr2=0x5647f4044aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb4980ce230 tx=0x7fb498018420 comp rx=0 tx=0).stop 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 shutdown_connections 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f4057e60 0x5647f405a2b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f4045520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403d2d0 0x5647f4044fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3405723824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4044aa0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb491600640 0 client.26713 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 >> 172.21.15.52:0/3405723824 conn(0x5647f4038ec0 msgr2=0x5647f403ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 shutdown_connections 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 1 -- 172.21.15.52:0/3405723824 wait complete. 2024-04-26T01:58:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.019+0000 7fb4b178a740 20 client.26713 trim_cache size 0 max 16384 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.035+0000 7fb4b178a740 1 Processor -- start 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.035+0000 7fb4b178a740 1 -- start start 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 0x5647f403f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 0x5647f40387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81b25c0 con 0x5647f4047b90 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81afe60 con 0x5647f403c9b0 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4b178a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81adcd0 con 0x5647f403d2d0 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4aca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 0x5647f403f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4aca00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59788/0 (socket says 172.21.15.52:59788) 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4aca00640 1 -- 172.21.15.52:0/3929057552 learned_addr learned my addr 172.21.15.52:0/3929057552 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7400640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 0x5647f40387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3929057552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 msgr2=0x5647f40387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 0x5647f40387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3929057552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 msgr2=0x5647f405fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7e00640 1 -- 172.21.15.52:0/3929057552 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647f4045a00 con 0x5647f403c9b0 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4a7400640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 0x5647f40387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.036+0000 7fb4aca00640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.037+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 0x5647f403f370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb4980d43c0 tx=0x7fb49809b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98d89016005750af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.037+0000 7fb493e00640 1 -- 172.21.15.52:0/3929057552 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49809c490 con 0x5647f403c9b0 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.037+0000 7fb493e00640 1 -- 172.21.15.52:0/3929057552 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4980a56f0 con 0x5647f403c9b0 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.037+0000 7fb493e00640 1 -- 172.21.15.52:0/3929057552 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498018d80 con 0x5647f403c9b0 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- 172.21.15.52:0/3929057552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 msgr2=0x5647f403f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 0x5647f403f370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb4980d43c0 tx=0x7fb49809b040 comp rx=0 tx=0).stop 2024-04-26T01:58:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- 172.21.15.52:0/3929057552 shutdown_connections 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f4047b90 0x5647f405fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403d2d0 0x5647f40387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 --2- 172.21.15.52:0/3929057552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f403c9b0 0x5647f403f370 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- 172.21.15.52:0/3929057552 >> 172.21.15.52:0/3929057552 conn(0x5647f4038ec0 msgr2=0x5647f40452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- 172.21.15.52:0/3929057552 shutdown_connections 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- 172.21.15.52:0/3929057552 wait complete. 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 -- start start 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.038+0000 7fb4b178a740 1 Processor -- start 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4050090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f4047b90 0x5647f404db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81ae320 con 0x5647f403d2d0 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81af790 con 0x5647f4047b90 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a81ae9f0 con 0x5647f403c9b0 2024-04-26T01:58:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4b178a740 10 client.0 _ll_register_callbacks cb 0x7fb498098dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4050090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4a7400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59790/0 (socket says 172.21.15.52:59790) 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4a7400640 1 -- 172.21.15.52:0/1982801734 learned_addr learned my addr 172.21.15.52:0/1982801734 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.039+0000 7fb4aca00640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f4047b90 0x5647f404db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 -- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f4047b90 msgr2=0x5647f404db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f4047b90 0x5647f404db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 -- 172.21.15.52:0/1982801734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 msgr2=0x5647f404d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 -- 172.21.15.52:0/1982801734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647f40bd440 con 0x5647f403c9b0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4a7e00640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4050090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5647f4044960 tx=0x7fb49800cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ff6d9d50ed4a140 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb491600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49809c490 con 0x5647f403c9b0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8565+0+0 (secure 0 0 0) 0x7fb49809bf80 con 0x5647f403c9b0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.040+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4980a56f0 con 0x5647f403c9b0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb4980bd0b0 con 0x5647f403c9b0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 10 client.26719 fetch_fsmap learned FSMap version 53 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5647f40be580 con 0x5647f403c9b0 2024-04-26T01:58:13.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4980a56f0 con 0x5647f403c9b0 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7fb4980a2150 con 0x5647f403c9b0 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 10 client.26719 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 10 client.26719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 20 client.26719 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f40bebe0 0x5647f40c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 4 client.26719 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.041+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5647f3fb0320 con 0x5647f40bebe0 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.042+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f40bebe0 0x5647f40c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.042+0000 7fb4a7400640 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f40bebe0 0x5647f40c0fd0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5768e139efbd376 server_cookie=98fcd6381ab6fca8 in_seq=0 out_seq=0 2024-04-26T01:58:13.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.042+0000 7fb491600640 10 client.26719 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.063+0000 7fb491600640 1 -- 172.21.15.52:0/1982801734 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5647f3fb0320 con 0x5647f40bebe0 2024-04-26T01:58:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.063+0000 7fb491600640 10 client.26719 handle_command_reply: tid=0 2024-04-26T01:58:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.063+0000 7fb4b178a740 1 client.26719 shutdown 2024-04-26T01:58:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.063+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 msgr2=0x5647f4050090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.063+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4050090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5647f4044960 tx=0x7fb49800cf20 comp rx=0 tx=0).stop 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 shutdown_connections 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5647f40bebe0 0x5647f40c0fd0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5647f4047b90 0x5647f404db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5647f403d2d0 0x5647f404d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 --2- 172.21.15.52:0/1982801734 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5647f403c9b0 0x5647f4050090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 >> 172.21.15.52:0/1982801734 conn(0x5647f4038ec0 msgr2=0x5647f40452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb491600640 0 client.26719 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 shutdown_connections 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 1 -- 172.21.15.52:0/1982801734 wait complete. 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.064+0000 7fb4b178a740 20 client.26719 trim_cache size 0 max 16384 2024-04-26T01:58:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 msgr2=0x7fb48c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 0x7fb48c0a61d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0006010 tx=0x7fb4a0005d80 comp rx=0 tx=0).stop 2024-04-26T01:58:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 msgr2=0x7fb4a8102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb49800e9f0 tx=0x7fb49800eeb0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 shutdown_connections 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb48c0a3d10 0x7fb48c0a61d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb4a81033c0 0x7fb4a81a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4a8105f80 0x7fb4a8102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 --2- 172.21.15.52:0/2394265729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4a8100eb0 0x7fb4a8102940 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 >> 172.21.15.52:0/2394265729 conn(0x7fb4a80fc770 msgr2=0x7fb4a8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 shutdown_connections 2024-04-26T01:58:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:13.065+0000 7fb4ad400640 1 -- 172.21.15.52:0/2394265729 wait complete. 2024-04-26T01:58:13.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.07 MB/sec execute 572 sec latency 24957.522 ms 2024-04-26T01:58:13.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:13 smithi191 ceph-mon[32745]: pgmap v415: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 125 KiB/s wr, 9 op/s 2024-04-26T01:58:13.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:13 smithi191 ceph-mon[32745]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:58:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:13 smithi183 ceph-mon[32752]: pgmap v415: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 125 KiB/s wr, 9 op/s 2024-04-26T01:58:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:13 smithi183 ceph-mon[32752]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:58:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:13 smithi052 ceph-mon[27851]: pgmap v415: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 125 KiB/s wr, 9 op/s 2024-04-26T01:58:13.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:13 smithi052 ceph-mon[27851]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T01:58:14.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.07 MB/sec execute 573 sec latency 25957.688 ms 2024-04-26T01:58:15.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273358 2.06 MB/sec execute 574 sec latency 26957.784 ms 2024-04-26T01:58:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 -- 172.21.15.52:0/2115630740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c109fa0 msgr2=0x7ff78c111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 --2- 172.21.15.52:0/2115630740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c109fa0 0x7ff78c111b70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff78400b0a0 tx=0x7ff78400b3b0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 -- 172.21.15.52:0/2115630740 shutdown_connections 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 --2- 172.21.15.52:0/2115630740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c109fa0 0x7ff78c111b70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 --2- 172.21.15.52:0/2115630740 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 --2- 172.21.15.52:0/2115630740 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c105870 0x7ff78c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.171+0000 7ff793600700 1 -- 172.21.15.52:0/2115630740 >> 172.21.15.52:0/2115630740 conn(0x7ff78c100750 msgr2=0x7ff78c102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.172+0000 7ff793600700 1 -- 172.21.15.52:0/2115630740 shutdown_connections 2024-04-26T01:58:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.172+0000 7ff793600700 1 -- 172.21.15.52:0/2115630740 wait complete. 2024-04-26T01:58:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 Processor -- start 2024-04-26T01:58:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 -- start start 2024-04-26T01:58:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c109fa0 0x7ff78c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c114e30 con 0x7ff78c104ec0 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c114cb0 con 0x7ff78c105870 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.173+0000 7ff793600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c114fb0 con 0x7ff78c109fa0 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff78be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51130/0 (socket says 172.21.15.52:51130) 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff78be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35430/0 (socket says 172.21.15.52:35430) 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 -- 172.21.15.52:0/591249351 learned_addr learned my addr 172.21.15.52:0/591249351 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff791600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c109fa0 0x7ff78c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 -- 172.21.15.52:0/591249351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c109fa0 msgr2=0x7ff78c19fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c109fa0 0x7ff78c19fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 -- 172.21.15.52:0/591249351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 msgr2=0x7ff78c19b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff790c00700 1 -- 172.21.15.52:0/591249351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c1a0270 con 0x7ff78c104ec0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.174+0000 7ff78be00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff790c00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff78000d950 tx=0x7ff78000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff789600700 1 -- 172.21.15.52:0/591249351 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780016070 con 0x7ff78c104ec0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff789600700 1 -- 172.21.15.52:0/591249351 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff78000fd40 con 0x7ff78c104ec0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c1a0560 con 0x7ff78c104ec0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff789600700 1 -- 172.21.15.52:0/591249351 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78000e3b0 con 0x7ff78c104ec0 2024-04-26T01:58:15.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.175+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c1a7de0 con 0x7ff78c104ec0 2024-04-26T01:58:15.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.178+0000 7ff789600700 1 -- 172.21.15.52:0/591249351 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff780019070 con 0x7ff78c104ec0 2024-04-26T01:58:15.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.180+0000 7ff789600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 0x7ff7740a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.180+0000 7ff79ded8b80 1 Processor -- start 2024-04-26T01:58:15.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.180+0000 7ff789600700 1 -- 172.21.15.52:0/591249351 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff7800d0320 con 0x7ff78c104ec0 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.180+0000 7ff79ded8b80 1 -- start start 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.180+0000 7ff78be00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 0x7ff7740a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eeddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eeeb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c72ee30f90 con 0x55c72eee90f0 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c72edd7830 con 0x55c72eede240 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c72ee2cf50 con 0x55c72eedd920 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff791600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eeeb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff791600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eeeb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51132/0 (socket says 172.21.15.52:51132) 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff791600700 1 -- 172.21.15.52:0/1697978444 learned_addr learned my addr 172.21.15.52:0/1697978444 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff78be00700 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.181+0000 7ff790c00700 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eeddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff78be00700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 0x7ff7740a6270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff78c10b4f0 tx=0x7ff778009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 -- 172.21.15.52:0/1697978444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 msgr2=0x55c72eee8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 -- 172.21.15.52:0/1697978444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 msgr2=0x55c72eeeb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eeeb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 -- 172.21.15.52:0/1697978444 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72edcd780 con 0x55c72eedd920 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff790c00700 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eeddd00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7800cda10 tx=0x7ff78009c260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=805c395ccc053972 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff77e000700 1 -- 172.21.15.52:0/1697978444 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780020040 con 0x55c72eedd920 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff77e000700 1 -- 172.21.15.52:0/1697978444 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7800a4ed0 con 0x55c72eedd920 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.182+0000 7ff77e000700 1 -- 172.21.15.52:0/1697978444 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800a4c40 con 0x55c72eedd920 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1697978444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 msgr2=0x55c72eeddd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eeddd00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7800cda10 tx=0x7ff78009c260 comp rx=0 tx=0).stop 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1697978444 shutdown_connections 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eeeb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eeddd00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1697978444 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1697978444 >> 172.21.15.52:0/1697978444 conn(0x55c72ee0c6a0 msgr2=0x55c72eed9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1697978444 shutdown_connections 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.183+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1697978444 wait complete. 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.184+0000 7ff79ded8b80 1 -- start start 2024-04-26T01:58:15.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.184+0000 7ff79ded8b80 1 Processor -- start 2024-04-26T01:58:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 0x55c72eee6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c72eeebb50 con 0x55c72eedd920 2024-04-26T01:58:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c72edea960 con 0x55c72eede240 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c72eeebcd0 con 0x55c72eee90f0 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff79ded8b80 10 client.0 _ll_register_callbacks cb 0x55c72edc9820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff790c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff790c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51144/0 (socket says 172.21.15.52:51144) 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff790c00700 1 -- 172.21.15.52:0/1509225665 learned_addr learned my addr 172.21.15.52:0/1509225665 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff791600700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 0x55c72eee6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.185+0000 7ff78be00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 -- 172.21.15.52:0/1509225665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 msgr2=0x55c72eee6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 0x55c72eee6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 -- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 msgr2=0x55c72eee5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 -- 172.21.15.52:0/1509225665 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72ed7e9d0 con 0x55c72eedd920 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff791600700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 0x55c72eee6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff78be00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff790c00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff7800a49a0 tx=0x7ff780000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85439f41200ca007 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff772a00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800ba060 con 0x55c72eedd920 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff78009c6d0 con 0x55c72eedd920 2024-04-26T01:58:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.186+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7800a4ed0 con 0x55c72eedd920 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ff7800b4030 con 0x55c72eedd920 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 10 client.16593 fetch_fsmap learned FSMap version 53 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55c72edc7bf0 con 0x55c72eedd920 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7800a4ed0 con 0x55c72eedd920 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7ff78009b6b0 con 0x55c72eedd920 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 10 client.16593 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 10 client.16593 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:58:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 20 client.16593 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:15.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72eef86c0 0x55c72eefaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 4 client.16593 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:15.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.187+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c72ee3e780 con 0x55c72eef86c0 2024-04-26T01:58:15.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.188+0000 7ff78be00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72eef86c0 0x55c72eefaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.188+0000 7ff78be00700 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72eef86c0 0x55c72eefaab0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebf1b030e10a8a5 server_cookie=f6633b26fff7dd24 in_seq=0 out_seq=0 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.188+0000 7ff772a00700 10 client.16593 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.189+0000 7ff772a00700 1 -- 172.21.15.52:0/1509225665 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c72ee3e780 con 0x55c72eef86c0 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.189+0000 7ff772a00700 10 client.16593 handle_command_reply: tid=0 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 client.16593 shutdown 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 msgr2=0x55c72eee8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff7800a49a0 tx=0x7ff780000bc0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 shutdown_connections 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72eef86c0 0x55c72eefaab0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eee90f0 0x55c72eee6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eedd920 0x55c72eee8830 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/1509225665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eee5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 >> 172.21.15.52:0/1509225665 conn(0x55c72ee0c6a0 msgr2=0x55c72eed99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff772a00700 0 client.16593 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:15.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 shutdown_connections 2024-04-26T01:58:15.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 1 -- 172.21.15.52:0/1509225665 wait complete. 2024-04-26T01:58:15.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.190+0000 7ff79ded8b80 20 client.16593 trim_cache size 0 max 16384 2024-04-26T01:58:15.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.212+0000 7ff79ded8b80 1 Processor -- start 2024-04-26T01:58:15.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.212+0000 7ff79ded8b80 1 -- start start 2024-04-26T01:58:15.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eedd920 0x55c72eee6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 0x55c72eefd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1b1e30 con 0x55c72eee90f0 2024-04-26T01:58:15.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1af690 con 0x55c72eedd920 2024-04-26T01:58:15.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1ad500 con 0x55c72eede240 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff791600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff78be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 0x55c72eefd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff791600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51146/0 (socket says 172.21.15.52:51146) 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff791600700 1 -- 172.21.15.52:0/2815963229 learned_addr learned my addr 172.21.15.52:0/2815963229 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.213+0000 7ff790c00700 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eedd920 0x55c72eee6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 -- 172.21.15.52:0/2815963229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eedd920 msgr2=0x55c72eee6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eedd920 0x55c72eee6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 -- 172.21.15.52:0/2815963229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 msgr2=0x55c72eefdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 -- 172.21.15.52:0/2815963229 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72eede870 con 0x55c72eede240 2024-04-26T01:58:15.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff78be00700 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 0x55c72eefd6f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x55c72eed8c20 tx=0x7ff77801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab43628afd068f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff77d600700 1 -- 172.21.15.52:0/2815963229 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778032070 con 0x55c72eede240 2024-04-26T01:58:15.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff77d600700 1 -- 172.21.15.52:0/2815963229 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff778003dc0 con 0x55c72eede240 2024-04-26T01:58:15.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.214+0000 7ff77d600700 1 -- 172.21.15.52:0/2815963229 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778027880 con 0x55c72eede240 2024-04-26T01:58:15.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 -- 172.21.15.52:0/2815963229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 msgr2=0x55c72eefd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 0x55c72eefd6f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x55c72eed8c20 tx=0x7ff77801c000 comp rx=0 tx=0).stop 2024-04-26T01:58:15.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 -- 172.21.15.52:0/2815963229 shutdown_connections 2024-04-26T01:58:15.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eedd920 0x55c72eee6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/2815963229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eede240 0x55c72eefd6f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 -- 172.21.15.52:0/2815963229 >> 172.21.15.52:0/2815963229 conn(0x55c72ee0c6a0 msgr2=0x55c72eee5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 -- 172.21.15.52:0/2815963229 shutdown_connections 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.215+0000 7ff79ded8b80 1 -- 172.21.15.52:0/2815963229 wait complete. 2024-04-26T01:58:15.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 -- start start 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 Processor -- start 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eefd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1adb50 con 0x55c72eee90f0 2024-04-26T01:58:15.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1aefc0 con 0x55c72eede240 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.216+0000 7ff79ded8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1ae220 con 0x55c72eedd920 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff79ded8b80 10 client.0 _ll_register_callbacks cb 0x55c72edc9820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff791600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35456/0 (socket says 172.21.15.52:35456) 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 -- 172.21.15.52:0/3015666634 learned_addr learned my addr 172.21.15.52:0/3015666634 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff790c00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eefd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 -- 172.21.15.52:0/3015666634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 msgr2=0x55c72eefd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eefd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 -- 172.21.15.52:0/3015666634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 msgr2=0x55c72eefa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:15.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.217+0000 7ff78be00700 1 -- 172.21.15.52:0/3015666634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72eefa7f0 con 0x55c72eede240 2024-04-26T01:58:15.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff78be00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff778005720 tx=0x7ff778005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ce5aff5c38cc6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:15.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff772a00700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:58:15.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77803b060 con 0x55c72eede240 2024-04-26T01:58:15.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff77802c4d0 con 0x55c72eede240 2024-04-26T01:58:15.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff778003dc0 con 0x55c72eede240 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7ff778032070 con 0x55c72eede240 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff79ded8b80 10 client.26729 fetch_fsmap learned FSMap version 53 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.218+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55c72ef5f760 con 0x55c72eede240 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff778003dc0 con 0x55c72eede240 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7ff77801a040 con 0x55c72eede240 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 10 client.26729 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 10 client.26729 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 20 client.26729 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72ef5fe80 0x55c72ef622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 4 client.26729 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bde8d999", "release": true, "await": true}] 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bde8d999", "release": true, "await": true}) -- 0x55c72ee38820 con 0x55c72ef5fe80 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.219+0000 7ff790c00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72ef5fe80 0x55c72ef622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:15.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.220+0000 7ff790c00700 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72ef5fe80 0x55c72ef622a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9cd50c5e5b20ab9 server_cookie=7cda89ad6ba15cf7 in_seq=0 out_seq=0 2024-04-26T01:58:15.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.220+0000 7ff772a00700 10 client.26729 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:15.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff772a00700 1 -- 172.21.15.52:0/3015666634 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c72ee38820 con 0x55c72ef5fe80 2024-04-26T01:58:15.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff772a00700 10 client.26729 handle_command_reply: tid=0 2024-04-26T01:58:15.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 client.26729 shutdown 2024-04-26T01:58:15.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 msgr2=0x55c72eef9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff778005720 tx=0x7ff778005750 comp rx=0 tx=0).stop 2024-04-26T01:58:15.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 shutdown_connections 2024-04-26T01:58:15.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c72ef5fe80 0x55c72ef622a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c72eee90f0 0x55c72eefa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c72eedd920 0x55c72eefd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 --2- 172.21.15.52:0/3015666634 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c72eede240 0x55c72eef9c70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 >> 172.21.15.52:0/3015666634 conn(0x55c72ee0c6a0 msgr2=0x55c72eee7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff772a00700 0 client.26729 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.225+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 shutdown_connections 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.226+0000 7ff79ded8b80 1 -- 172.21.15.52:0/3015666634 wait complete. 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.226+0000 7ff79ded8b80 20 client.26729 trim_cache size 0 max 16384 2024-04-26T01:58:15.225 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 53, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 40, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "bde8d999": { 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 40, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:58:15.226 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 79.2, 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:58:15.227 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:58:15.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.226+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 msgr2=0x7ff7740a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 0x7ff7740a6270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff78c10b4f0 tx=0x7ff778009260 comp rx=0 tx=0).stop 2024-04-26T01:58:15.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 msgr2=0x7ff78c19b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:15.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff78000d950 tx=0x7ff78000dc60 comp rx=0 tx=0).stop 2024-04-26T01:58:15.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 shutdown_connections 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7740a3db0 0x7ff7740a6270 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff78c109fa0 0x7ff78c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff78c104ec0 0x7ff78c19b260 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 --2- 172.21.15.52:0/591249351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff78c105870 0x7ff78c19b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 >> 172.21.15.52:0/591249351 conn(0x7ff78c100750 msgr2=0x7ff78c102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:15.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 shutdown_connections 2024-04-26T01:58:15.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:58:15.227+0000 7ff793600700 1 -- 172.21.15.52:0/591249351 wait complete. 2024-04-26T01:58:15.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:15 smithi052 ceph-mon[27851]: pgmap v417: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-04-26T01:58:15.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:15 smithi191 ceph-mon[32745]: pgmap v417: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-04-26T01:58:15.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:15 smithi183 ceph-mon[32752]: pgmap v417: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 4 op/s 2024-04-26T01:58:16.151 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273612 2.06 MB/sec execute 575 sec latency 27041.915 ms 2024-04-26T01:58:16.165 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bde8d999', seconds elapsed: 34.6 2024-04-26T01:58:17.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273771 2.06 MB/sec execute 576 sec latency 614.610 ms 2024-04-26T01:58:17.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:17 smithi191 ceph-mon[32745]: pgmap v418: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 101 KiB/s wr, 20 op/s 2024-04-26T01:58:17.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:17 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:17 smithi183 ceph-mon[32752]: pgmap v418: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 101 KiB/s wr, 20 op/s 2024-04-26T01:58:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:17 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:17 smithi052 ceph-mon[27851]: pgmap v418: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 101 KiB/s wr, 20 op/s 2024-04-26T01:58:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:17 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:18.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273771 2.05 MB/sec execute 577 sec latency 1614.742 ms 2024-04-26T01:58:19.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273771 2.05 MB/sec execute 578 sec latency 2614.911 ms 2024-04-26T01:58:19.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:19 smithi191 ceph-mon[32745]: pgmap v419: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:19 smithi183 ceph-mon[32752]: pgmap v419: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:19 smithi052 ceph-mon[27851]: pgmap v419: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:20.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273771 2.05 MB/sec execute 579 sec latency 3615.060 ms 2024-04-26T01:58:21.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273819 2.04 MB/sec execute 580 sec latency 4462.188 ms 2024-04-26T01:58:21.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:21 smithi191 ceph-mon[32745]: pgmap v420: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:21 smithi183 ceph-mon[32752]: pgmap v420: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:21 smithi052 ceph-mon[27851]: pgmap v420: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 19 op/s 2024-04-26T01:58:22.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273874 2.04 MB/sec execute 581 sec latency 927.101 ms 2024-04-26T01:58:23.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 273942 2.04 MB/sec execute 582 sec latency 1628.623 ms 2024-04-26T01:58:23.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:23 smithi191 ceph-mon[32745]: pgmap v421: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-04-26T01:58:23.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:23 smithi183 ceph-mon[32752]: pgmap v421: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-04-26T01:58:23.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:23 smithi052 ceph-mon[27851]: pgmap v421: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-04-26T01:58:24.152 INFO:tasks.workunit.client.0.smithi052.stdout: 1 274224 2.04 MB/sec execute 583 sec latency 763.897 ms 2024-04-26T01:58:25.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 274843 2.04 MB/sec execute 584 sec latency 34.744 ms 2024-04-26T01:58:25.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:25 smithi191 ceph-mon[32745]: pgmap v422: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 120 KiB/s wr, 26 op/s 2024-04-26T01:58:25.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:25 smithi183 ceph-mon[32752]: pgmap v422: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 120 KiB/s wr, 26 op/s 2024-04-26T01:58:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:25 smithi052 ceph-mon[27851]: pgmap v422: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 120 KiB/s wr, 26 op/s 2024-04-26T01:58:26.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 275910 2.05 MB/sec execute 585 sec latency 36.646 ms 2024-04-26T01:58:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 -- 172.21.15.52:0/237769305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 msgr2=0x7f5ae010e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 --2- 172.21.15.52:0/237769305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae010e800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009900 tx=0x7f5ad002a410 comp rx=0 tx=0).stop 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 -- 172.21.15.52:0/237769305 shutdown_connections 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 --2- 172.21.15.52:0/237769305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108e10 0x7f5ae0115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 --2- 172.21.15.52:0/237769305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108490 0x7f5ae01088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.227+0000 7f5adfe00640 1 --2- 172.21.15.52:0/237769305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae010e800 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.228+0000 7f5adfe00640 1 -- 172.21.15.52:0/237769305 >> 172.21.15.52:0/237769305 conn(0x7f5ae0077d30 msgr2=0x7f5ae0106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.228+0000 7f5adfe00640 1 -- 172.21.15.52:0/237769305 shutdown_connections 2024-04-26T01:58:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.228+0000 7f5adfe00640 1 -- 172.21.15.52:0/237769305 wait complete. 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 Processor -- start 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 -- start start 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108490 0x7f5ae01a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01185f0 con 0x7f5ae0108490 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae0118470 con 0x7f5ae010e420 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.229+0000 7f5adfe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae0118770 con 0x7f5ae0108e10 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5adea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108490 0x7f5ae01a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33818/0 (socket says 172.21.15.52:33818) 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 -- 172.21.15.52:0/1624702169 learned_addr learned my addr 172.21.15.52:0/1624702169 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5adf400640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 -- 172.21.15.52:0/1624702169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 msgr2=0x7f5ae019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 -- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108490 msgr2=0x7f5ae01a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108490 0x7f5ae01a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 -- 172.21.15.52:0/1624702169 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae019f130 con 0x7f5ae0108e10 2024-04-26T01:58:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.230+0000 7f5ade000640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00e9f0 tx=0x7f5acc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.231+0000 7f5ae4a00640 1 -- 172.21.15.52:0/1624702169 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00ccc0 con 0x7f5ae0108e10 2024-04-26T01:58:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.231+0000 7f5ae4a00640 1 -- 172.21.15.52:0/1624702169 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5acc00ce60 con 0x7f5ae0108e10 2024-04-26T01:58:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.231+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae019f420 con 0x7f5ae0108e10 2024-04-26T01:58:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.231+0000 7f5ae4a00640 1 -- 172.21.15.52:0/1624702169 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc010430 con 0x7f5ae0108e10 2024-04-26T01:58:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.231+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae01ab6c0 con 0x7f5ae0108e10 2024-04-26T01:58:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.235+0000 7f5ae4a00640 1 -- 172.21.15.52:0/1624702169 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5acc0046c0 con 0x7f5ae0108e10 2024-04-26T01:58:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.235+0000 7f5ae8bfe740 1 Processor -- start 2024-04-26T01:58:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.235+0000 7f5ae8bfe740 1 -- start start 2024-04-26T01:58:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0039b0 0x560bfe003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe00e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe010f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560bfdf69900 con 0x560bfe0039b0 2024-04-26T01:58:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560bfdf0fef0 con 0x560bfe0042d0 2024-04-26T01:58:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560bfdf658c0 con 0x560bfe00eb90 2024-04-26T01:58:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe010f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5adea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe00e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe010f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33826/0 (socket says 172.21.15.52:33826) 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5ade000640 1 -- 172.21.15.52:0/3926202048 learned_addr learned my addr 172.21.15.52:0/3926202048 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.236+0000 7f5adf400640 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0039b0 0x560bfe003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 -- 172.21.15.52:0/3926202048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 msgr2=0x560bfe010f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe010f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 -- 172.21.15.52:0/3926202048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0039b0 msgr2=0x560bfe003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0039b0 0x560bfe003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 -- 172.21.15.52:0/3926202048 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfdc7c210 con 0x560bfe0042d0 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5ae4a00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adea00640 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe00e650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0005940 tx=0x7f5ad0005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b97d30beae3ec7d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5ae4a00640 1 -- 172.21.15.52:0/1624702169 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f5acc0d0c20 con 0x7f5ae0108e10 2024-04-26T01:58:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.237+0000 7f5adf400640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.238+0000 7f5acbe00640 1 -- 172.21.15.52:0/3926202048 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0040070 con 0x560bfe0042d0 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.238+0000 7f5acbe00640 1 -- 172.21.15.52:0/3926202048 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad0005020 con 0x560bfe0042d0 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.238+0000 7f5acbe00640 1 -- 172.21.15.52:0/3926202048 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0035040 con 0x560bfe0042d0 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3926202048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 msgr2=0x560bfe00e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5adf400640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8005fd0 tx=0x7f5ad8009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe00e650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0005940 tx=0x7f5ad0005bd0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3926202048 shutdown_connections 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe010f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe00e650 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3926202048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0039b0 0x560bfe003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3926202048 >> 172.21.15.52:0/3926202048 conn(0x560bfdfffec0 msgr2=0x560bfe0002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3926202048 shutdown_connections 2024-04-26T01:58:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3926202048 wait complete. 2024-04-26T01:58:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.239+0000 7f5ae8bfe740 1 -- start start 2024-04-26T01:58:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 Processor -- start 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe00bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe00c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560bfe011760 con 0x560bfe00eb90 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560bfdf23310 con 0x560bfe0039b0 2024-04-26T01:58:26.242 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560bfe0118e0 con 0x560bfe0042d0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ae8bfe740 10 client.0 _ll_register_callbacks cb 0x560bfdef1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5adea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe00bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ade000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33838/0 (socket says 172.21.15.52:33838) 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5ade000640 1 -- 172.21.15.52:0/461759131 learned_addr learned my addr 172.21.15.52:0/461759131 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.240+0000 7f5adea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe00bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45162/0 (socket says 172.21.15.52:45162) 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5adf400640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe00c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 -- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 msgr2=0x560bfe00bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe00bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 -- 172.21.15.52:0/461759131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 msgr2=0x560bfe00c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe00c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 -- 172.21.15.52:0/461759131 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfdd48620 con 0x560bfe0042d0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ade000640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5acc099530 tx=0x7f5acc005680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c545de8f4718dd2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ac8c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:58:26.243 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc09c6a0 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f5acc0a5340 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.241+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad0005020 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5acc09c4c0 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 10 client.26749 fetch_fsmap learned FSMap version 54 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560bfe01da00 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad0005020 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f5acc061ba0 con 0x560bfe0042d0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 10 client.26749 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 10 client.26749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 20 client.26749 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe01eec0 0x560bfe0212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 4 client.26749 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.242+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bfdf7d8b0 con 0x560bfe01eec0 2024-04-26T01:58:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.243+0000 7f5adea00640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe01eec0 0x560bfe0212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.243+0000 7f5adea00640 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe01eec0 0x560bfe0212b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bbe10db8791f287 server_cookie=fc9b5326807691e0 in_seq=0 out_seq=0 2024-04-26T01:58:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.243+0000 7f5ac8c00640 10 client.26749 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ac8c00640 1 -- 172.21.15.52:0/461759131 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560bfdf7d8b0 con 0x560bfe01eec0 2024-04-26T01:58:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ac8c00640 10 client.26749 handle_command_reply: tid=0 2024-04-26T01:58:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 client.26749 shutdown 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 msgr2=0x560bfe00c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5acc099530 tx=0x7f5acc005680 comp rx=0 tx=0).stop 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 shutdown_connections 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe01eec0 0x560bfe0212b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe00c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0042d0 0x560bfe00c040 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/461759131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe00bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ac8c00640 0 client.26749 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 >> 172.21.15.52:0/461759131 conn(0x560bfdfffec0 msgr2=0x560bfe005c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 shutdown_connections 2024-04-26T01:58:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.244+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/461759131 wait complete. 2024-04-26T01:58:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.245+0000 7f5ae8bfe740 20 client.26749 trim_cache size 0 max 16384 2024-04-26T01:58:26.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 Processor -- start 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 -- start start 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe01a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0042d0 0x560bfe026bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b68a0 con 0x560bfe0042d0 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b4110 con 0x560bfe0039b0 2024-04-26T01:58:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.264+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b1f80 con 0x560bfe00eb90 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe01a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33844/0 (socket says 172.21.15.52:33844) 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adf400640 1 -- 172.21.15.52:0/2432850666 learned_addr learned my addr 172.21.15.52:0/2432850666 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5ade000640 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0042d0 0x560bfe026bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 -- 172.21.15.52:0/2432850666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 msgr2=0x560bfe027100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 -- 172.21.15.52:0/2432850666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0042d0 msgr2=0x560bfe026bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0042d0 0x560bfe026bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 -- 172.21.15.52:0/2432850666 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfe01bab0 con 0x560bfe0039b0 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adf400640 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.265+0000 7f5adea00640 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe01a620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f9954cf18cdda2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.266+0000 7f5acb400640 1 -- 172.21.15.52:0/2432850666 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0040040 con 0x560bfe0039b0 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.266+0000 7f5acb400640 1 -- 172.21.15.52:0/2432850666 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad0031630 con 0x560bfe0039b0 2024-04-26T01:58:26.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.266+0000 7f5acb400640 1 -- 172.21.15.52:0/2432850666 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0035040 con 0x560bfe0039b0 2024-04-26T01:58:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/2432850666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 msgr2=0x560bfe01a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe01a620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad0000bc0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/2432850666 shutdown_connections 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe00eb90 0x560bfe027100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe0042d0 0x560bfe026bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/2432850666 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0039b0 0x560bfe01a620 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/2432850666 >> 172.21.15.52:0/2432850666 conn(0x560bfdfffec0 msgr2=0x560bfe006e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/2432850666 shutdown_connections 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/2432850666 wait complete. 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.267+0000 7f5ae8bfe740 1 -- start start 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 Processor -- start 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe014310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe014890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b25d0 con 0x560bfe00eb90 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b3a40 con 0x560bfe0042d0 2024-04-26T01:58:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae01b2ca0 con 0x560bfe0039b0 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ae8bfe740 10 client.0 _ll_register_callbacks cb 0x7f5acc0ce2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5adea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5adf400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe014890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5adea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33860/0 (socket says 172.21.15.52:33860) 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5adea00640 1 -- 172.21.15.52:0/3772820959 learned_addr learned my addr 172.21.15.52:0/3772820959 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.268+0000 7f5ade000640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe014310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 -- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 msgr2=0x560bfe014310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe014310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 -- 172.21.15.52:0/3772820959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 msgr2=0x560bfe014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe014890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 -- 172.21.15.52:0/3772820959 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfe0150c0 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5adea00640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009a20 tx=0x7f5ad00095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0dd1ad384b2694 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.269+0000 7f5ac8c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0035040 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f5ad0005030 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad0031630 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f5ad0040040 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ae8bfe740 10 client.26755 fetch_fsmap learned FSMap version 54 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560bfe0854f0 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.270+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad0031630 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f5ad0050020 con 0x560bfe0039b0 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 10 client.26755 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 10 client.26755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 20 client.26755 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:26.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe085890 0x560bfe087ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 4 client.26755 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bfdf77320 con 0x560bfe085890 2024-04-26T01:58:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.271+0000 7f5ade000640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe085890 0x560bfe087ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.272+0000 7f5ade000640 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe085890 0x560bfe087ca0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e039a4404fbc418 server_cookie=eede09d52b3a9db9 in_seq=0 out_seq=0 2024-04-26T01:58:26.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.272+0000 7f5ac8c00640 10 client.26755 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ac8c00640 1 -- 172.21.15.52:0/3772820959 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bfdf77320 con 0x560bfe085890 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ac8c00640 10 client.26755 handle_command_reply: tid=0 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 client.26755 shutdown 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 msgr2=0x560bfe016f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009a20 tx=0x7f5ad00095f0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 shutdown_connections 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x560bfe085890 0x560bfe087ca0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560bfe00eb90 0x560bfe014890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560bfe0042d0 0x560bfe014310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 --2- 172.21.15.52:0/3772820959 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560bfe0039b0 0x560bfe016f80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ac8c00640 0 client.26755 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 >> 172.21.15.52:0/3772820959 conn(0x560bfdfffec0 msgr2=0x560bfe00d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.296+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 shutdown_connections 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.297+0000 7f5ae8bfe740 1 -- 172.21.15.52:0/3772820959 wait complete. 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.297+0000 7f5ae8bfe740 20 client.26755 trim_cache size 0 max 16384 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:58:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:58:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:58:26.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.297+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 msgr2=0x7f5ac40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.297+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8005fd0 tx=0x7f5ad8009500 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 msgr2=0x7f5ae01a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00e9f0 tx=0x7f5acc00edf0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 shutdown_connections 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5ac40a3d10 0x7f5ac40a61d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae010e420 0x7f5ae019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae0108e10 0x7f5ae01a4d40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 --2- 172.21.15.52:0/1624702169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae0108490 0x7f5ae01a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 >> 172.21.15.52:0/1624702169 conn(0x7f5ae0077d30 msgr2=0x7f5ae0078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 shutdown_connections 2024-04-26T01:58:26.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:26.298+0000 7f5adfe00640 1 -- 172.21.15.52:0/1624702169 wait complete. 2024-04-26T01:58:27.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 276766 2.05 MB/sec execute 586 sec latency 225.350 ms 2024-04-26T01:58:27.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:27 smithi191 ceph-mon[32745]: pgmap v423: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 352 KiB/s wr, 54 op/s 2024-04-26T01:58:27.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:27 smithi183 ceph-mon[32752]: pgmap v423: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 352 KiB/s wr, 54 op/s 2024-04-26T01:58:27.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:27 smithi052 ceph-mon[27851]: pgmap v423: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 352 KiB/s wr, 54 op/s 2024-04-26T01:58:28.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 276778 2.05 MB/sec execute 587 sec latency 1143.811 ms 2024-04-26T01:58:29.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 276833 2.05 MB/sec execute 588 sec latency 825.558 ms 2024-04-26T01:58:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:29 smithi191 ceph-mon[32745]: pgmap v424: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:29.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:29 smithi183 ceph-mon[32752]: pgmap v424: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:29.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:29 smithi052 ceph-mon[27851]: pgmap v424: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:30.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 276833 2.04 MB/sec execute 589 sec latency 1825.693 ms 2024-04-26T01:58:31.153 INFO:tasks.workunit.client.0.smithi052.stdout: 1 276833 2.04 MB/sec execute 590 sec latency 2825.855 ms 2024-04-26T01:58:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:31 smithi183 ceph-mon[32752]: pgmap v425: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:31 smithi052 ceph-mon[27851]: pgmap v425: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:31.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:31 smithi191 ceph-mon[32745]: pgmap v425: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 276 KiB/s wr, 41 op/s 2024-04-26T01:58:32.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277057 2.04 MB/sec execute 591 sec latency 3105.973 ms 2024-04-26T01:58:33.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277404 2.04 MB/sec execute 592 sec latency 26.809 ms 2024-04-26T01:58:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:33 smithi183 ceph-mon[32752]: pgmap v426: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 376 KiB/s wr, 64 op/s 2024-04-26T01:58:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:33 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:33.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:33 smithi052 ceph-mon[27851]: pgmap v426: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 376 KiB/s wr, 64 op/s 2024-04-26T01:58:33.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:33 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:33.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:33 smithi191 ceph-mon[32745]: pgmap v426: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 376 KiB/s wr, 64 op/s 2024-04-26T01:58:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:33 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:58:34.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277484 2.03 MB/sec execute 593 sec latency 896.827 ms 2024-04-26T01:58:35.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277484 2.03 MB/sec execute 594 sec latency 1896.959 ms 2024-04-26T01:58:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:35 smithi052 ceph-mon[27851]: pgmap v427: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 342 KiB/s wr, 54 op/s 2024-04-26T01:58:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:35 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:35 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:35 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:58:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:35 smithi191 ceph-mon[32745]: pgmap v427: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 342 KiB/s wr, 54 op/s 2024-04-26T01:58:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:35 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:35 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:35 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:58:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:35 smithi183 ceph-mon[32752]: pgmap v427: 97 pgs: 97 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 342 KiB/s wr, 54 op/s 2024-04-26T01:58:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:35 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:58:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:35 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:58:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:35 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:58:36.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277484 2.03 MB/sec execute 595 sec latency 2897.083 ms 2024-04-26T01:58:37.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277484 2.02 MB/sec execute 596 sec latency 3897.276 ms 2024-04-26T01:58:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:37 smithi052 ceph-mon[27851]: pgmap v428: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 793 KiB/s wr, 84 op/s 2024-04-26T01:58:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:37 smithi191 ceph-mon[32745]: pgmap v428: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 793 KiB/s wr, 84 op/s 2024-04-26T01:58:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:37 smithi183 ceph-mon[32752]: pgmap v428: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 793 KiB/s wr, 84 op/s 2024-04-26T01:58:38.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 277731 2.02 MB/sec execute 597 sec latency 4590.824 ms 2024-04-26T01:58:39.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 278352 2.02 MB/sec execute 598 sec latency 42.207 ms 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 -- 172.21.15.52:0/2279534417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38100d60 msgr2=0x7f8a38101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/2279534417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38100d60 0x7f8a38101140 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009ca0 tx=0x7f8a2802a8c0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 -- 172.21.15.52:0/2279534417 shutdown_connections 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/2279534417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a3810e910 0x7f8a38110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/2279534417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 0x7f8a3810e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/2279534417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38100d60 0x7f8a38101140 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 -- 172.21.15.52:0/2279534417 >> 172.21.15.52:0/2279534417 conn(0x7f8a38077fd0 msgr2=0x7f8a380783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 -- 172.21.15.52:0/2279534417 shutdown_connections 2024-04-26T01:58:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.459+0000 7f8a3fc00640 1 -- 172.21.15.52:0/2279534417 wait complete. 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 Processor -- start 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 -- start start 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 0x7f8a381a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a3810e910 0x7f8a3819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38114430 con 0x7f8a38101680 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381142b0 con 0x7f8a38100d60 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.460+0000 7f8a3fc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381145b0 con 0x7f8a3810e910 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51170/0 (socket says 172.21.15.52:51170) 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 -- 172.21.15.52:0/71892852 learned_addr learned my addr 172.21.15.52:0/71892852 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a37e00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 0x7f8a381a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a3810e910 0x7f8a3819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 -- 172.21.15.52:0/71892852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a3810e910 msgr2=0x7f8a3819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a3810e910 0x7f8a3819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 -- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 msgr2=0x7f8a381a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 0x7f8a381a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.461+0000 7f8a3d200640 1 -- 172.21.15.52:0/71892852 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3819afc0 con 0x7f8a38100d60 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a3d200640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a28005e30 tx=0x7f8a28031840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a35600640 1 -- 172.21.15.52:0/71892852 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28003b70 con 0x7f8a38100d60 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a35600640 1 -- 172.21.15.52:0/71892852 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a28003d10 con 0x7f8a38100d60 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a35600640 1 -- 172.21.15.52:0/71892852 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2802fa30 con 0x7f8a38100d60 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3819b250 con 0x7f8a38100d60 2024-04-26T01:58:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.462+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a381a7480 con 0x7f8a38100d60 2024-04-26T01:58:39.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.465+0000 7f8a35600640 1 -- 172.21.15.52:0/71892852 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8a2802fbd0 con 0x7f8a38100d60 2024-04-26T01:58:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 Processor -- start 2024-04-26T01:58:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 -- start start 2024-04-26T01:58:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 0x555cdfc22c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc23140 0x555cdfc2d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 0x555cdfc2fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555cdfb88900 con 0x555cdfc23140 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555cdfb2eef0 con 0x555cdfc2da00 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.466+0000 7f8a42a28740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555cdfb848c0 con 0x555cdfc22820 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc23140 0x555cdfc2d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a3dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 0x555cdfc22c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a3dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 0x555cdfc22c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50698/0 (socket says 172.21.15.52:50698) 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a3dc00640 1 -- 172.21.15.52:0/1514099243 learned_addr learned my addr 172.21.15.52:0/1514099243 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 0x555cdfc2fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a35600640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 0x7f8a1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 -- 172.21.15.52:0/1514099243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 msgr2=0x555cdfc22c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 0x555cdfc22c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 -- 172.21.15.52:0/1514099243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc23140 msgr2=0x555cdfc2d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a35600640 1 -- 172.21.15.52:0/71892852 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f8a280f19f0 con 0x7f8a38100d60 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc23140 0x555cdfc2d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a37e00640 1 -- 172.21.15.52:0/1514099243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cdf89b210 con 0x555cdfc2da00 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.467+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 0x7f8a1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.468+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 0x555cdfc2fdf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a3819baf0 tx=0x7f8a20011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bd985f9bc70bb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.468+0000 7f8a2e000640 1 -- 172.21.15.52:0/1514099243 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004050 con 0x555cdfc2da00 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.468+0000 7f8a2e000640 1 -- 172.21.15.52:0/1514099243 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a200054d0 con 0x555cdfc2da00 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.468+0000 7f8a2e000640 1 -- 172.21.15.52:0/1514099243 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004c80 con 0x555cdfc2da00 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- 172.21.15.52:0/1514099243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 msgr2=0x555cdfc2fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 0x555cdfc2fdf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a3819baf0 tx=0x7f8a20011810 comp rx=0 tx=0).stop 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 0x7f8a1c0a6300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8a300045e0 tx=0x7f8a300046c0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- 172.21.15.52:0/1514099243 shutdown_connections 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc2da00 0x555cdfc2fdf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc23140 0x555cdfc2d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1514099243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc22820 0x555cdfc22c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- 172.21.15.52:0/1514099243 >> 172.21.15.52:0/1514099243 conn(0x555cdfc1ed30 msgr2=0x555cdfc1f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- 172.21.15.52:0/1514099243 shutdown_connections 2024-04-26T01:58:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- 172.21.15.52:0/1514099243 wait complete. 2024-04-26T01:58:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.469+0000 7f8a42a28740 1 -- start start 2024-04-26T01:58:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 Processor -- start 2024-04-26T01:58:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc23140 0x555cdfc2ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc2da00 0x555cdfc2b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555cdfc30630 con 0x555cdfc2da00 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555cdfb42310 con 0x555cdfc22820 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555cdfc307b0 con 0x555cdfc23140 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a42a28740 10 client.0 _ll_register_callbacks cb 0x555cdfb10cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.470+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51186/0 (socket says 172.21.15.52:51186) 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 -- 172.21.15.52:0/3884022637 learned_addr learned my addr 172.21.15.52:0/3884022637 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc2da00 0x555cdfc2b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a37e00640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc23140 0x555cdfc2ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 -- 172.21.15.52:0/3884022637 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc23140 msgr2=0x555cdfc2ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc23140 0x555cdfc2ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 -- 172.21.15.52:0/3884022637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc2da00 msgr2=0x555cdfc2b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc2da00 0x555cdfc2b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 -- 172.21.15.52:0/3884022637 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cdf967620 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a3d200640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a280ef250 tx=0x7f8a280093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bd21cceef7b9535 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a26a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a280de0b0 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.471+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f8a280bdca0 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a200054d0 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f8a280ef560 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 10 client.26759 fetch_fsmap learned FSMap version 54 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x555cdfc30630 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a200054d0 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f8a280c6220 con 0x555cdfc22820 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 10 client.26759 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 10 client.26759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 20 client.26759 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfc3dd00 0x555cdfc40130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 4 client.26759 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.472+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cdfb9c8b0 con 0x555cdfc3dd00 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.473+0000 7f8a37e00640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfc3dd00 0x555cdfc40130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.473+0000 7f8a37e00640 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfc3dd00 0x555cdfc40130 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4897c9d0a8522c2 server_cookie=d883f3969bcddb95 in_seq=0 out_seq=0 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.473+0000 7f8a26a00640 10 client.26759 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.474+0000 7f8a26a00640 1 -- 172.21.15.52:0/3884022637 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555cdfb9c8b0 con 0x555cdfc3dd00 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.474+0000 7f8a26a00640 10 client.26759 handle_command_reply: tid=0 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.474+0000 7f8a42a28740 1 client.26759 shutdown 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.474+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 msgr2=0x555cdfc2a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.474+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a280ef250 tx=0x7f8a280093d0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 shutdown_connections 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfc3dd00 0x555cdfc40130 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc2da00 0x555cdfc2b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc23140 0x555cdfc2ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 --2- 172.21.15.52:0/3884022637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc22820 0x555cdfc2a910 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a26a00640 0 client.26759 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 >> 172.21.15.52:0/3884022637 conn(0x555cdfc1ed30 msgr2=0x555cdfc24ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 shutdown_connections 2024-04-26T01:58:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 1 -- 172.21.15.52:0/3884022637 wait complete. 2024-04-26T01:58:39.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.475+0000 7f8a42a28740 20 client.26759 trim_cache size 0 max 16384 2024-04-26T01:58:39.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.496+0000 7f8a42a28740 1 Processor -- start 2024-04-26T01:58:39.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.497+0000 7f8a42a28740 1 -- start start 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc45a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b27a0 con 0x555cdfc22820 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381b0010 con 0x555cdfc23140 2024-04-26T01:58:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a42a28740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381ade80 con 0x555cdfc2da00 2024-04-26T01:58:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a3dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc45a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a3dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc45a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50712/0 (socket says 172.21.15.52:50712) 2024-04-26T01:58:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a3dc00640 1 -- 172.21.15.52:0/2047630874 learned_addr learned my addr 172.21.15.52:0/2047630874 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.498+0000 7f8a37e00640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a3d200640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a37e00640 1 -- 172.21.15.52:0/2047630874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 msgr2=0x555cdfc45a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a37e00640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc45a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a37e00640 1 -- 172.21.15.52:0/2047630874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 msgr2=0x555cdfc394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a37e00640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc394f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a37e00640 1 -- 172.21.15.52:0/2047630874 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cdfc25720 con 0x555cdfc23140 2024-04-26T01:58:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.499+0000 7f8a3d200640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc394f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:58:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.500+0000 7f8a37e00640 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc251e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a2001b040 tx=0x7f8a20009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=425fbc89ad2826ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.500+0000 7f8a2d600640 1 -- 172.21.15.52:0/2047630874 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2002edd0 con 0x555cdfc23140 2024-04-26T01:58:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.500+0000 7f8a2d600640 1 -- 172.21.15.52:0/2047630874 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a20005040 con 0x555cdfc23140 2024-04-26T01:58:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.500+0000 7f8a2d600640 1 -- 172.21.15.52:0/2047630874 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004900 con 0x555cdfc23140 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 -- 172.21.15.52:0/2047630874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 msgr2=0x555cdfc251e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc251e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a2001b040 tx=0x7f8a20009680 comp rx=0 tx=0).stop 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 -- 172.21.15.52:0/2047630874 shutdown_connections 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc45a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc251e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 --2- 172.21.15.52:0/2047630874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc394f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 -- 172.21.15.52:0/2047630874 >> 172.21.15.52:0/2047630874 conn(0x555cdfc1ed30 msgr2=0x555cdfc25c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 -- 172.21.15.52:0/2047630874 shutdown_connections 2024-04-26T01:58:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.501+0000 7f8a42a28740 1 -- 172.21.15.52:0/2047630874 wait complete. 2024-04-26T01:58:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.502+0000 7f8a42a28740 1 -- start start 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.502+0000 7f8a42a28740 1 Processor -- start 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc35720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc32bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381ae4d0 con 0x555cdfc22820 2024-04-26T01:58:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381af940 con 0x555cdfc23140 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381aeba0 con 0x555cdfc2da00 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a42a28740 10 client.0 _ll_register_callbacks cb 0x7f8a280f7220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a37e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a37e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:51204/0 (socket says 172.21.15.52:51204) 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a37e00640 1 -- 172.21.15.52:0/1662274417 learned_addr learned my addr 172.21.15.52:0/1662274417 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a3d200640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc35720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc32bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.503+0000 7f8a37e00640 1 -- 172.21.15.52:0/1662274417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 msgr2=0x555cdfc32bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc32bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a37e00640 1 -- 172.21.15.52:0/1662274417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 msgr2=0x555cdfc35720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc35720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a37e00640 1 -- 172.21.15.52:0/1662274417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cdfc33390 con 0x555cdfc23140 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a3d200640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc35720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a3dc00640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc32bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a37e00640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a200059d0 tx=0x7f8a20009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2797232fcc4e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a26a00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004460 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7f8a200051f0 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.504+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a20005040 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f8a20045050 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 10 client.26765 fetch_fsmap learned FSMap version 54 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x555cdfcac4d0 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a20005040 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f8a200035f0 con 0x555cdfc23140 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 10 client.26765 fetch_fsmap finished waiting for FSMap version 54 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 10 client.26765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 20 client.26765 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfcacad0 0x555cdfcaeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 4 client.26765 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.505+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cdfb96320 con 0x555cdfcacad0 2024-04-26T01:58:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.506+0000 7f8a3d200640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfcacad0 0x555cdfcaeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.506+0000 7f8a3d200640 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfcacad0 0x555cdfcaeef0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3283b87401b052a server_cookie=c7482085838113fe in_seq=0 out_seq=0 2024-04-26T01:58:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.506+0000 7f8a26a00640 10 client.26765 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a26a00640 1 -- 172.21.15.52:0/1662274417 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cdfb96320 con 0x555cdfcacad0 2024-04-26T01:58:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a26a00640 10 client.26765 handle_command_reply: tid=0 2024-04-26T01:58:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 client.26765 shutdown 2024-04-26T01:58:39.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 msgr2=0x555cdfc35c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a200059d0 tx=0x7f8a20009d70 comp rx=0 tx=0).stop 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 shutdown_connections 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555cdfcacad0 0x555cdfcaeef0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555cdfc2da00 0x555cdfc32bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555cdfc23140 0x555cdfc35c60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 --2- 172.21.15.52:0/1662274417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555cdfc22820 0x555cdfc35720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 >> 172.21.15.52:0/1662274417 conn(0x555cdfc1ed30 msgr2=0x555cdfc1d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.530+0000 7f8a26a00640 0 client.26765 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.531+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 shutdown_connections 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.531+0000 7f8a42a28740 1 -- 172.21.15.52:0/1662274417 wait complete. 2024-04-26T01:58:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.531+0000 7f8a42a28740 20 client.26765 trim_cache size 0 max 16384 2024-04-26T01:58:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:58:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:58:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:58:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 msgr2=0x7f8a1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 0x7f8a1c0a6300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8a300045e0 tx=0x7f8a300046c0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 msgr2=0x7f8a381a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a28005e30 tx=0x7f8a28031840 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 shutdown_connections 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f8a1c0a3e40 0x7f8a1c0a6300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a3810e910 0x7f8a3819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a38101680 0x7f8a381a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 --2- 172.21.15.52:0/71892852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8a38100d60 0x7f8a381a0630 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 >> 172.21.15.52:0/71892852 conn(0x7f8a38077fd0 msgr2=0x7f8a3810fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 shutdown_connections 2024-04-26T01:58:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:39.532+0000 7f8a3fc00640 1 -- 172.21.15.52:0/71892852 wait complete. 2024-04-26T01:58:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:39 smithi052 ceph-mon[27851]: pgmap v429: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:39.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:39 smithi191 ceph-mon[32745]: pgmap v429: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:39 smithi183 ceph-mon[32752]: pgmap v429: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:40.154 INFO:tasks.workunit.client.0.smithi052.stdout: 1 279444 2.03 MB/sec execute 599 sec latency 37.252 ms 2024-04-26T01:58:41.155 INFO:tasks.workunit.client.0.smithi052.stdout: 1 cleanup 600 sec 2024-04-26T01:58:41.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:41 smithi052 ceph-mon[27851]: pgmap v430: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:41.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:41 smithi191 ceph-mon[32745]: pgmap v430: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:41 smithi183 ceph-mon[32752]: pgmap v430: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 550 KiB/s wr, 53 op/s 2024-04-26T01:58:42.155 INFO:tasks.workunit.client.0.smithi052.stdout: 1 cleanup 601 sec 2024-04-26T01:58:43.155 INFO:tasks.workunit.client.0.smithi052.stdout: 1 cleanup 602 sec 2024-04-26T01:58:43.598 INFO:tasks.workunit.client.0.smithi052.stdout: 0 cleanup 603 sec 2024-04-26T01:58:43.598 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-04-26T01:58:43.598 INFO:tasks.workunit.client.0.smithi052.stdout: Operation Count AvgLat MaxLat 2024-04-26T01:58:43.598 INFO:tasks.workunit.client.0.smithi052.stdout: ---------------------------------------- 2024-04-26T01:58:43.598 INFO:tasks.workunit.client.0.smithi052.stdout: NTCreateX 39102 6.859 36814.864 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Close 28759 0.013 0.475 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Rename 1661 64.013 30009.590 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Unlink 7892 14.557 37879.372 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Qpathinfo 35513 2.358 9396.903 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Qfileinfo 6204 0.005 0.092 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Qfsinfo 6514 0.675 13.346 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Sfileinfo 3200 0.132 17.612 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Find 13742 0.480 50.740 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: WriteX 19367 0.290 95.121 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: ReadX 61566 0.073 15.669 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: LockX 128 1.035 1.793 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: UnlockX 128 0.833 2.701 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: Flush 2752 5.327 68.219 2024-04-26T01:58:43.599 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-04-26T01:58:43.600 INFO:tasks.workunit.client.0.smithi052.stdout:Throughput 2.02935 MB/sec 1 clients 1 procs max_latency=37879.386 ms 2024-04-26T01:58:43.600 INFO:tasks.workunit.client.0.smithi052.stderr:+ dbench 10 2024-04-26T01:58:43.602 INFO:tasks.workunit.client.0.smithi052.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T01:58:43.602 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-04-26T01:58:43.602 INFO:tasks.workunit.client.0.smithi052.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T01:58:43.606 INFO:tasks.workunit.client.0.smithi052.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-26T01:58:43.706 INFO:tasks.workunit.client.0.smithi052.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-26T01:58:43.706 INFO:tasks.workunit.client.0.smithi052.stdout:releasing clients 2024-04-26T01:58:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:43 smithi052 ceph-mon[27851]: pgmap v431: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 856 KiB/s wr, 92 op/s 2024-04-26T01:58:43.726 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:43 smithi183 ceph-mon[32752]: pgmap v431: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 856 KiB/s wr, 92 op/s 2024-04-26T01:58:43.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:43 smithi191 ceph-mon[32745]: pgmap v431: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 856 KiB/s wr, 92 op/s 2024-04-26T01:58:44.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 365 130.70 MB/sec warmup 1 sec latency 74.965 ms 2024-04-26T01:58:45.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 791 103.62 MB/sec warmup 2 sec latency 96.216 ms 2024-04-26T01:58:45.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:45 smithi052 ceph-mon[27851]: pgmap v432: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 756 KiB/s wr, 69 op/s 2024-04-26T01:58:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:45 smithi191 ceph-mon[32745]: pgmap v432: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 756 KiB/s wr, 69 op/s 2024-04-26T01:58:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:45 smithi183 ceph-mon[32752]: pgmap v432: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 756 KiB/s wr, 69 op/s 2024-04-26T01:58:46.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 1224 77.66 MB/sec warmup 3 sec latency 123.075 ms 2024-04-26T01:58:47.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 1909 65.08 MB/sec warmup 4 sec latency 107.686 ms 2024-04-26T01:58:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:47 smithi052 ceph-mon[27851]: pgmap v433: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:58:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:47 smithi191 ceph-mon[32745]: pgmap v433: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:58:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:47 smithi183 ceph-mon[32752]: pgmap v433: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-26T01:58:48.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 2448 56.79 MB/sec warmup 5 sec latency 87.000 ms 2024-04-26T01:58:49.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 3157 57.44 MB/sec warmup 6 sec latency 92.259 ms 2024-04-26T01:58:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:49 smithi191 ceph-mon[32745]: pgmap v434: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:49 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:49 smithi183 ceph-mon[32752]: pgmap v434: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:49 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:49 smithi052 ceph-mon[27851]: pgmap v434: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:49 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:50.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 3565 54.87 MB/sec warmup 7 sec latency 130.536 ms 2024-04-26T01:58:51.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 3751 48.40 MB/sec warmup 8 sec latency 124.877 ms 2024-04-26T01:58:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:51 smithi191 ceph-mon[32745]: pgmap v435: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:51.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:51 smithi183 ceph-mon[32752]: pgmap v435: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:51 smithi052 ceph-mon[27851]: pgmap v435: 97 pgs: 97 active+clean; 116 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 985 KiB/s wr, 97 op/s 2024-04-26T01:58:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 -- 172.21.15.52:0/3346071176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 msgr2=0x7ff55c1153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 --2- 172.21.15.52:0/3346071176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1153e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff554009cd0 tx=0x7ff55402a8f0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 -- 172.21.15.52:0/3346071176 shutdown_connections 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 --2- 172.21.15.52:0/3346071176 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1153e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 --2- 172.21.15.52:0/3346071176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c108560 0x7ff55c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 --2- 172.21.15.52:0/3346071176 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c10e620 0x7ff55c10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 -- 172.21.15.52:0/3346071176 >> 172.21.15.52:0/3346071176 conn(0x7ff55c0764e0 msgr2=0x7ff55c078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 -- 172.21.15.52:0/3346071176 shutdown_connections 2024-04-26T01:58:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.687+0000 7ff562800640 1 -- 172.21.15.52:0/3346071176 wait complete. 2024-04-26T01:58:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.688+0000 7ff562800640 1 Processor -- start 2024-04-26T01:58:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.688+0000 7ff562800640 1 -- start start 2024-04-26T01:58:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c10e620 0x7ff55c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1187b0 con 0x7ff55c10e620 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c118630 con 0x7ff55c108f20 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff562800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c118930 con 0x7ff55c108560 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff55b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff55b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39134/0 (socket says 172.21.15.52:39134) 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff55b400640 1 -- 172.21.15.52:0/3784247949 learned_addr learned my addr 172.21.15.52:0/3784247949 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44650/0 (socket says 172.21.15.52:44650) 2024-04-26T01:58:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff55be00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c10e620 0x7ff55c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 -- 172.21.15.52:0/3784247949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 msgr2=0x7ff55c1a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 -- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c10e620 msgr2=0x7ff55c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c10e620 0x7ff55c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff553e00640 1 -- 172.21.15.52:0/3784247949 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c19f170 con 0x7ff55c108f20 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.689+0000 7ff55b400640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff553e00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff54400e9f0 tx=0x7ff54400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff558c00640 1 -- 172.21.15.52:0/3784247949 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54400cd00 con 0x7ff55c108f20 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff558c00640 1 -- 172.21.15.52:0/3784247949 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5440035f0 con 0x7ff55c108f20 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c19f460 con 0x7ff55c108f20 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff558c00640 1 -- 172.21.15.52:0/3784247949 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544010670 con 0x7ff55c108f20 2024-04-26T01:58:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.690+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c1ab700 con 0x7ff55c108f20 2024-04-26T01:58:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff558c00640 1 -- 172.21.15.52:0/3784247949 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff5440037a0 con 0x7ff55c108f20 2024-04-26T01:58:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 Processor -- start 2024-04-26T01:58:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 -- start start 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede71c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7eddd7900 con 0x55d7ede7ca00 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7edd7def0 con 0x55d7ede72140 2024-04-26T01:58:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.694+0000 7ff56578a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7eddd38c0 con 0x55d7ede71820 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44666/0 (socket says 172.21.15.52:44666) 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 -- 172.21.15.52:0/1635039742 learned_addr learned my addr 172.21.15.52:0/1635039742 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55be00640 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede71c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff553e00640 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 -- 172.21.15.52:0/1635039742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 msgr2=0x55d7ede71c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede71c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 -- 172.21.15.52:0/1635039742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 msgr2=0x55d7ede7edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 -- 172.21.15.52:0/1635039742 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7edaea210 con 0x55d7ede72140 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff558c00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 0x7ff5400a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff55b400640 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0115f0 tx=0x7ff54c011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a871dd7cf56927d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.695+0000 7ff558c00640 1 -- 172.21.15.52:0/3784247949 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff5440d0960 con 0x7ff55c108f20 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff550c00640 1 -- 172.21.15.52:0/1635039742 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c005020 con 0x55d7ede72140 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff55be00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 0x7ff5400a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff550c00640 1 -- 172.21.15.52:0/1635039742 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54c0047a0 con 0x55d7ede72140 2024-04-26T01:58:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff550c00640 1 -- 172.21.15.52:0/1635039742 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c00f980 con 0x55d7ede72140 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff56578a740 1 -- 172.21.15.52:0/1635039742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 msgr2=0x55d7ede7c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff56578a740 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0115f0 tx=0x7ff54c011ab0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff55be00640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 0x7ff5400a6350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff554006010 tx=0x7ff554004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff56578a740 1 -- 172.21.15.52:0/1635039742 shutdown_connections 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff56578a740 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.696+0000 7ff56578a740 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede7c4c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 --2- 172.21.15.52:0/1635039742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede71c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 -- 172.21.15.52:0/1635039742 >> 172.21.15.52:0/1635039742 conn(0x55d7ede6dd30 msgr2=0x55d7ede6e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 -- 172.21.15.52:0/1635039742 shutdown_connections 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 -- 172.21.15.52:0/1635039742 wait complete. 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 -- start start 2024-04-26T01:58:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.697+0000 7ff56578a740 1 Processor -- start 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede72140 0x55d7ede79e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7ede7f630 con 0x55d7ede7ca00 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7edd91310 con 0x55d7ede71820 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7ede7f7b0 con 0x55d7ede72140 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff56578a740 10 client.0 _ll_register_callbacks cb 0x55d7edd5fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff55b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff55be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff55b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44670/0 (socket says 172.21.15.52:44670) 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff55b400640 1 -- 172.21.15.52:0/2743555085 learned_addr learned my addr 172.21.15.52:0/2743555085 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.698+0000 7ff553e00640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede72140 0x55d7ede79e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 -- 172.21.15.52:0/2743555085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede72140 msgr2=0x55d7ede79e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede72140 0x55d7ede79e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 -- 172.21.15.52:0/2743555085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 msgr2=0x55d7ede7a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 -- 172.21.15.52:0/2743555085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7edbb6620 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff55b400640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff54c004ea0 tx=0x7ff54c005130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ff713742c62acce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff549600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c028070 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff54c0054e0 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.699+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff54c0047a0 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff54c034050 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 10 client.26777 fetch_fsmap learned FSMap version 55 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d7ede8b7f0 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54c0047a0 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 55) ==== 8358+0+0 (secure 0 0 0) 0x7ff54c024710 con 0x55d7ede71820 2024-04-26T01:58:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 10 client.26777 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 10 client.26777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 20 client.26777 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ede8cc00 0x55d7ede8eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 4 client.26777 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.700+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7eddeb8b0 con 0x55d7ede8cc00 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.701+0000 7ff553e00640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ede8cc00 0x55d7ede8eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.701+0000 7ff553e00640 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ede8cc00 0x55d7ede8eff0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1833006117f01983 server_cookie=eb9632017504fcb9 in_seq=0 out_seq=0 2024-04-26T01:58:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.701+0000 7ff549600640 10 client.26777 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff549600640 1 -- 172.21.15.52:0/2743555085 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7eddeb8b0 con 0x55d7ede8cc00 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff549600640 10 client.26777 handle_command_reply: tid=0 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 client.26777 shutdown 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 msgr2=0x55d7ede79900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff54c004ea0 tx=0x7ff54c005130 comp rx=0 tx=0).stop 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 shutdown_connections 2024-04-26T01:58:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ede8cc00 0x55d7ede8eff0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede7a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede72140 0x55d7ede79e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 --2- 172.21.15.52:0/2743555085 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede79900 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 >> 172.21.15.52:0/2743555085 conn(0x55d7ede6dd30 msgr2=0x55d7ede73a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.705+0000 7ff549600640 0 client.26777 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.706+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 shutdown_connections 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.706+0000 7ff56578a740 1 -- 172.21.15.52:0/2743555085 wait complete. 2024-04-26T01:58:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.706+0000 7ff56578a740 20 client.26777 trim_cache size 0 max 16384 2024-04-26T01:58:52.707 INFO:tasks.workunit.client.0.smithi052.stdout: 10 3943 43.21 MB/sec warmup 9 sec latency 74.603 ms 2024-04-26T01:58:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.725+0000 7ff56578a740 1 Processor -- start 2024-04-26T01:58:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.725+0000 7ff56578a740 1 -- start start 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 0x55d7ede6cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b6a30 con 0x55d7ede72140 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b4270 con 0x55d7ede71820 2024-04-26T01:58:52.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff56578a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b20e0 con 0x55d7ede6dd30 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff55b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff55be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 0x55d7ede6cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44672/0 (socket says 172.21.15.52:44672) 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff55b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39174/0 (socket says 172.21.15.52:39174) 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.726+0000 7ff55b400640 1 -- 172.21.15.52:0/2420118907 learned_addr learned my addr 172.21.15.52:0/2420118907 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 -- 172.21.15.52:0/2420118907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 msgr2=0x55d7ede741e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 -- 172.21.15.52:0/2420118907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 msgr2=0x55d7ede6cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 0x55d7ede6cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 -- 172.21.15.52:0/2420118907 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ede74720 con 0x55d7ede6dd30 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff553e00640 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55be00640 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 0x55d7ede6cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff55b400640 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff54c006010 tx=0x7ff54c022410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=177203019e7efca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff54be00640 1 -- 172.21.15.52:0/2420118907 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c028040 con 0x55d7ede6dd30 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff54be00640 1 -- 172.21.15.52:0/2420118907 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54c0248f0 con 0x55d7ede6dd30 2024-04-26T01:58:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.727+0000 7ff54be00640 1 -- 172.21.15.52:0/2420118907 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c00f920 con 0x55d7ede6dd30 2024-04-26T01:58:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 -- 172.21.15.52:0/2420118907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 msgr2=0x55d7ede884f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff54c006010 tx=0x7ff54c022410 comp rx=0 tx=0).stop 2024-04-26T01:58:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 -- 172.21.15.52:0/2420118907 shutdown_connections 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede72140 0x55d7ede6cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede71820 0x55d7ede741e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 --2- 172.21.15.52:0/2420118907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede6dd30 0x55d7ede884f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 -- 172.21.15.52:0/2420118907 >> 172.21.15.52:0/2420118907 conn(0x55d7ede8f360 msgr2=0x55d7ede6c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 -- 172.21.15.52:0/2420118907 shutdown_connections 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.728+0000 7ff56578a740 1 -- 172.21.15.52:0/2420118907 wait complete. 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.729+0000 7ff56578a740 1 -- start start 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.729+0000 7ff56578a740 1 Processor -- start 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.729+0000 7ff56578a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede84770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.729+0000 7ff56578a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.729+0000 7ff56578a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede81b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff56578a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b2730 con 0x55d7ede7ca00 2024-04-26T01:58:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff56578a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b3ba0 con 0x55d7ede72140 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff56578a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1b2e00 con 0x55d7ede71820 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff56578a740 10 client.0 _ll_register_callbacks cb 0x7ff54c004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44684/0 (socket says 172.21.15.52:44684) 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 -- 172.21.15.52:0/1971332941 learned_addr learned my addr 172.21.15.52:0/1971332941 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff55be00640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede81b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff55b400640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede84770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 -- 172.21.15.52:0/1971332941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 msgr2=0x55d7ede84770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede84770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 -- 172.21.15.52:0/1971332941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 msgr2=0x55d7ede81b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede81b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff553e00640 1 -- 172.21.15.52:0/1971332941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ede82380 con 0x55d7ede72140 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.730+0000 7ff55b400640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede84770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:58:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff553e00640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff54409c890 tx=0x7ff54409c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a872652dbada41a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5440a0cb0 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8521+0+0 (secure 0 0 0) 0x7ff544098f70 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff54c0248f0 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7ff5440a5570 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 10 client.26783 fetch_fsmap learned FSMap version 55 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55d7edef2b90 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54c0248f0 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 55) ==== 8358+0+0 (secure 0 0 0) 0x7ff544098650 con 0x55d7ede72140 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 10 client.26783 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 10 client.26783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 20 client.26783 populate_metadata read hostname 'smithi052' 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.731+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7edef3320 0x55d7edef5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.732+0000 7ff56578a740 4 client.26783 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:58:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.732+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7edde5320 con 0x55d7edef3320 2024-04-26T01:58:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.732+0000 7ff55b400640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7edef3320 0x55d7edef5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:58:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.732+0000 7ff55b400640 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7edef3320 0x55d7edef5740 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71cf0b9958caa3db server_cookie=f0f0d04ae0365ff6 in_seq=0 out_seq=0 2024-04-26T01:58:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:52.732+0000 7ff549600640 10 client.26783 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.441+0000 7ff549600640 1 -- 172.21.15.52:0/1971332941 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7edde5320 con 0x55d7edef3320 2024-04-26T01:58:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.441+0000 7ff549600640 10 client.26783 handle_command_reply: tid=0 2024-04-26T01:58:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.442+0000 7ff56578a740 1 client.26783 shutdown 2024-04-26T01:58:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.442+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 msgr2=0x55d7ede84cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.442+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff54409c890 tx=0x7ff54409c560 comp rx=0 tx=0).stop 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.442+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 shutdown_connections 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7edef3320 0x55d7edef5740 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ede7ca00 0x55d7ede81b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff549600640 0 client.26783 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ede72140 0x55d7ede84cb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 --2- 172.21.15.52:0/1971332941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ede71820 0x55d7ede84770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 >> 172.21.15.52:0/1971332941 conn(0x55d7ede8f360 msgr2=0x55d7ede6ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 shutdown_connections 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.443+0000 7ff56578a740 1 -- 172.21.15.52:0/1971332941 wait complete. 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.444+0000 7ff56578a740 20 client.26783 trim_cache size 0 max 16384 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:58:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:58:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 msgr2=0x7ff5400a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 0x7ff5400a6350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff554006010 tx=0x7ff554004a40 comp rx=0 tx=0).stop 2024-04-26T01:58:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 msgr2=0x7ff55c1a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:58:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff54400e9f0 tx=0x7ff54400eeb0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 shutdown_connections 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5400a3e90 0x7ff5400a6350 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff55c10e620 0x7ff55c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff55c108f20 0x7ff55c1a4d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 --2- 172.21.15.52:0/3784247949 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff55c108560 0x7ff55c1a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.447+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 >> 172.21.15.52:0/3784247949 conn(0x7ff55c0764e0 msgr2=0x7ff55c077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:58:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.448+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 shutdown_connections 2024-04-26T01:58:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:58:53.448+0000 7ff562800640 1 -- 172.21.15.52:0/3784247949 wait complete. 2024-04-26T01:58:53.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 4069 39.11 MB/sec warmup 10 sec latency 292.284 ms 2024-04-26T01:58:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:53 smithi191 ceph-mon[32745]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-04-26T01:58:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:53 smithi183 ceph-mon[32752]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-04-26T01:58:53.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:53 smithi052 ceph-mon[27851]: pgmap v436: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-04-26T01:58:54.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 4446 37.42 MB/sec warmup 11 sec latency 70.053 ms 2024-04-26T01:58:54.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:54 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:54.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:54 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:54 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:58:55.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 4933 36.87 MB/sec warmup 12 sec latency 86.096 ms 2024-04-26T01:58:55.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:55 smithi191 ceph-mon[32745]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-04-26T01:58:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:55 smithi183 ceph-mon[32752]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-04-26T01:58:55.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:55 smithi052 ceph-mon[27851]: pgmap v437: 97 pgs: 97 active+clean; 128 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 1.8 MiB/s wr, 136 op/s 2024-04-26T01:58:56.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 5694 35.54 MB/sec warmup 13 sec latency 90.032 ms 2024-04-26T01:58:57.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 6110 35.86 MB/sec warmup 14 sec latency 100.973 ms 2024-04-26T01:58:57.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:57 smithi191 ceph-mon[32745]: pgmap v438: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 468 op/s 2024-04-26T01:58:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:57 smithi183 ceph-mon[32752]: pgmap v438: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 468 op/s 2024-04-26T01:58:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:57 smithi052 ceph-mon[27851]: pgmap v438: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 468 op/s 2024-04-26T01:58:58.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 6674 36.00 MB/sec warmup 15 sec latency 49.347 ms 2024-04-26T01:58:59.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 7016 35.62 MB/sec warmup 16 sec latency 68.085 ms 2024-04-26T01:58:59.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:58:59 smithi191 ceph-mon[32745]: pgmap v439: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:58:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:58:59 smithi183 ceph-mon[32752]: pgmap v439: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:58:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:58:59 smithi052 ceph-mon[27851]: pgmap v439: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:59:00.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 7174 33.72 MB/sec warmup 17 sec latency 67.876 ms 2024-04-26T01:59:01.708 INFO:tasks.workunit.client.0.smithi052.stdout: 10 7399 31.92 MB/sec warmup 18 sec latency 167.958 ms 2024-04-26T01:59:01.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:01 smithi191 ceph-mon[32745]: pgmap v440: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:59:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:01 smithi183 ceph-mon[32752]: pgmap v440: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:59:01.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:01 smithi052 ceph-mon[27851]: pgmap v440: 97 pgs: 97 active+clean; 125 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 409 op/s 2024-04-26T01:59:02.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 7755 30.80 MB/sec warmup 19 sec latency 144.410 ms 2024-04-26T01:59:02.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:02 smithi191 ceph-mon[32745]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:59:02.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:02 smithi183 ceph-mon[32752]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:59:02.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:02 smithi052 ceph-mon[27851]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T01:59:03.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 8318 30.95 MB/sec warmup 20 sec latency 85.559 ms 2024-04-26T01:59:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:03 smithi191 ceph-mon[32745]: pgmap v442: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 495 op/s 2024-04-26T01:59:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:03 smithi183 ceph-mon[32752]: pgmap v442: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 495 op/s 2024-04-26T01:59:03.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:03 smithi052 ceph-mon[27851]: pgmap v442: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 495 op/s 2024-04-26T01:59:04.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 9624 32.46 MB/sec warmup 21 sec latency 38.676 ms 2024-04-26T01:59:05.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 10582 34.56 MB/sec warmup 22 sec latency 32.490 ms 2024-04-26T01:59:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:05 smithi191 ceph-mon[32745]: pgmap v443: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 496 op/s 2024-04-26T01:59:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:05 smithi183 ceph-mon[32752]: pgmap v443: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 496 op/s 2024-04-26T01:59:05.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:05 smithi052 ceph-mon[27851]: pgmap v443: 97 pgs: 97 active+clean; 135 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 496 op/s 2024-04-26T01:59:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 -- 172.21.15.52:0/3793382628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074c40 msgr2=0x7f135810e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 --2- 172.21.15.52:0/3793382628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074c40 0x7f135810e0d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1344009900 tx=0x7f134402a490 comp rx=0 tx=0).stop 2024-04-26T01:59:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 -- 172.21.15.52:0/3793382628 shutdown_connections 2024-04-26T01:59:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 --2- 172.21.15.52:0/3793382628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f135810e610 0x7f1358110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 --2- 172.21.15.52:0/3793382628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074c40 0x7f135810e0d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 --2- 172.21.15.52:0/3793382628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1358074320 0x7f1358074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.651+0000 7f135e600640 1 -- 172.21.15.52:0/3793382628 >> 172.21.15.52:0/3793382628 conn(0x7f13580fd8a0 msgr2=0x7f13580ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.652+0000 7f135e600640 1 -- 172.21.15.52:0/3793382628 shutdown_connections 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.652+0000 7f135e600640 1 -- 172.21.15.52:0/3793382628 wait complete. 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.653+0000 7f135e600640 1 Processor -- start 2024-04-26T01:59:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.653+0000 7f135e600640 1 -- start start 2024-04-26T01:59:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 0x7f13581a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f135810e610 0x7f135819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1358114240 con 0x7f135810e610 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13581140c0 con 0x7f1358074320 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135e600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13581143c0 con 0x7f1358074c40 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44620/0 (socket says 172.21.15.52:44620) 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 -- 172.21.15.52:0/3231752525 learned_addr learned my addr 172.21.15.52:0/3231752525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f1357e00640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 0x7f13581a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135dc00640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f135810e610 0x7f135819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 -- 172.21.15.52:0/3231752525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 msgr2=0x7f13581a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 0x7f13581a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 -- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f135810e610 msgr2=0x7f135819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f135810e610 0x7f135819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f135d200640 1 -- 172.21.15.52:0/3231752525 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135819ada0 con 0x7f1358074320 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.654+0000 7f1357e00640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 0x7f13581a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f135d200640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d950 tx=0x7f134c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f1355600640 1 -- 172.21.15.52:0/3231752525 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c017030 con 0x7f1358074320 2024-04-26T01:59:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f1355600640 1 -- 172.21.15.52:0/3231752525 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134c004020 con 0x7f1358074320 2024-04-26T01:59:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f1355600640 1 -- 172.21.15.52:0/3231752525 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0047a0 con 0x7f1358074320 2024-04-26T01:59:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f135819b090 con 0x7f1358074320 2024-04-26T01:59:06.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.655+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13581a7330 con 0x7f1358074320 2024-04-26T01:59:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.660+0000 7f136298c740 1 Processor -- start 2024-04-26T01:59:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.661+0000 7f136298c740 1 -- start start 2024-04-26T01:59:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.662+0000 7f1355600640 1 -- 172.21.15.52:0/3231752525 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f134c009a30 con 0x7f1358074320 2024-04-26T01:59:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d2cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 0x563094d37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d37b90 0x563094d39f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563094c92900 con 0x563094d2c9b0 2024-04-26T01:59:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563094c38ef0 con 0x563094d37b90 2024-04-26T01:59:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.663+0000 7f136298c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563094c8e8c0 con 0x563094d2d2d0 2024-04-26T01:59:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f1355600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 0x7f13400a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f135dc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d2cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f135dc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d2cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52912/0 (socket says 172.21.15.52:52912) 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f135dc00640 1 -- 172.21.15.52:0/540494200 learned_addr learned my addr 172.21.15.52:0/540494200 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f1355600640 1 -- 172.21.15.52:0/3231752525 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8565+0+0 (secure 0 0 0) 0x7f134c0d0330 con 0x7f1358074320 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f1357e00640 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d37b90 0x563094d39f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.664+0000 7f135d200640 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 0x563094d37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 -- 172.21.15.52:0/540494200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d37b90 msgr2=0x563094d39f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d37b90 0x563094d39f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 -- 172.21.15.52:0/540494200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 msgr2=0x563094d2cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d2cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 -- 172.21.15.52:0/540494200 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630949a5210 con 0x563094d2d2d0 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135dc00640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 0x7f13400a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.665+0000 7f135d200640 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 0x563094d37650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f134c0cd9f0 tx=0x7f134c09beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e434d264fefb362 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.666+0000 7f134a000640 1 -- 172.21.15.52:0/540494200 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0a05f0 con 0x563094d2d2d0 2024-04-26T01:59:06.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.666+0000 7f134a000640 1 -- 172.21.15.52:0/540494200 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134c0a4eb0 con 0x563094d2d2d0 2024-04-26T01:59:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.666+0000 7f134a000640 1 -- 172.21.15.52:0/540494200 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0cdc70 con 0x563094d2d2d0 2024-04-26T01:59:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.667+0000 7f136298c740 1 -- 172.21.15.52:0/540494200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 msgr2=0x563094d37650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.667+0000 7f136298c740 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 0x563094d37650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f134c0cd9f0 tx=0x7f134c09beb0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f135dc00640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 0x7f13400a62d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13500128b0 tx=0x7f1350009b50 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f136298c740 1 -- 172.21.15.52:0/540494200 shutdown_connections 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f136298c740 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d37b90 0x563094d39f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f136298c740 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2d2d0 0x563094d37650 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f136298c740 1 --2- 172.21.15.52:0/540494200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d2cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.668+0000 7f136298c740 1 -- 172.21.15.52:0/540494200 >> 172.21.15.52:0/540494200 conn(0x563094d28ec0 msgr2=0x563094d292a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.669+0000 7f136298c740 1 -- 172.21.15.52:0/540494200 shutdown_connections 2024-04-26T01:59:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.669+0000 7f136298c740 1 -- 172.21.15.52:0/540494200 wait complete. 2024-04-26T01:59:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.669+0000 7f136298c740 1 -- start start 2024-04-26T01:59:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.670+0000 7f136298c740 1 Processor -- start 2024-04-26T01:59:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.673+0000 7f136298c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.673+0000 7f136298c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 0x563094d350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.673+0000 7f136298c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094d35600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f136298c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563094d3a760 con 0x563094d2d2d0 2024-04-26T01:59:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f136298c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563094c4c310 con 0x563094d2c9b0 2024-04-26T01:59:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f136298c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563094d3a8e0 con 0x563094d37b90 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f136298c740 10 client.0 _ll_register_callbacks cb 0x563094c1acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.673+0000 7f135d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.673+0000 7f1357e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 0x563094d350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f135d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44646/0 (socket says 172.21.15.52:44646) 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f135d200640 1 -- 172.21.15.52:0/3440085833 learned_addr learned my addr 172.21.15.52:0/3440085833 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f135dc00640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094d35600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.674+0000 7f135d200640 1 -- 172.21.15.52:0/3440085833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 msgr2=0x563094d35600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135d200640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094d35600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135d200640 1 -- 172.21.15.52:0/3440085833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 msgr2=0x563094d350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135d200640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 0x563094d350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135d200640 1 -- 172.21.15.52:0/3440085833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563094a71620 con 0x563094d2c9b0 2024-04-26T01:59:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135dc00640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094d35600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f1357e00640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 0x563094d350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f135d200640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f134c0986a0 tx=0x7f134c004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3a7088fa7c5ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f133ea00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0b9020 con 0x563094d2c9b0 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.675+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8565+0+0 (secure 0 0 0) 0x7f134c0988b0 con 0x563094d2c9b0 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.676+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f134c0a4eb0 con 0x563094d2c9b0 2024-04-26T01:59:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.676+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f134c0cdc70 con 0x563094d2c9b0 2024-04-26T01:59:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.676+0000 7f136298c740 10 client.26795 fetch_fsmap learned FSMap version 56 2024-04-26T01:59:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.676+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x563094d46a10 con 0x563094d2c9b0 2024-04-26T01:59:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.676+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134c0cde10 con 0x563094d2c9b0 2024-04-26T01:59:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 56) ==== 8362+0+0 (secure 0 0 0) 0x7f134c0b0ce0 con 0x563094d2c9b0 2024-04-26T01:59:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 10 client.26795 fetch_fsmap finished waiting for FSMap version 56 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 10 client.26795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 20 client.26795 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094d4a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 4 client.26795 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.677+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f134c0a4eb0 con 0x563094d47d60 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.678+0000 7f1357e00640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094d4a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.678+0000 7f1357e00640 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094d4a150 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da54bfd6faade997 server_cookie=d7b084192ca2a99b in_seq=0 out_seq=0 2024-04-26T01:59:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.678+0000 7f133ea00640 10 client.26795 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.680+0000 7f133ea00640 1 -- 172.21.15.52:0/3440085833 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f134c0a4eb0 con 0x563094d47d60 2024-04-26T01:59:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.680+0000 7f133ea00640 10 client.26795 handle_command_reply: tid=0 2024-04-26T01:59:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.680+0000 7f136298c740 1 client.26795 shutdown 2024-04-26T01:59:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 msgr2=0x563094d34b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f134c0986a0 tx=0x7f134c004700 comp rx=0 tx=0).stop 2024-04-26T01:59:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 shutdown_connections 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094d4a150 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094d35600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f133ea00640 0 client.26795 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2d2d0 0x563094d350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 --2- 172.21.15.52:0/3440085833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2c9b0 0x563094d34b80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.681+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 >> 172.21.15.52:0/3440085833 conn(0x563094d28ec0 msgr2=0x563094d2ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.682+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 shutdown_connections 2024-04-26T01:59:06.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.682+0000 7f136298c740 1 -- 172.21.15.52:0/3440085833 wait complete. 2024-04-26T01:59:06.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.682+0000 7f136298c740 20 client.26795 trim_cache size 0 max 16384 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 Processor -- start 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 -- start start 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2c9b0 0x563094d2f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d37b90 0x563094d4d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f13581b2700 con 0x563094d37b90 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13581aff10 con 0x563094d2d2d0 2024-04-26T01:59:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.705+0000 7f136298c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13581add80 con 0x563094d2c9b0 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f135d200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2c9b0 0x563094d2f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44658/0 (socket says 172.21.15.52:44658) 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 -- 172.21.15.52:0/4151723726 learned_addr learned my addr 172.21.15.52:0/4151723726 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f135dc00640 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d37b90 0x563094d4d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 -- 172.21.15.52:0/4151723726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2c9b0 msgr2=0x563094d2f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2c9b0 0x563094d2f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 -- 172.21.15.52:0/4151723726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d37b90 msgr2=0x563094d4d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d37b90 0x563094d4d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.706+0000 7f1357e00640 1 -- 172.21.15.52:0/4151723726 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563094d2e0d0 con 0x563094d2d2d0 2024-04-26T01:59:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.707+0000 7f1357e00640 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1344031440 tx=0x7f13440328e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d8b5d3bfb3da9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.707+0000 7f1349600640 1 -- 172.21.15.52:0/4151723726 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344035040 con 0x563094d2d2d0 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.707+0000 7f1349600640 1 -- 172.21.15.52:0/4151723726 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134404a070 con 0x563094d2d2d0 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.707+0000 7f1349600640 1 -- 172.21.15.52:0/4151723726 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13440452c0 con 0x563094d2d2d0 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 -- 172.21.15.52:0/4151723726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 msgr2=0x563094d4d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1344031440 tx=0x7f13440328e0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 -- 172.21.15.52:0/4151723726 shutdown_connections 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d37b90 0x563094d4d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4d170 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 --2- 172.21.15.52:0/4151723726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d2c9b0 0x563094d2f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.708+0000 7f136298c740 1 -- 172.21.15.52:0/4151723726 >> 172.21.15.52:0/4151723726 conn(0x563094d28ec0 msgr2=0x563094d292a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.709+0000 7f136298c740 1 -- 172.21.15.52:0/4151723726 shutdown_connections 2024-04-26T01:59:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.709+0000 7f136298c740 1 -- 172.21.15.52:0/4151723726 wait complete. 2024-04-26T01:59:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.710+0000 7f136298c740 1 -- start start 2024-04-26T01:59:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.710+0000 7f136298c740 1 Processor -- start 2024-04-26T01:59:06.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 11001 33.48 MB/sec warmup 23 sec latency 81.270 ms 2024-04-26T01:59:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094dabf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f13581ae3d0 con 0x563094d2c9b0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13581af840 con 0x563094d2d2d0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13581aeaa0 con 0x563094d37b90 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f136298c740 10 client.0 _ll_register_callbacks cb 0x7f134c09bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f135d200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f135dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094dabf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f135d200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52948/0 (socket says 172.21.15.52:52948) 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.711+0000 7f135d200640 1 -- 172.21.15.52:0/772571407 learned_addr learned my addr 172.21.15.52:0/772571407 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f1357e00640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 -- 172.21.15.52:0/772571407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 msgr2=0x563094dabf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094dabf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 -- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 msgr2=0x563094d4c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 -- 172.21.15.52:0/772571407 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563094dac480 con 0x563094d2c9b0 2024-04-26T01:59:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135dc00640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094dabf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.712+0000 7f135d200640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f134c09bf40 tx=0x7f134c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9259e6864f3ead server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f133ea00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c0b9040 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8565+0+0 (secure 0 0 0) 0x7f134c0cde10 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f134404a070 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) ==== 24+0+0 (secure 0 0 0) 0x7f134c0b0050 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f136298c740 10 client.16668 fetch_fsmap learned FSMap version 56 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.713+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=4+}) -- 0x563094dad6c0 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f134404a070 con 0x563094d2c9b0 2024-04-26T01:59:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 56) ==== 8362+0+0 (secure 0 0 0) 0x7f134c0b0ce0 con 0x563094d2c9b0 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 10 client.16668 fetch_fsmap finished waiting for FSMap version 56 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 20 client.16668 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094dafc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 4 client.16668 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563094ca68b0 con 0x563094d47d60 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.714+0000 7f1357e00640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094dafc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.715+0000 7f1357e00640 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094dafc90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed8be06fbe4bb487 server_cookie=322be090f499d3ad in_seq=0 out_seq=0 2024-04-26T01:59:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.715+0000 7f133ea00640 10 client.16668 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.990+0000 7f133ea00640 1 -- 172.21.15.52:0/772571407 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563094ca68b0 con 0x563094d47d60 2024-04-26T01:59:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.990+0000 7f133ea00640 10 client.16668 handle_command_reply: tid=0 2024-04-26T01:59:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.990+0000 7f136298c740 1 client.16668 shutdown 2024-04-26T01:59:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 msgr2=0x563094d4bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f134c09bf40 tx=0x7f134c0097b0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 shutdown_connections 2024-04-26T01:59:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563094d47d60 0x563094dafc90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563094d37b90 0x563094dabf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563094d2d2d0 0x563094d4c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 --2- 172.21.15.52:0/772571407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563094d2c9b0 0x563094d4bf80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 >> 172.21.15.52:0/772571407 conn(0x563094d28ec0 msgr2=0x563094d42b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.991+0000 7f133ea00640 0 client.16668 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.992+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 shutdown_connections 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.992+0000 7f136298c740 1 -- 172.21.15.52:0/772571407 wait complete. 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.992+0000 7f136298c740 20 client.16668 trim_cache size 0 max 16384 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:59:06.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.993+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 msgr2=0x7f13400a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.993+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 0x7f13400a62d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13500128b0 tx=0x7f1350009b50 comp rx=0 tx=0).stop 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.993+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 msgr2=0x7f13581a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.993+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d950 tx=0x7f134c00de10 comp rx=0 tx=0).stop 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 shutdown_connections 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f13400a3e10 0x7f13400a62d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f135810e610 0x7f135819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1358074c40 0x7f13581a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 --2- 172.21.15.52:0/3231752525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1358074320 0x7f13581a0470 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.994+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 >> 172.21.15.52:0/3231752525 conn(0x7f13580fd8a0 msgr2=0x7f13580fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.995+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 shutdown_connections 2024-04-26T01:59:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:06.995+0000 7f135e600640 1 -- 172.21.15.52:0/3231752525 wait complete. 2024-04-26T01:59:07.567 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T01:59:07.567 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T01:59:07.567 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/2670226262 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f20fcf7a40 con 0x55f20fccb000 2024-04-26T01:59:07.567 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T01:59:07.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 11260 32.58 MB/sec warmup 24 sec latency 222.475 ms 2024-04-26T01:59:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:07 smithi191 ceph-mon[32745]: pgmap v444: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.5 MiB/s wr, 533 op/s 2024-04-26T01:59:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:07 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T01:59:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-mon[32752]: pgmap v444: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.5 MiB/s wr, 533 op/s 2024-04-26T01:59:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:07 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T01:59:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:07 smithi052 ceph-mon[27851]: pgmap v444: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.5 MiB/s wr, 533 op/s 2024-04-26T01:59:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:07 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T01:59:08.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 11407 31.37 MB/sec warmup 25 sec latency 547.515 ms 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/3742874930,v1:172.21.15.183:6837/3742874930] up:boot 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:08 smithi191 ceph-mon[32745]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/3742874930,v1:172.21.15.183:6837/3742874930] up:boot 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:08 smithi183 ceph-mon[32752]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/3742874930,v1:172.21.15.183:6837/3742874930] up:boot 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T01:59:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:08 smithi052 ceph-mon[27851]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T01:59:09.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 12341 32.10 MB/sec warmup 26 sec latency 56.571 ms 2024-04-26T01:59:09.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:09 smithi191 ceph-mon[32745]: pgmap v446: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 666 op/s 2024-04-26T01:59:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:09 smithi183 ceph-mon[32752]: pgmap v446: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 666 op/s 2024-04-26T01:59:09.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:09 smithi052 ceph-mon[27851]: pgmap v446: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 666 op/s 2024-04-26T01:59:10.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 13420 33.32 MB/sec warmup 27 sec latency 45.001 ms 2024-04-26T01:59:11.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 14036 33.60 MB/sec warmup 28 sec latency 58.730 ms 2024-04-26T01:59:11.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:11 smithi191 ceph-mon[32745]: pgmap v448: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 9.3 MiB/s wr, 543 op/s 2024-04-26T01:59:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:11 smithi183 ceph-mon[32752]: pgmap v448: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 9.3 MiB/s wr, 543 op/s 2024-04-26T01:59:11.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:11 smithi052 ceph-mon[27851]: pgmap v448: 97 pgs: 97 active+clean; 198 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 9.3 MiB/s wr, 543 op/s 2024-04-26T01:59:12.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 14658 33.10 MB/sec warmup 29 sec latency 42.767 ms 2024-04-26T01:59:13.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 15508 33.48 MB/sec warmup 30 sec latency 60.519 ms 2024-04-26T01:59:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:13 smithi191 ceph-mon[32745]: pgmap v449: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 1.31k op/s 2024-04-26T01:59:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:13 smithi183 ceph-mon[32752]: pgmap v449: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 1.31k op/s 2024-04-26T01:59:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:13 smithi052 ceph-mon[27851]: pgmap v449: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 39 MiB/s wr, 1.31k op/s 2024-04-26T01:59:14.709 INFO:tasks.workunit.client.0.smithi052.stdout: 10 16518 34.06 MB/sec warmup 31 sec latency 84.124 ms 2024-04-26T01:59:15.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 17349 34.99 MB/sec warmup 32 sec latency 71.987 ms 2024-04-26T01:59:15.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:15 smithi191 ceph-mon[32745]: pgmap v450: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 29 MiB/s wr, 763 op/s 2024-04-26T01:59:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:15 smithi183 ceph-mon[32752]: pgmap v450: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 29 MiB/s wr, 763 op/s 2024-04-26T01:59:15.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:15 smithi052 ceph-mon[27851]: pgmap v450: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 29 MiB/s wr, 763 op/s 2024-04-26T01:59:16.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 18035 34.76 MB/sec warmup 33 sec latency 44.778 ms 2024-04-26T01:59:17.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 18671 34.64 MB/sec warmup 34 sec latency 83.539 ms 2024-04-26T01:59:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:17 smithi191 ceph-mon[32745]: pgmap v451: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 31 MiB/s wr, 980 op/s 2024-04-26T01:59:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:17 smithi183 ceph-mon[32752]: pgmap v451: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 31 MiB/s wr, 980 op/s 2024-04-26T01:59:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:17 smithi052 ceph-mon[27851]: pgmap v451: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 31 MiB/s wr, 980 op/s 2024-04-26T01:59:18.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 19387 34.89 MB/sec warmup 35 sec latency 130.749 ms 2024-04-26T01:59:19.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 20391 35.15 MB/sec warmup 36 sec latency 44.567 ms 2024-04-26T01:59:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:19 smithi191 ceph-mon[32745]: pgmap v452: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 26 MiB/s wr, 850 op/s 2024-04-26T01:59:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:19 smithi183 ceph-mon[32752]: pgmap v452: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 26 MiB/s wr, 850 op/s 2024-04-26T01:59:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:19 smithi052 ceph-mon[27851]: pgmap v452: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 26 MiB/s wr, 850 op/s 2024-04-26T01:59:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 -- 172.21.15.52:0/2769567464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 msgr2=0x7f099810dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 --2- 172.21.15.52:0/2769567464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f099810dd50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0990009900 tx=0x7f099002a400 comp rx=0 tx=0).stop 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 -- 172.21.15.52:0/2769567464 shutdown_connections 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 --2- 172.21.15.52:0/2769567464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f099810dd50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 --2- 172.21.15.52:0/2769567464 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998101a60 0x7f0998105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 --2- 172.21.15.52:0/2769567464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f0998101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 -- 172.21.15.52:0/2769567464 >> 172.21.15.52:0/2769567464 conn(0x7f0998078000 msgr2=0x7f0998078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 -- 172.21.15.52:0/2769567464 shutdown_connections 2024-04-26T01:59:20.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.192+0000 7f099d800640 1 -- 172.21.15.52:0/2769567464 wait complete. 2024-04-26T01:59:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.194+0000 7f099d800640 1 Processor -- start 2024-04-26T01:59:20.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.194+0000 7f099d800640 1 -- start start 2024-04-26T01:59:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f0998196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f0996a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f0996a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:56252/0 (socket says 172.21.15.52:56252) 2024-04-26T01:59:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 0x7f09981aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110080 con 0x7f0998196d60 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f099810ff00 con 0x7f09981010b0 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.195+0000 7f099d800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0998110200 con 0x7f0998106180 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f0998196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f0996a00640 1 -- 172.21.15.52:0/1946730108 learned_addr learned my addr 172.21.15.52:0/1946730108 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f0997400640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 0x7f09981aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 -- 172.21.15.52:0/1946730108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 msgr2=0x7f09981962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 -- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 msgr2=0x7f09981aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 0x7f09981aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f098e000640 1 -- 172.21.15.52:0/1946730108 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09981ab3c0 con 0x7f0998106180 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.196+0000 7f0996a00640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.197+0000 7f0997400640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 0x7f09981aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.197+0000 7f098e000640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f0998196820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f098000d6a0 tx=0x7f098000db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.197+0000 7f098fe00640 1 -- 172.21.15.52:0/1946730108 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980017070 con 0x7f0998106180 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.197+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09981ab650 con 0x7f0998106180 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.197+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09981abbc0 con 0x7f0998106180 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.198+0000 7f098fe00640 1 -- 172.21.15.52:0/1946730108 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0980009820 con 0x7f0998106180 2024-04-26T01:59:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.198+0000 7f098fe00640 1 -- 172.21.15.52:0/1946730108 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980004c40 con 0x7f0998106180 2024-04-26T01:59:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.201+0000 7f098fe00640 1 -- 172.21.15.52:0/1946730108 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0980005220 con 0x7f0998106180 2024-04-26T01:59:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.202+0000 7f09a05f8740 1 Processor -- start 2024-04-26T01:59:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.203+0000 7f09a05f8740 1 -- start start 2024-04-26T01:59:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.204+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096f0820 0x5653096f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.204+0000 7f098fe00640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 0x7f097c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f098fe00640 1 -- 172.21.15.52:0/1946730108 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f09800cffe0 con 0x7f0998106180 2024-04-26T01:59:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096fba00 0x5653096fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f09a05f8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x565309656900 con 0x5653096f1140 2024-04-26T01:59:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f09a05f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5653095fcef0 con 0x5653096fba00 2024-04-26T01:59:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.205+0000 7f09a05f8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5653096528c0 con 0x5653096f0820 2024-04-26T01:59:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f098e000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f098e000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44952/0 (socket says 172.21.15.52:44952) 2024-04-26T01:59:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f0997400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096fba00 0x5653096fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f098e000640 1 -- 172.21.15.52:0/2174541947 learned_addr learned my addr 172.21.15.52:0/2174541947 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f0997400640 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096f0820 0x5653096f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 -- 172.21.15.52:0/2174541947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096f0820 msgr2=0x5653096f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.206+0000 7f0996a00640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 0x7f097c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096f0820 0x5653096f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 -- 172.21.15.52:0/2174541947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096fba00 msgr2=0x5653096fddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096fba00 0x5653096fddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 -- 172.21.15.52:0/2174541947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565309369210 con 0x5653096f1140 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f098e000640 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0980098950 tx=0x7f098009ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6edb7660592fe50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.207+0000 7f0987e00640 1 -- 172.21.15.52:0/2174541947 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980020040 con 0x5653096f1140 2024-04-26T01:59:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.208+0000 7f0987e00640 1 -- 172.21.15.52:0/2174541947 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09800cd9b0 con 0x5653096f1140 2024-04-26T01:59:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.208+0000 7f0987e00640 1 -- 172.21.15.52:0/2174541947 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980096af0 con 0x5653096f1140 2024-04-26T01:59:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 -- 172.21.15.52:0/2174541947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 msgr2=0x5653096fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0980098950 tx=0x7f098009ba10 comp rx=0 tx=0).stop 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f0996a00640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 0x7f097c0a6300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0988009fd0 tx=0x7f0988009630 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 -- 172.21.15.52:0/2174541947 shutdown_connections 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096fba00 0x5653096fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096fb4c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 --2- 172.21.15.52:0/2174541947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096f0820 0x5653096f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.209+0000 7f09a05f8740 1 -- 172.21.15.52:0/2174541947 >> 172.21.15.52:0/2174541947 conn(0x5653096ecd30 msgr2=0x5653096ed110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.210+0000 7f09a05f8740 1 -- 172.21.15.52:0/2174541947 shutdown_connections 2024-04-26T01:59:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.210+0000 7f09a05f8740 1 -- 172.21.15.52:0/2174541947 wait complete. 2024-04-26T01:59:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.210+0000 7f09a05f8740 1 -- start start 2024-04-26T01:59:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.211+0000 7f09a05f8740 1 Processor -- start 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.211+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.211+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.211+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f0996a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f098e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f0996a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44956/0 (socket says 172.21.15.52:44956) 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f098e000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:56268/0 (socket says 172.21.15.52:56268) 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f09a05f8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5653096fe630 con 0x5653096f0820 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f0996a00640 1 -- 172.21.15.52:0/4104003080 learned_addr learned my addr 172.21.15.52:0/4104003080 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f0997400640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.212+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x565309610310 con 0x5653096f1140 2024-04-26T01:59:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5653096fe7b0 con 0x5653096fba00 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f09a05f8740 10 client.0 _ll_register_callbacks cb 0x5653095decb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f098e000640 1 -- 172.21.15.52:0/4104003080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 msgr2=0x5653096f9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f098e000640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f098e000640 1 -- 172.21.15.52:0/4104003080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 msgr2=0x5653096f8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f098e000640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.213+0000 7f098e000640 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565309435620 con 0x5653096f1140 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f0996a00640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f098e000640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09800cdcb0 tx=0x7f09800cd710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454fbff7d84a71bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f0984c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800b9070 con 0x5653096f1140 2024-04-26T01:59:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f098009b3d0 con 0x5653096f1140 2024-04-26T01:59:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.214+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09800cd9b0 con 0x5653096f1140 2024-04-26T01:59:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.215+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0980082050 con 0x5653096f1140 2024-04-26T01:59:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.215+0000 7f09a05f8740 10 client.26816 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.215+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5653096fe630 con 0x5653096f1140 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.215+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09800cd9b0 con 0x5653096f1140 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.215+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f0980097cd0 con 0x5653096f1140 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 10 client.26816 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 10 client.26816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 20 client.26816 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56530970ba70 0x56530970de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 4 client.26816 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56530966a8b0 con 0x56530970ba70 2024-04-26T01:59:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f0996a00640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56530970ba70 0x56530970de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f0996a00640 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56530970ba70 0x56530970de60 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb5b2bfdd014cf2 server_cookie=705a5aa7b683a668 in_seq=0 out_seq=0 2024-04-26T01:59:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.216+0000 7f0984c00640 10 client.26816 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.218+0000 7f0984c00640 1 -- 172.21.15.52:0/4104003080 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56530966a8b0 con 0x56530970ba70 2024-04-26T01:59:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.218+0000 7f0984c00640 10 client.26816 handle_command_reply: tid=0 2024-04-26T01:59:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.218+0000 7f09a05f8740 1 client.26816 shutdown 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.218+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 msgr2=0x5653096f8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.218+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09800cdcb0 tx=0x7f09800cd710 comp rx=0 tx=0).stop 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 shutdown_connections 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56530970ba70 0x56530970de60 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f1140 0x5653096f8f40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f0984c00640 0 client.26816 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4104003080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f0820 0x5653096f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 >> 172.21.15.52:0/4104003080 conn(0x5653096ecd30 msgr2=0x5653096f2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 shutdown_connections 2024-04-26T01:59:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 1 -- 172.21.15.52:0/4104003080 wait complete. 2024-04-26T01:59:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.219+0000 7f09a05f8740 20 client.26816 trim_cache size 0 max 16384 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 Processor -- start 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 -- start start 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x5653097074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0974011090 con 0x5653096f1140 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f097400e8a0 con 0x5653096f0820 2024-04-26T01:59:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.245+0000 7f09a05f8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f097400c710 con 0x5653096fba00 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f0996a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x5653097074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f0997400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f0997400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:54062/0 (socket says 172.21.15.52:54062) 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f0997400640 1 -- 172.21.15.52:0/614164774 learned_addr learned my addr 172.21.15.52:0/614164774 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 -- 172.21.15.52:0/614164774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 msgr2=0x5653096f90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 -- 172.21.15.52:0/614164774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 msgr2=0x5653097074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x5653097074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.246+0000 7f098e000640 1 -- 172.21.15.52:0/614164774 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653096e8690 con 0x5653096f1140 2024-04-26T01:59:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.247+0000 7f0996a00640 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x5653097074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.247+0000 7f098e000640 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096f31e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0980098950 tx=0x7f0980098800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ce9a1a8496d581 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.247+0000 7f0987400640 1 -- 172.21.15.52:0/614164774 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09800b9040 con 0x5653096f1140 2024-04-26T01:59:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.247+0000 7f0987400640 1 -- 172.21.15.52:0/614164774 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09800a49e0 con 0x5653096f1140 2024-04-26T01:59:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.247+0000 7f0987400640 1 -- 172.21.15.52:0/614164774 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980020050 con 0x5653096f1140 2024-04-26T01:59:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.248+0000 7f09a05f8740 1 -- 172.21.15.52:0/614164774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 msgr2=0x5653096f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.248+0000 7f09a05f8740 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096f31e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0980098950 tx=0x7f0980098800 comp rx=0 tx=0).stop 2024-04-26T01:59:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.249+0000 7f09a05f8740 1 -- 172.21.15.52:0/614164774 shutdown_connections 2024-04-26T01:59:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.249+0000 7f09a05f8740 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653096f90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.249+0000 7f09a05f8740 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x5653096f31e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.249+0000 7f09a05f8740 1 --2- 172.21.15.52:0/614164774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x5653097074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.249+0000 7f09a05f8740 1 -- 172.21.15.52:0/614164774 >> 172.21.15.52:0/614164774 conn(0x5653096ecd30 msgr2=0x5653096f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.250+0000 7f09a05f8740 1 -- 172.21.15.52:0/614164774 shutdown_connections 2024-04-26T01:59:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.250+0000 7f09a05f8740 1 -- 172.21.15.52:0/614164774 wait complete. 2024-04-26T01:59:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.251+0000 7f09a05f8740 1 -- start start 2024-04-26T01:59:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.251+0000 7f09a05f8740 1 Processor -- start 2024-04-26T01:59:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x565309701290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653097018a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f097400cd60 con 0x5653096f1140 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f097400e1d0 con 0x5653096f0820 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f097400d430 con 0x5653096fba00 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f09a05f8740 10 client.0 _ll_register_callbacks cb 0x7f098009bf80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f0996a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f0996a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:56276/0 (socket says 172.21.15.52:56276) 2024-04-26T01:59:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.252+0000 7f0996a00640 1 -- 172.21.15.52:0/4068010027 learned_addr learned my addr 172.21.15.52:0/4068010027 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0997400640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653097018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f098e000640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x565309701290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 -- 172.21.15.52:0/4068010027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 msgr2=0x5653097018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653097018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 -- 172.21.15.52:0/4068010027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 msgr2=0x565309701290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x565309701290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 -- 172.21.15.52:0/4068010027 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653097020d0 con 0x5653096f0820 2024-04-26T01:59:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0997400640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653097018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0996a00640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0988000f80 tx=0x7f0988004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c00ed0911b5eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.253+0000 7f0984c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988028350 con 0x5653096f0820 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f09880284f0 con 0x5653096f0820 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09800a49e0 con 0x5653096f0820 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f09880286f0 con 0x5653096f0820 2024-04-26T01:59:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f09a05f8740 10 client.26822 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.254+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x565309783040 con 0x5653096f0820 2024-04-26T01:59:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09800a49e0 con 0x5653096f0820 2024-04-26T01:59:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f098803b420 con 0x5653096f0820 2024-04-26T01:59:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 10 client.26822 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 10 client.26822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 20 client.26822 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565309783840 0x565309785c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 4 client.26822 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.255+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565309664320 con 0x565309783840 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.256+0000 7f098e000640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565309783840 0x565309785c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.256+0000 7f098e000640 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565309783840 0x565309785c60 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4207d0c41617eefc server_cookie=3121e8980dd3b9a2 in_seq=0 out_seq=0 2024-04-26T01:59:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.256+0000 7f0984c00640 10 client.26822 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.501+0000 7f0984c00640 1 -- 172.21.15.52:0/4068010027 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565309664320 con 0x565309783840 2024-04-26T01:59:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.501+0000 7f0984c00640 10 client.26822 handle_command_reply: tid=0 2024-04-26T01:59:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.501+0000 7f09a05f8740 1 client.26822 shutdown 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.502+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 msgr2=0x565309703eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.502+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0988000f80 tx=0x7f0988004a40 comp rx=0 tx=0).stop 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.502+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 shutdown_connections 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.502+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565309783840 0x565309785c60 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653096fba00 0x5653097018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653096f1140 0x565309701290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 --2- 172.21.15.52:0/4068010027 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5653096f0820 0x565309703eb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 >> 172.21.15.52:0/4068010027 conn(0x5653096ecd30 msgr2=0x5653096f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f0984c00640 0 client.26822 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 shutdown_connections 2024-04-26T01:59:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 1 -- 172.21.15.52:0/4068010027 wait complete. 2024-04-26T01:59:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.503+0000 7f09a05f8740 20 client.26822 trim_cache size 0 max 16384 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 msgr2=0x7f097c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 0x7f097c0a6300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0988009fd0 tx=0x7f0988009630 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 msgr2=0x7f0998196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f0998196820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f098000d6a0 tx=0x7f098000db60 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 shutdown_connections 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f097c0a3e40 0x7f097c0a6300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0998196d60 0x7f09981aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0998106180 0x7f0998196820 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 --2- 172.21.15.52:0/1946730108 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09981010b0 0x7f09981962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.504+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 >> 172.21.15.52:0/1946730108 conn(0x7f0998078000 msgr2=0x7f09980fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.505+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 shutdown_connections 2024-04-26T01:59:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:20.505+0000 7f099d800640 1 -- 172.21.15.52:0/1946730108 wait complete. 2024-04-26T01:59:20.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 21110 35.66 MB/sec warmup 37 sec latency 150.188 ms 2024-04-26T01:59:21.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 21741 35.56 MB/sec warmup 38 sec latency 165.507 ms 2024-04-26T01:59:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:21 smithi191 ceph-mon[32745]: pgmap v453: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 715 op/s 2024-04-26T01:59:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:21 smithi183 ceph-mon[32752]: pgmap v453: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 715 op/s 2024-04-26T01:59:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:21 smithi052 ceph-mon[27851]: pgmap v453: 97 pgs: 97 active+clean; 392 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 22 MiB/s wr, 715 op/s 2024-04-26T01:59:22.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 22336 35.37 MB/sec warmup 39 sec latency 63.765 ms 2024-04-26T01:59:23.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 23024 35.23 MB/sec warmup 40 sec latency 143.078 ms 2024-04-26T01:59:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:23 smithi191 ceph-mon[32745]: pgmap v454: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 1.05k op/s 2024-04-26T01:59:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:23 smithi183 ceph-mon[32752]: pgmap v454: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 1.05k op/s 2024-04-26T01:59:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:23 smithi052 ceph-mon[27851]: pgmap v454: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 1.05k op/s 2024-04-26T01:59:24.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 24120 35.87 MB/sec warmup 41 sec latency 38.679 ms 2024-04-26T01:59:25.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 24992 36.38 MB/sec warmup 42 sec latency 51.951 ms 2024-04-26T01:59:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:25 smithi191 ceph-mon[32745]: pgmap v455: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-26T01:59:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:25 smithi183 ceph-mon[32752]: pgmap v455: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-26T01:59:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:25 smithi052 ceph-mon[27851]: pgmap v455: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 539 op/s 2024-04-26T01:59:26.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 25558 36.12 MB/sec warmup 43 sec latency 79.535 ms 2024-04-26T01:59:27.667 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T01:59:27.667 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.6 2024-04-26T01:59:27.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T01:59:27.710 INFO:tasks.workunit.client.0.smithi052.stdout: 10 26265 36.16 MB/sec warmup 44 sec latency 57.585 ms 2024-04-26T01:59:27.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:27 smithi191 ceph-mon[32745]: pgmap v456: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-04-26T01:59:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:27 smithi183 ceph-mon[32752]: pgmap v456: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-04-26T01:59:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:27 smithi052 ceph-mon[27851]: pgmap v456: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-04-26T01:59:28.300 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:59:28.712 INFO:tasks.workunit.client.0.smithi052.stdout: 10 27216 36.20 MB/sec warmup 45 sec latency 60.484 ms 2024-04-26T01:59:29.712 INFO:tasks.workunit.client.0.smithi052.stdout: 10 28023 36.61 MB/sec warmup 46 sec latency 349.060 ms 2024-04-26T01:59:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:29 smithi191 ceph-mon[32745]: pgmap v457: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:29 smithi183 ceph-mon[32752]: pgmap v457: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:29.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:29 smithi052 ceph-mon[27851]: pgmap v457: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:30.712 INFO:tasks.workunit.client.0.smithi052.stdout: 10 28424 36.49 MB/sec warmup 47 sec latency 306.183 ms 2024-04-26T01:59:30.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:30 smithi183 ceph-mon[32752]: pgmap v458: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:30.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:30 smithi052 ceph-mon[27851]: pgmap v458: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:31.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:30 smithi191 ceph-mon[32745]: pgmap v458: 97 pgs: 97 active+clean; 428 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-26T01:59:31.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 28779 36.19 MB/sec warmup 48 sec latency 437.011 ms 2024-04-26T01:59:32.712 INFO:tasks.workunit.client.0.smithi052.stdout: 10 29020 35.52 MB/sec warmup 49 sec latency 386.681 ms 2024-04-26T01:59:33.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:33 smithi191 ceph-mon[32745]: pgmap v459: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.3 MiB/s wr, 736 op/s 2024-04-26T01:59:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.673+0000 7f978ac00640 1 -- 172.21.15.52:0/3372071113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784106100 msgr2=0x7f9784106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.673+0000 7f978ac00640 1 --2- 172.21.15.52:0/3372071113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784106100 0x7f9784106590 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f977c009ca0 tx=0x7f977c02a940 comp rx=0 tx=0).stop 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 -- 172.21.15.52:0/3372071113 shutdown_connections 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 --2- 172.21.15.52:0/3372071113 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784106100 0x7f9784106590 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 --2- 172.21.15.52:0/3372071113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f9784105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 --2- 172.21.15.52:0/3372071113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784100d00 0x7f97841010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 -- 172.21.15.52:0/3372071113 >> 172.21.15.52:0/3372071113 conn(0x7f97840fc770 msgr2=0x7f97840feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 -- 172.21.15.52:0/3372071113 shutdown_connections 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.674+0000 7f978ac00640 1 -- 172.21.15.52:0/3372071113 wait complete. 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.675+0000 7f978ac00640 1 Processor -- start 2024-04-26T01:59:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.675+0000 7f978ac00640 1 -- start start 2024-04-26T01:59:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:33 smithi183 ceph-mon[32752]: pgmap v459: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.3 MiB/s wr, 736 op/s 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f978406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784106100 0x7f978406cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9784118620 con 0x7f9784101620 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f97841184a0 con 0x7f9784106100 2024-04-26T01:59:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.676+0000 7f978ac00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97841187a0 con 0x7f9784100d00 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46320/0 (socket says 172.21.15.52:46320) 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 -- 172.21.15.52:0/2677645985 learned_addr learned my addr 172.21.15.52:0/2677645985 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f978a200640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784106100 0x7f978406cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9788e00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f978406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 -- 172.21.15.52:0/2677645985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784106100 msgr2=0x7f978406cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784106100 0x7f978406cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 -- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 msgr2=0x7f978406c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f978406c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 -- 172.21.15.52:0/2677645985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978406d320 con 0x7f9784100d00 2024-04-26T01:59:33.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9788e00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f978406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:59:33.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.677+0000 7f9789800640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f977800e9f0 tx=0x7f977800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.678+0000 7f9782000640 1 -- 172.21.15.52:0/2677645985 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977800cd80 con 0x7f9784100d00 2024-04-26T01:59:33.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.678+0000 7f9782000640 1 -- 172.21.15.52:0/2677645985 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97780054d0 con 0x7f9784100d00 2024-04-26T01:59:33.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.678+0000 7f9782000640 1 -- 172.21.15.52:0/2677645985 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778010690 con 0x7f9784100d00 2024-04-26T01:59:33.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.678+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978406d610 con 0x7f9784100d00 2024-04-26T01:59:33.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.678+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97840715c0 con 0x7f9784100d00 2024-04-26T01:59:33.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.683+0000 7f978f051740 1 Processor -- start 2024-04-26T01:59:33.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.683+0000 7f9782000640 1 -- 172.21.15.52:0/2677645985 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9778004e90 con 0x7f9784100d00 2024-04-26T01:59:33.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 -- start start 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddaad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56161dd10900 con 0x56161ddb5b90 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56161dcb6ef0 con 0x56161ddab2d0 2024-04-26T01:59:33.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.684+0000 7f978f051740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56161dd0c8c0 con 0x56161ddaa9b0 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.685+0000 7f978a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddaad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.685+0000 7f9788e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.685+0000 7f9788e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58672/0 (socket says 172.21.15.52:58672) 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.685+0000 7f9788e00640 1 -- 172.21.15.52:0/477150324 learned_addr learned my addr 172.21.15.52:0/477150324 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.685+0000 7f9782000640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 0x7f976c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9782000640 1 -- 172.21.15.52:0/2677645985 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f97780d0d10 con 0x7f9784100d00 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 -- 172.21.15.52:0/477150324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 msgr2=0x56161ddaad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddaad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 -- 172.21.15.52:0/477150324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 msgr2=0x56161ddb5650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb5650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 -- 172.21.15.52:0/477150324 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56161da23210 con 0x56161ddb5b90 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.686+0000 7f9788e00640 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9770009c10 tx=0x7f9770010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fcac127d1a6ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.687+0000 7f9776a00640 1 -- 172.21.15.52:0/477150324 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9770004030 con 0x56161ddb5b90 2024-04-26T01:59:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.687+0000 7f978a200640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 0x7f976c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.687+0000 7f9776a00640 1 -- 172.21.15.52:0/477150324 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97700041d0 con 0x56161ddb5b90 2024-04-26T01:59:33.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.687+0000 7f9776a00640 1 -- 172.21.15.52:0/477150324 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700126b0 con 0x56161ddb5b90 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978a200640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 0x7f976c0a6250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f977c006010 tx=0x7f977c005ef0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 -- 172.21.15.52:0/477150324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 msgr2=0x56161ddb7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9770009c10 tx=0x7f9770010740 comp rx=0 tx=0).stop 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 -- 172.21.15.52:0/477150324 shutdown_connections 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb7f80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 --2- 172.21.15.52:0/477150324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddaad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 -- 172.21.15.52:0/477150324 >> 172.21.15.52:0/477150324 conn(0x56161dda6ec0 msgr2=0x56161dda72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 -- 172.21.15.52:0/477150324 shutdown_connections 2024-04-26T01:59:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.688+0000 7f978f051740 1 -- 172.21.15.52:0/477150324 wait complete. 2024-04-26T01:59:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.689+0000 7f978f051740 1 -- start start 2024-04-26T01:59:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.689+0000 7f978f051740 1 Processor -- start 2024-04-26T01:59:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56161ddb87c0 con 0x56161ddb5b90 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56161dcca310 con 0x56161ddab2d0 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56161ddb8940 con 0x56161ddaa9b0 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f9789800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f9789800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46328/0 (socket says 172.21.15.52:46328) 2024-04-26T01:59:33.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f9789800640 1 -- 172.21.15.52:0/3844583831 learned_addr learned my addr 172.21.15.52:0/3844583831 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.690+0000 7f978f051740 10 client.0 _ll_register_callbacks cb 0x56161dc98cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9788e00640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f978a200640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 -- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 msgr2=0x56161ddb3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 -- 172.21.15.52:0/3844583831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 msgr2=0x56161ddb35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 -- 172.21.15.52:0/3844583831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56161daef620 con 0x56161ddaa9b0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f978a200640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f9789800640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f977800e9c0 tx=0x7f9778005180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f31e81e4634d4511 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f976b400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97780ce880 con 0x56161ddaa9b0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f97780bd070 con 0x56161ddaa9b0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.691+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97700041d0 con 0x56161ddaa9b0 2024-04-26T01:59:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.692+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f97780a1000 con 0x56161ddaa9b0 2024-04-26T01:59:33.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.692+0000 7f978f051740 10 client.26845 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:33.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.692+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56161ddc4a90 con 0x56161ddaa9b0 2024-04-26T01:59:33.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.692+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97700041d0 con 0x56161ddaa9b0 2024-04-26T01:59:33.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.692+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f977809c0a0 con 0x56161ddaa9b0 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 10 client.26845 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 10 client.26845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 20 client.26845 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161ddc5f00 0x56161ddc82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 4 client.26845 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56161dd248b0 con 0x56161ddc5f00 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.693+0000 7f9788e00640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161ddc5f00 0x56161ddc82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.694+0000 7f9788e00640 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161ddc5f00 0x56161ddc82f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf446ab0ac8bb63 server_cookie=b3f395a9aa01c545 in_seq=0 out_seq=0 2024-04-26T01:59:33.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.694+0000 7f976b400640 10 client.26845 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.695+0000 7f976b400640 1 -- 172.21.15.52:0/3844583831 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56161dd248b0 con 0x56161ddc5f00 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.695+0000 7f976b400640 10 client.26845 handle_command_reply: tid=0 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.695+0000 7f978f051740 1 client.26845 shutdown 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 msgr2=0x56161ddb2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f977800e9c0 tx=0x7f9778005180 comp rx=0 tx=0).stop 2024-04-26T01:59:33.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 shutdown_connections 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161ddc5f00 0x56161ddc82f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddb5b90 0x56161ddb35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddb3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 --2- 172.21.15.52:0/3844583831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddaa9b0 0x56161ddb2b50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f976b400640 0 client.26845 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.696+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 >> 172.21.15.52:0/3844583831 conn(0x56161dda6ec0 msgr2=0x56161ddacbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:33.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.697+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 shutdown_connections 2024-04-26T01:59:33.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.697+0000 7f978f051740 1 -- 172.21.15.52:0/3844583831 wait complete. 2024-04-26T01:59:33.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.697+0000 7f978f051740 20 client.26845 trim_cache size 0 max 16384 2024-04-26T01:59:33.698 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:33 smithi052 ceph-mon[27851]: pgmap v459: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 7.3 MiB/s wr, 736 op/s 2024-04-26T01:59:33.712 INFO:tasks.workunit.client.0.smithi052.stdout: 10 29673 35.41 MB/sec warmup 50 sec latency 77.657 ms 2024-04-26T01:59:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.730+0000 7f978f051740 1 Processor -- start 2024-04-26T01:59:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 -- start start 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 0x56161ddc1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddcdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f97841ae240 con 0x56161ddab2d0 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f97841aba50 con 0x56161ddaa9b0 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f978f051740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97841a98c0 con 0x56161ddb5b90 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f9788e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f9789800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 0x56161ddc1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.731+0000 7f9788e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58704/0 (socket says 172.21.15.52:58704) 2024-04-26T01:59:33.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 -- 172.21.15.52:0/561151023 learned_addr learned my addr 172.21.15.52:0/561151023 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 -- 172.21.15.52:0/561151023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 msgr2=0x56161ddcdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddcdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 -- 172.21.15.52:0/561151023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 msgr2=0x56161ddc1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 0x56161ddc1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 -- 172.21.15.52:0/561151023 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56161ddc8920 con 0x56161ddab2d0 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9789800640 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 0x56161ddc1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.732+0000 7f9788e00640 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9770019000 tx=0x7f9770009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184fefbe6eca2328 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.733+0000 7f9776000640 1 -- 172.21.15.52:0/561151023 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700035f0 con 0x56161ddab2d0 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.733+0000 7f9776000640 1 -- 172.21.15.52:0/561151023 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97700054f0 con 0x56161ddab2d0 2024-04-26T01:59:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.733+0000 7f9776000640 1 -- 172.21.15.52:0/561151023 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9770028380 con 0x56161ddab2d0 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 -- 172.21.15.52:0/561151023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 msgr2=0x56161ddb4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9770019000 tx=0x7f9770009b70 comp rx=0 tx=0).stop 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 -- 172.21.15.52:0/561151023 shutdown_connections 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddcdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddab2d0 0x56161ddb4a90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 --2- 172.21.15.52:0/561151023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddaa9b0 0x56161ddc1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 -- 172.21.15.52:0/561151023 >> 172.21.15.52:0/561151023 conn(0x56161dda6ec0 msgr2=0x56161ddb4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 -- 172.21.15.52:0/561151023 shutdown_connections 2024-04-26T01:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.734+0000 7f978f051740 1 -- 172.21.15.52:0/561151023 wait complete. 2024-04-26T01:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.735+0000 7f978f051740 1 -- start start 2024-04-26T01:59:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.735+0000 7f978f051740 1 Processor -- start 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 0x56161ddbe060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddbb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f97841a9f10 con 0x56161ddaa9b0 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f97841ab380 con 0x56161ddab2d0 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.736+0000 7f978f051740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97841aa5e0 con 0x56161ddb5b90 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.737+0000 7f978a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.737+0000 7f978a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46334/0 (socket says 172.21.15.52:46334) 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.737+0000 7f978a200640 1 -- 172.21.15.52:0/3819046655 learned_addr learned my addr 172.21.15.52:0/3819046655 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:33.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.737+0000 7f9788e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddbb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.737+0000 7f9789800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 0x56161ddbe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978f051740 10 client.0 _ll_register_callbacks cb 0x7f977809c790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978a200640 1 -- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 msgr2=0x56161ddbb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978a200640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddbb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978a200640 1 -- 172.21.15.52:0/3819046655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 msgr2=0x56161ddbe060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978a200640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 0x56161ddbe060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f978a200640 1 -- 172.21.15.52:0/3819046655 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56161ddbc200 con 0x56161ddb5b90 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f9788e00640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddbb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.738+0000 7f9789800640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 0x56161ddbe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97700054f0 con 0x56161ddb5b90 2024-04-26T01:59:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f978a200640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f977c009ca0 tx=0x7f977c005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fe9a487c8bd96e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f976b400640 10 client.26848 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f976b400640 1 -- 172.21.15.52:0/3819046655 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c034040 con 0x56161ddb5b90 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f976b400640 1 -- 172.21.15.52:0/3819046655 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f977c048440 con 0x56161ddb5b90 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f976b400640 1 -- 172.21.15.52:0/3819046655 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f977c048640 con 0x56161ddb5b90 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.739+0000 7f978f051740 10 client.26848 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({fsmap=58}) -- 0x56161de3d460 con 0x56161ddb5b90 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f976b400640 1 -- 172.21.15.52:0/3819046655 <== mon.1 v2:172.21.15.191:3300/0 4 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f977c02f070 con 0x56161ddb5b90 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 10 client.26848 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 20 client.26848 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161de3dbe0 0x56161de40000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 4 client.26848 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.740+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56161dd1e320 con 0x56161de3dbe0 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.741+0000 7f9789800640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161de3dbe0 0x56161de40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.741+0000 7f9789800640 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161de3dbe0 0x56161de40000 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a5b37464bd1fd server_cookie=c909db8ea836c165 in_seq=0 out_seq=0 2024-04-26T01:59:33.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:33.741+0000 7f976b400640 10 client.26848 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:34.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 29744 34.79 MB/sec warmup 51 sec latency 739.070 ms 2024-04-26T01:59:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.724+0000 7f976b400640 1 -- 172.21.15.52:0/3819046655 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56161dd1e320 con 0x56161de3dbe0 2024-04-26T01:59:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.724+0000 7f976b400640 10 client.26848 handle_command_reply: tid=0 2024-04-26T01:59:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.724+0000 7f978f051740 1 client.26848 shutdown 2024-04-26T01:59:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.726+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 msgr2=0x56161ddbba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.726+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f977c009ca0 tx=0x7f977c005d80 comp rx=0 tx=0).stop 2024-04-26T01:59:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 shutdown_connections 2024-04-26T01:59:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56161de3dbe0 0x56161de40000 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56161ddb5b90 0x56161ddbba00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56161ddab2d0 0x56161ddbb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 --2- 172.21.15.52:0/3819046655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56161ddaa9b0 0x56161ddbe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.729+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 >> 172.21.15.52:0/3819046655 conn(0x56161dda6ec0 msgr2=0x56161ddb4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.730+0000 7f976b400640 0 client.26848 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.731+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 shutdown_connections 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.731+0000 7f978f051740 1 -- 172.21.15.52:0/3819046655 wait complete. 2024-04-26T01:59:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.731+0000 7f978f051740 20 client.26848 trim_cache size 0 max 16384 2024-04-26T01:59:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:59:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:59:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:59:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.732+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 msgr2=0x7f976c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.732+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 0x7f976c0a6250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f977c006010 tx=0x7f977c005ef0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.732+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 msgr2=0x7f9784073770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.732+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f977800e9f0 tx=0x7f977800edf0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 shutdown_connections 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f976c0a3d90 0x7f976c0a6250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9784106100 0x7f978406cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9784101620 0x7f978406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 --2- 172.21.15.52:0/2677645985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9784100d00 0x7f9784073770 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 >> 172.21.15.52:0/2677645985 conn(0x7f97840fc770 msgr2=0x7f97840fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 shutdown_connections 2024-04-26T01:59:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:34.733+0000 7f978ac00640 1 -- 172.21.15.52:0/2677645985 wait complete. 2024-04-26T01:59:35.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:35 smithi052 ceph-mon[27851]: pgmap v460: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 396 op/s 2024-04-26T01:59:35.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:35 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:35 smithi191 ceph-mon[32745]: pgmap v460: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 396 op/s 2024-04-26T01:59:35.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:35 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:35.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:35 smithi183 ceph-mon[32752]: pgmap v460: 97 pgs: 97 active+clean; 448 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.7 MiB/s wr, 396 op/s 2024-04-26T01:59:35.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:35 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T01:59:35.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 30084 34.41 MB/sec warmup 52 sec latency 454.500 ms 2024-04-26T01:59:36.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 31048 34.55 MB/sec warmup 53 sec latency 131.444 ms 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.432+0000 7f3fbec00700 1 -- 172.21.15.52:0/1484034301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb806a750 msgr2=0x7f3fb806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.432+0000 7f3fbec00700 1 --2- 172.21.15.52:0/1484034301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb806a750 0x7f3fb806abb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009990 tx=0x7f3fa4009ca0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 -- 172.21.15.52:0/1484034301 shutdown_connections 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 --2- 172.21.15.52:0/1484034301 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 --2- 172.21.15.52:0/1484034301 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb8108a30 0x7f3fb8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 --2- 172.21.15.52:0/1484034301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb806a750 0x7f3fb806abb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 -- 172.21.15.52:0/1484034301 >> 172.21.15.52:0/1484034301 conn(0x7f3fb80788f0 msgr2=0x7f3fb8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 -- 172.21.15.52:0/1484034301 shutdown_connections 2024-04-26T01:59:37.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.433+0000 7f3fbec00700 1 -- 172.21.15.52:0/1484034301 wait complete. 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.434+0000 7f3fbec00700 1 Processor -- start 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.434+0000 7f3fbec00700 1 -- start start 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 0x7f3fb819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb8112c20 con 0x7f3fb806b0f0 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb8112aa0 con 0x7f3fb8108a30 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbec00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb8112da0 con 0x7f3fb806a750 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbcc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.436+0000 7f3fb7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.436+0000 7f3fb7400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42372/0 (socket says 172.21.15.52:42372) 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.436+0000 7f3fb7400700 1 -- 172.21.15.52:0/810802449 learned_addr learned my addr 172.21.15.52:0/810802449 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.436+0000 7f3fb7e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 0x7f3fb819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:37.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.435+0000 7f3fbcc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34300/0 (socket says 172.21.15.52:34300) 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7400700 1 -- 172.21.15.52:0/810802449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 msgr2=0x7f3fb819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7400700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 0x7f3fb819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7400700 1 -- 172.21.15.52:0/810802449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 msgr2=0x7f3fb819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7400700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7400700 1 -- 172.21.15.52:0/810802449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb81a0330 con 0x7f3fb806b0f0 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fbcc00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.437+0000 7f3fb7e00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 0x7f3fb819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:37.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.438+0000 7f3fb7400700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4000c00 tx=0x7f3fa40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:37.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.439+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4027070 con 0x7f3fb806b0f0 2024-04-26T01:59:37.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.439+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa4022470 con 0x7f3fb806b0f0 2024-04-26T01:59:37.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.440+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4017650 con 0x7f3fb806b0f0 2024-04-26T01:59:37.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.440+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb81a05c0 con 0x7f3fb806b0f0 2024-04-26T01:59:37.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.440+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb81a7ea0 con 0x7f3fb806b0f0 2024-04-26T01:59:37.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.443+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb8104170 con 0x7f3fb806b0f0 2024-04-26T01:59:37.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.444+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3fa402a070 con 0x7f3fb806b0f0 2024-04-26T01:59:37.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.446+0000 7f3fb4c00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 0x7f3f9c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:37.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.447+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f3fa40d14c0 con 0x7f3fb806b0f0 2024-04-26T01:59:37.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.448+0000 7f3fb7e00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 0x7f3f9c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:37.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.449+0000 7f3fb7e00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 0x7f3f9c0a62f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009680 tx=0x7f3fa8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:37.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.452+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fa4013070 con 0x7f3fb806b0f0 2024-04-26T01:59:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:37 smithi191 ceph-mon[32745]: pgmap v461: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-26T01:59:37.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:37 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:37.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:37 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:37.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:37 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:59:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:37 smithi183 ceph-mon[32752]: pgmap v461: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-26T01:59:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:37 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:37 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:37 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:59:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:37 smithi052 ceph-mon[27851]: pgmap v461: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-26T01:59:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:37 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T01:59:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:37 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T01:59:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:37 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T01:59:37.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 32020 35.48 MB/sec warmup 54 sec latency 50.421 ms 2024-04-26T01:59:37.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.885+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fb819c6b0 con 0x7f3fb806b0f0 2024-04-26T01:59:37.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.887+0000 7f3fb4c00700 1 -- 172.21.15.52:0/810802449 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11891 (secure 0 0 0) 0x7f3fb819c6b0 con 0x7f3fb806b0f0 2024-04-26T01:59:37.885 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T01:59:37.886 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:59:07.564820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/3742874930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3742874930},{"type":"v1","addr":"172.21.15.183:6837","nonce":3742874930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 msgr2=0x7f3f9c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 0x7f3f9c0a62f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009680 tx=0x7f3fa8009260 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 msgr2=0x7f3fb819b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4000c00 tx=0x7f3fa40057c0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 shutdown_connections 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f3f9c0a3e30 0x7f3f9c0a62f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3fb8108a30 0x7f3fb819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fb806a750 0x7f3fb819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 --2- 172.21.15.52:0/810802449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fb806b0f0 0x7f3fb819b860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:37.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.891+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 >> 172.21.15.52:0/810802449 conn(0x7f3fb80788f0 msgr2=0x7f3fb80ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:37.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.892+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 shutdown_connections 2024-04-26T01:59:37.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:37.892+0000 7f3fbec00700 1 -- 172.21.15.52:0/810802449 wait complete. 2024-04-26T01:59:37.892 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 58 2024-04-26T01:59:38.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:38 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/810802449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:38.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:38 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/810802449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:38.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:38 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/810802449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T01:59:38.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 32422 35.08 MB/sec warmup 55 sec latency 45.745 ms 2024-04-26T01:59:38.779 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0' 2024-04-26T01:59:38.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0 2024-04-26T01:59:39.386 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T01:59:39.630 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:39 smithi052 ceph-mon[27851]: pgmap v462: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:39.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:39 smithi191 ceph-mon[32745]: pgmap v462: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:39.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:39 smithi183 ceph-mon[32752]: pgmap v462: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:39.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 32791 34.61 MB/sec warmup 56 sec latency 252.359 ms 2024-04-26T01:59:40.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 33000 34.21 MB/sec warmup 57 sec latency 635.834 ms 2024-04-26T01:59:41.574 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:41 smithi052 ceph-mon[27851]: pgmap v463: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:41 smithi191 ceph-mon[32745]: pgmap v463: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:41.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:41 smithi183 ceph-mon[32752]: pgmap v463: 97 pgs: 97 active+clean; 456 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 693 op/s 2024-04-26T01:59:41.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 33800 34.22 MB/sec warmup 58 sec latency 92.674 ms 2024-04-26T01:59:42.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 -- 172.21.15.52:0/1720139938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984074c70 msgr2=0x7fc984075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 --2- 172.21.15.52:0/1720139938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984074c70 0x7fc984075050 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc9740098f0 tx=0x7fc974009c00 comp rx=0 tx=0).stop 2024-04-26T01:59:42.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 -- 172.21.15.52:0/1720139938 shutdown_connections 2024-04-26T01:59:42.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 --2- 172.21.15.52:0/1720139938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc98410f4a0 0x7fc984111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 --2- 172.21.15.52:0/1720139938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984074c70 0x7fc984075050 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 --2- 172.21.15.52:0/1720139938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc984075590 0x7fc98410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.047+0000 7fc98b200700 1 -- 172.21.15.52:0/1720139938 >> 172.21.15.52:0/1720139938 conn(0x7fc984079b10 msgr2=0x7fc984079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.048+0000 7fc98b200700 1 -- 172.21.15.52:0/1720139938 shutdown_connections 2024-04-26T01:59:42.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.048+0000 7fc98b200700 1 -- 172.21.15.52:0/1720139938 wait complete. 2024-04-26T01:59:42.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.049+0000 7fc98b200700 1 Processor -- start 2024-04-26T01:59:42.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.050+0000 7fc98b200700 1 -- start start 2024-04-26T01:59:42.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc984074c70 0x7fc98419b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98410f4a0 0x7fc98419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc984114e30 con 0x7fc984074c70 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc984114cb0 con 0x7fc98410f4a0 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc98b200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc984114fb0 con 0x7fc984075590 2024-04-26T01:59:42.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc983400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc983400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50594/0 (socket says 172.21.15.52:50594) 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc983400700 1 -- 172.21.15.52:0/2241429380 learned_addr learned my addr 172.21.15.52:0/2241429380 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.051+0000 7fc989200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98410f4a0 0x7fc98419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983e00700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc984074c70 0x7fc98419b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 -- 172.21.15.52:0/2241429380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98410f4a0 msgr2=0x7fc98419faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98410f4a0 0x7fc98419faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 -- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc984074c70 msgr2=0x7fc98419b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc984074c70 0x7fc98419b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 -- 172.21.15.52:0/2241429380 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9841a0230 con 0x7fc984075590 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc983400700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9c0 tx=0x7fc97000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc980c00700 1 -- 172.21.15.52:0/2241429380 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97000cbd0 con 0x7fc984075590 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc980c00700 1 -- 172.21.15.52:0/2241429380 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97000cd70 con 0x7fc984075590 2024-04-26T01:59:42.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.052+0000 7fc980c00700 1 -- 172.21.15.52:0/2241429380 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970005580 con 0x7fc984075590 2024-04-26T01:59:42.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.053+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9841a0520 con 0x7fc984075590 2024-04-26T01:59:42.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.053+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc984102af0 con 0x7fc984075590 2024-04-26T01:59:42.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.057+0000 7fc980c00700 1 -- 172.21.15.52:0/2241429380 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc970004750 con 0x7fc984075590 2024-04-26T01:59:42.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.060+0000 7fc995c1cb80 1 Processor -- start 2024-04-26T01:59:42.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.061+0000 7fc995c1cb80 1 -- start start 2024-04-26T01:59:42.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b12d0 0x5632021bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021be570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563202104020 con 0x5632021bc180 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5632020aa8c0 con 0x5632021b12d0 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5632020fffe0 con 0x5632021b09b0 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc983e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc980c00700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 0x7fc9680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc983e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50598/0 (socket says 172.21.15.52:50598) 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc983e00700 1 -- 172.21.15.52:0/1841599792 learned_addr learned my addr 172.21.15.52:0/1841599792 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.062+0000 7fc980c00700 1 -- 172.21.15.52:0/2241429380 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fc9700d0a30 con 0x7fc984075590 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983400700 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b12d0 0x5632021bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc989200700 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983e00700 1 -- 172.21.15.52:0/1841599792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b12d0 msgr2=0x5632021bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983e00700 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b12d0 0x5632021bbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983e00700 1 -- 172.21.15.52:0/1841599792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 msgr2=0x5632021be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983e00700 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021be570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.063+0000 7fc983e00700 1 -- 172.21.15.52:0/1841599792 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632020a0810 con 0x5632021b09b0 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.064+0000 7fc989200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 0x7fc9680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.064+0000 7fc983e00700 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc9740098f0 tx=0x7fc97400f4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c27475abcde8fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.064+0000 7fc979600700 1 -- 172.21.15.52:0/1841599792 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974027070 con 0x5632021b09b0 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.064+0000 7fc979600700 1 -- 172.21.15.52:0/1841599792 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97400f980 con 0x5632021b09b0 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.064+0000 7fc979600700 1 -- 172.21.15.52:0/1841599792 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97400b560 con 0x5632021b09b0 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1841599792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 msgr2=0x5632021b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc9740098f0 tx=0x7fc97400f4e0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc989200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 0x7fc9680a6390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc97c006130 tx=0x7fc97c009040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1841599792 shutdown_connections 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b09b0 0x5632021b0d90 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc9740098f0 tx=0x7fc97400f4e0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1841599792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b12d0 0x5632021bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.065+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1841599792 >> 172.21.15.52:0/1841599792 conn(0x5632020df730 msgr2=0x5632021ace60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.066+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1841599792 shutdown_connections 2024-04-26T01:59:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.066+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1841599792 wait complete. 2024-04-26T01:59:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.066+0000 7fc995c1cb80 1 -- start start 2024-04-26T01:59:42.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 Processor -- start 2024-04-26T01:59:42.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021bba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021bc180 0x5632021b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5632021cfc50 con 0x5632021b12d0 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5632020bd9f0 con 0x5632021b92a0 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5632021cfdd0 con 0x5632021bc180 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.067+0000 7fc995c1cb80 10 client.0 _ll_register_callbacks cb 0x56320209c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34344/0 (socket says 172.21.15.52:34344) 2024-04-26T01:59:42.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 -- 172.21.15.52:0/1979228752 learned_addr learned my addr 172.21.15.52:0/1979228752 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 -- 172.21.15.52:0/1979228752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021bc180 msgr2=0x5632021b8d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021bc180 0x5632021b8d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 -- 172.21.15.52:0/1979228752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 msgr2=0x5632021bba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021bba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.068+0000 7fc989200700 1 -- 172.21.15.52:0/1979228752 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563202051a60 con 0x5632021b92a0 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.069+0000 7fc989200700 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc97c015dc0 tx=0x7fc97c007950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09bca23257c0fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.069+0000 7fc96e000700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.069+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c025020 con 0x5632021b92a0 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.069+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fc97c014d90 con 0x5632021b92a0 2024-04-26T01:59:42.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.069+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97400f980 con 0x5632021b92a0 2024-04-26T01:59:42.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.070+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc97c021450 con 0x5632021b92a0 2024-04-26T01:59:42.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.070+0000 7fc995c1cb80 10 client.26855 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:42.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.070+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56320209ac80 con 0x5632021b92a0 2024-04-26T01:59:42.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97400f980 con 0x5632021b92a0 2024-04-26T01:59:42.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fc97c032020 con 0x5632021b92a0 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 10 client.26855 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 10 client.26855 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 20 client.26855 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632021dd000 0x5632021df4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 4 client.26855 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563202111810 con 0x5632021dd000 2024-04-26T01:59:42.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.071+0000 7fc983e00700 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632021dd000 0x5632021df4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.072+0000 7fc96e000700 10 client.26855 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:42.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.073+0000 7fc983e00700 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632021dd000 0x5632021df4a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28eb523fbf1cf220 server_cookie=a73a9a4453e41c61 in_seq=0 out_seq=0 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.075+0000 7fc96e000700 1 -- 172.21.15.52:0/1979228752 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563202111810 con 0x5632021dd000 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.075+0000 7fc96e000700 10 client.26855 handle_command_reply: tid=0 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 client.26855 shutdown 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 msgr2=0x5632021b9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc97c015dc0 tx=0x7fc97c007950 comp rx=0 tx=0).stop 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 shutdown_connections 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5632021dd000 0x5632021df4a0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021b9720 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021bba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1979228752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021bc180 0x5632021b8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 >> 172.21.15.52:0/1979228752 conn(0x5632020df730 msgr2=0x5632021aca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.076+0000 7fc96e000700 0 client.26855 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:42.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.077+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 shutdown_connections 2024-04-26T01:59:42.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.077+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1979228752 wait complete. 2024-04-26T01:59:42.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.078+0000 7fc995c1cb80 20 client.26855 trim_cache size 0 max 16384 2024-04-26T01:59:42.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.113+0000 7fc995c1cb80 1 Processor -- start 2024-04-26T01:59:42.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.114+0000 7fc995c1cb80 1 -- start start 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.114+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021d8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.114+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.114+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021bc180 0x5632021e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841ada30 con 0x5632021b12d0 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841ab200 con 0x5632021bc180 2024-04-26T01:59:42.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841a9070 con 0x5632021b92a0 2024-04-26T01:59:42.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc983400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc983400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50612/0 (socket says 172.21.15.52:50612) 2024-04-26T01:59:42.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc983400700 1 -- 172.21.15.52:0/733200044 learned_addr learned my addr 172.21.15.52:0/733200044 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:42.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.115+0000 7fc989200700 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021bc180 0x5632021e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.116+0000 7fc983400700 1 -- 172.21.15.52:0/733200044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021bc180 msgr2=0x5632021e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.116+0000 7fc983400700 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021bc180 0x5632021e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.116+0000 7fc983400700 1 -- 172.21.15.52:0/733200044 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 msgr2=0x5632021d8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.116+0000 7fc983400700 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021d8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.116+0000 7fc983400700 1 -- 172.21.15.52:0/733200044 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632021b72d0 con 0x5632021b92a0 2024-04-26T01:59:42.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.118+0000 7fc983400700 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc970099340 tx=0x7fc970005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aeaab99dda7d2f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.118+0000 7fc978c00700 1 -- 172.21.15.52:0/733200044 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970005720 con 0x5632021b92a0 2024-04-26T01:59:42.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.120+0000 7fc978c00700 1 -- 172.21.15.52:0/733200044 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97009d070 con 0x5632021b92a0 2024-04-26T01:59:42.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.120+0000 7fc978c00700 1 -- 172.21.15.52:0/733200044 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9700ce3a0 con 0x5632021b92a0 2024-04-26T01:59:42.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 -- 172.21.15.52:0/733200044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 msgr2=0x5632021e4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc970099340 tx=0x7fc970005910 comp rx=0 tx=0).stop 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 -- 172.21.15.52:0/733200044 shutdown_connections 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021bc180 0x5632021e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021b12d0 0x5632021d8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/733200044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b92a0 0x5632021e4db0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.121+0000 7fc995c1cb80 1 -- 172.21.15.52:0/733200044 >> 172.21.15.52:0/733200044 conn(0x5632020df730 msgr2=0x5632021b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.122+0000 7fc995c1cb80 1 -- 172.21.15.52:0/733200044 shutdown_connections 2024-04-26T01:59:42.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.122+0000 7fc995c1cb80 1 -- 172.21.15.52:0/733200044 wait complete. 2024-04-26T01:59:42.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.123+0000 7fc995c1cb80 1 -- start start 2024-04-26T01:59:42.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.123+0000 7fc995c1cb80 1 Processor -- start 2024-04-26T01:59:42.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 0x5632021d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841a96c0 con 0x5632021bc180 2024-04-26T01:59:42.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841aab30 con 0x5632021b92a0 2024-04-26T01:59:42.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841a9d90 con 0x5632021b12d0 2024-04-26T01:59:42.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.124+0000 7fc995c1cb80 10 client.0 _ll_register_callbacks cb 0x56320209c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.125+0000 7fc983e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 0x5632021d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.125+0000 7fc983400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.125+0000 7fc983400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34362/0 (socket says 172.21.15.52:34362) 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.125+0000 7fc983400700 1 -- 172.21.15.52:0/1411247555 learned_addr learned my addr 172.21.15.52:0/1411247555 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.125+0000 7fc989200700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983400700 1 -- 172.21.15.52:0/1411247555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 msgr2=0x5632021d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983400700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 0x5632021d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983400700 1 -- 172.21.15.52:0/1411247555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 msgr2=0x5632021d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983400700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021d2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983400700 1 -- 172.21.15.52:0/1411247555 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563202243910 con 0x5632021b92a0 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.126+0000 7fc983e00700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 0x5632021d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.127+0000 7fc983400700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc97000cfd0 tx=0x7fc970098d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff16d358eac37a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.127+0000 7fc96e000700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:42.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.127+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9700bb020 con 0x5632021b92a0 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.128+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fc97009cdd0 con 0x5632021b92a0 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.128+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97009d070 con 0x5632021b92a0 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc97009c970 con 0x5632021b92a0 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 10 client.26861 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563202244bf0 con 0x5632021b92a0 2024-04-26T01:59:42.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc97009d070 con 0x5632021b92a0 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fc97009bdc0 con 0x5632021b92a0 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 10 client.26861 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 10 client.26861 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 20 client.26861 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.129+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563202245230 0x563202247650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.130+0000 7fc995c1cb80 4 client.26861 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}] 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.130+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}) -- 0x56320210b8b0 con 0x563202245230 2024-04-26T01:59:42.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.130+0000 7fc983e00700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563202245230 0x563202247650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:42.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.132+0000 7fc983e00700 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563202245230 0x563202247650 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9d9b3eda0049f38 server_cookie=28291258a0434393 in_seq=0 out_seq=0 2024-04-26T01:59:42.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.132+0000 7fc96e000700 10 client.26861 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:42.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.625+0000 7fc96e000700 1 -- 172.21.15.52:0/1411247555 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56320210b8b0 con 0x563202245230 2024-04-26T01:59:42.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.625+0000 7fc96e000700 10 client.26861 handle_command_reply: tid=0 2024-04-26T01:59:42.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.625+0000 7fc995c1cb80 1 client.26861 shutdown 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.625+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 msgr2=0x5632021d22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc97000cfd0 tx=0x7fc970098d30 comp rx=0 tx=0).stop 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 shutdown_connections 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563202245230 0x563202247650 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5632021bc180 0x5632021d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5632021b12d0 0x5632021d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 --2- 172.21.15.52:0/1411247555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5632021b92a0 0x5632021d22c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 >> 172.21.15.52:0/1411247555 conn(0x5632020df730 msgr2=0x5632021ba590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc96e000700 0 client.26861 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 shutdown_connections 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 1 -- 172.21.15.52:0/1411247555 wait complete. 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.626+0000 7fc995c1cb80 20 client.26861 trim_cache size 0 max 16384 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 58, 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 42, 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T01:59:42.626 INFO:teuthology.orchestra.run.smithi052.stdout: "c5916cc": { 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 42, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 84.6, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T01:59:42.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 msgr2=0x7fc9680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 0x7fc9680a6390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc97c006130 tx=0x7fc97c009040 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 msgr2=0x7fc98419b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9c0 tx=0x7fc97000ecd0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 shutdown_connections 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc9680a3ed0 0x7fc9680a6390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc98410f4a0 0x7fc98419faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc984074c70 0x7fc98419b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 --2- 172.21.15.52:0/2241429380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc984075590 0x7fc98419b760 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.627+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 >> 172.21.15.52:0/2241429380 conn(0x7fc984079b10 msgr2=0x7fc984100c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.628+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 shutdown_connections 2024-04-26T01:59:42.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T01:59:42.628+0000 7fc98b200700 1 -- 172.21.15.52:0/2241429380 wait complete. 2024-04-26T01:59:42.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 33.99 MB/sec warmup 59 sec latency 585.345 ms 2024-04-26T01:59:43.398 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c5916cc', quiesce took 15.7 seconds. Will release after: 24.6 2024-04-26T01:59:43.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:43 smithi191 ceph-mon[32745]: pgmap v464: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.09k op/s 2024-04-26T01:59:43.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:43 smithi183 ceph-mon[32752]: pgmap v464: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.09k op/s 2024-04-26T01:59:43.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:43 smithi052 ceph-mon[27851]: pgmap v464: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.09k op/s 2024-04-26T01:59:43.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 33.42 MB/sec warmup 60 sec latency 1585.437 ms 2024-04-26T01:59:44.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 32.87 MB/sec warmup 61 sec latency 2585.586 ms 2024-04-26T01:59:45.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:45 smithi191 ceph-mon[32745]: pgmap v465: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 19 MiB/s wr, 923 op/s 2024-04-26T01:59:45.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:45 smithi183 ceph-mon[32752]: pgmap v465: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 19 MiB/s wr, 923 op/s 2024-04-26T01:59:45.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:45 smithi052 ceph-mon[27851]: pgmap v465: 97 pgs: 97 active+clean; 471 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 19 MiB/s wr, 923 op/s 2024-04-26T01:59:45.713 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 32.34 MB/sec warmup 62 sec latency 3585.736 ms 2024-04-26T01:59:46.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 31.83 MB/sec warmup 63 sec latency 4585.886 ms 2024-04-26T01:59:47.650 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:47 smithi052 ceph-mon[27851]: pgmap v466: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 20 MiB/s wr, 1.03k op/s 2024-04-26T01:59:47.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:47 smithi191 ceph-mon[32745]: pgmap v466: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 20 MiB/s wr, 1.03k op/s 2024-04-26T01:59:47.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:47 smithi183 ceph-mon[32752]: pgmap v466: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 20 MiB/s wr, 1.03k op/s 2024-04-26T01:59:47.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 31.33 MB/sec warmup 64 sec latency 5586.022 ms 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 -- 172.21.15.52:0/2424276446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380100d60 msgr2=0x7f5380101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 --2- 172.21.15.52:0/2424276446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380100d60 0x7f5380101140 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5370009900 tx=0x7f537002a3f0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 -- 172.21.15.52:0/2424276446 shutdown_connections 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 --2- 172.21.15.52:0/2424276446 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 0x7f5380110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 --2- 172.21.15.52:0/2424276446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380101680 0x7f538010e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 --2- 172.21.15.52:0/2424276446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380100d60 0x7f5380101140 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 -- 172.21.15.52:0/2424276446 >> 172.21.15.52:0/2424276446 conn(0x7f5380077fd0 msgr2=0x7f53800783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 -- 172.21.15.52:0/2424276446 shutdown_connections 2024-04-26T01:59:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.884+0000 7f5386c00640 1 -- 172.21.15.52:0/2424276446 wait complete. 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.885+0000 7f5386c00640 1 Processor -- start 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.885+0000 7f5386c00640 1 -- start start 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380100d60 0x7f53801a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 0x7f538019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5380114430 con 0x7f5380101680 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53801142b0 con 0x7f538010e910 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5386c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f53801145b0 con 0x7f5380100d60 2024-04-26T01:59:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45496/0 (socket says 172.21.15.52:45496) 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 -- 172.21.15.52:0/1151111671 learned_addr learned my addr 172.21.15.52:0/1151111671 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537fe00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380100d60 0x7f53801a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f5384c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 0x7f538019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 -- 172.21.15.52:0/1151111671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380100d60 msgr2=0x7f53801a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380100d60 0x7f53801a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 -- 172.21.15.52:0/1151111671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 msgr2=0x7f538019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 0x7f538019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.886+0000 7f537f400640 1 -- 172.21.15.52:0/1151111671 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538019aff0 con 0x7f5380101680 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f537f400640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f536c00e9f0 tx=0x7f536c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f537cc00640 1 -- 172.21.15.52:0/1151111671 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c00cd80 con 0x7f5380101680 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538019b2e0 con 0x7f5380101680 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f537cc00640 1 -- 172.21.15.52:0/1151111671 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f536c0054d0 con 0x7f5380101680 2024-04-26T01:59:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f537cc00640 1 -- 172.21.15.52:0/1151111671 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c010690 con 0x7f5380101680 2024-04-26T01:59:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.887+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53801a75f0 con 0x7f5380101680 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 Processor -- start 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 -- start start 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f537cc00640 1 -- 172.21.15.52:0/1151111671 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f536c004e90 con 0x7f5380101680 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b44d007900 con 0x55b44d0a2140 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b44cfadef0 con 0x55b44d0a1820 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5389a00740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b44d0038c0 con 0x55b44d0aca00 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f537fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5384c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5384c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48422/0 (socket says 172.21.15.52:48422) 2024-04-26T01:59:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.891+0000 7f5384c00640 1 -- 172.21.15.52:0/337509210 learned_addr learned my addr 172.21.15.52:0/337509210 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f537f400640 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 -- 172.21.15.52:0/337509210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 msgr2=0x55b44d0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 -- 172.21.15.52:0/337509210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 msgr2=0x55b44d0ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0ac4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 -- 172.21.15.52:0/337509210 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44cd1a210 con 0x55b44d0a1820 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f537f400640 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.892+0000 7f5384c00640 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5378009f30 tx=0x7f53780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78fb08b2ee05e115 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f537cc00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 0x7f53640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f5375200640 1 -- 172.21.15.52:0/337509210 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378004d20 con 0x55b44d0a1820 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f537cc00640 1 -- 172.21.15.52:0/1151111671 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f536c0d0bd0 con 0x7f5380101680 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f5375200640 1 -- 172.21.15.52:0/337509210 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5378004050 con 0x55b44d0a1820 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f5375200640 1 -- 172.21.15.52:0/337509210 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378005280 con 0x55b44d0a1820 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.893+0000 7f537fe00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 0x7f53640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- 172.21.15.52:0/337509210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 msgr2=0x55b44d0a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5378009f30 tx=0x7f53780118d0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- 172.21.15.52:0/337509210 shutdown_connections 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 --2- 172.21.15.52:0/337509210 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a1c00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- 172.21.15.52:0/337509210 >> 172.21.15.52:0/337509210 conn(0x55b44d09dd30 msgr2=0x55b44d09e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f537fe00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 0x7f53640a6300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5370005ab0 tx=0x7f5370005a20 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- 172.21.15.52:0/337509210 shutdown_connections 2024-04-26T01:59:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- 172.21.15.52:0/337509210 wait complete. 2024-04-26T01:59:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.894+0000 7f5389a00740 1 -- start start 2024-04-26T01:59:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 Processor -- start 2024-04-26T01:59:47.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b44d0af630 con 0x55b44d0a2140 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b44cfc1310 con 0x55b44d0a1820 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b44d0af7b0 con 0x55b44d0aca00 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f5389a00740 10 client.0 _ll_register_callbacks cb 0x55b44cf8fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f537f400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f537f400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45510/0 (socket says 172.21.15.52:45510) 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.895+0000 7f537f400640 1 -- 172.21.15.52:0/3474539525 learned_addr learned my addr 172.21.15.52:0/3474539525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537fe00640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f5384c00640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 -- 172.21.15.52:0/3474539525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 msgr2=0x55b44d0aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 -- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 msgr2=0x55b44d0a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 -- 172.21.15.52:0/3474539525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44cde6620 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f537f400640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c6a0 tx=0x7f536c09ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3b10f74cc23fbd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f5369600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c09d210 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.896+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f536c0ce540 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5378004050 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f536c0ce740 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 10 client.16728 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55b44d0bb800 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5378004050 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f536c0b8070 con 0x55b44d0a2140 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 10 client.16728 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:47.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.897+0000 7f5389a00740 20 client.16728 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d0bcd80 0x55b44d0bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f5389a00740 4 client.16728 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b44d01b8b0 con 0x55b44d0bcd80 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f537fe00640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d0bcd80 0x55b44d0bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f537fe00640 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d0bcd80 0x55b44d0bf1b0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de199a6b4d85695 server_cookie=6f779e90eeb9c37f in_seq=0 out_seq=0 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.898+0000 7f5369600640 10 client.16728 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.899+0000 7f5369600640 1 -- 172.21.15.52:0/3474539525 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b44d01b8b0 con 0x55b44d0bcd80 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.899+0000 7f5369600640 10 client.16728 handle_command_reply: tid=0 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.899+0000 7f5389a00740 1 client.16728 shutdown 2024-04-26T01:59:47.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.899+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 msgr2=0x55b44d0a9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.899+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c6a0 tx=0x7f536c09ca10 comp rx=0 tx=0).stop 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 shutdown_connections 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d0bcd80 0x55b44d0bf1b0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a2140 0x55b44d0a9e70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 --2- 172.21.15.52:0/3474539525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a1820 0x55b44d0a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 >> 172.21.15.52:0/3474539525 conn(0x55b44d09dd30 msgr2=0x55b44d0a3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5369600640 0 client.16728 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 shutdown_connections 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 1 -- 172.21.15.52:0/3474539525 wait complete. 2024-04-26T01:59:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.900+0000 7f5389a00740 20 client.16728 trim_cache size 0 max 16384 2024-04-26T01:59:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.916+0000 7f5389a00740 1 Processor -- start 2024-04-26T01:59:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 -- start start 2024-04-26T01:59:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f53801b2710 con 0x55b44d0a1820 2024-04-26T01:59:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53801aff20 con 0x55b44d0a2140 2024-04-26T01:59:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5389a00740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f53801add90 con 0x55b44d0aca00 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f537fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f537fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45524/0 (socket says 172.21.15.52:45524) 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f537fe00640 1 -- 172.21.15.52:0/380786547 learned_addr learned my addr 172.21.15.52:0/380786547 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f537f400640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.917+0000 7f5384c00640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 -- 172.21.15.52:0/380786547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 msgr2=0x55b44d0c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 -- 172.21.15.52:0/380786547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 msgr2=0x55b44d0a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0a41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 -- 172.21.15.52:0/380786547 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44d0a4720 con 0x55b44d0a1820 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537f400640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f537fe00640 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5370009230 tx=0x7f537003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631f32051d9eb25d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.918+0000 7f536be00640 1 -- 172.21.15.52:0/380786547 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370035040 con 0x55b44d0a1820 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f536be00640 1 -- 172.21.15.52:0/380786547 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537005a070 con 0x55b44d0a1820 2024-04-26T01:59:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f536be00640 1 -- 172.21.15.52:0/380786547 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53700035f0 con 0x55b44d0a1820 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 -- 172.21.15.52:0/380786547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 msgr2=0x55b44d0b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5370009230 tx=0x7f537003f000 comp rx=0 tx=0).stop 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 -- 172.21.15.52:0/380786547 shutdown_connections 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0aca00 0x55b44d0c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 --2- 172.21.15.52:0/380786547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0a1820 0x55b44d0b84f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 -- 172.21.15.52:0/380786547 >> 172.21.15.52:0/380786547 conn(0x55b44d09dd30 msgr2=0x55b44d0aa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.919+0000 7f5389a00740 1 -- 172.21.15.52:0/380786547 shutdown_connections 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 -- 172.21.15.52:0/380786547 wait complete. 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 -- start start 2024-04-26T01:59:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 Processor -- start 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0a1820 0x55b44d0b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.920+0000 7f5389a00740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 0x55b44d0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5389a00740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f53801ae3e0 con 0x55b44d0aca00 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5389a00740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53801af850 con 0x55b44d0a2140 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5389a00740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f53801aeab0 con 0x55b44d0a1820 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5389a00740 10 client.0 _ll_register_callbacks cb 0x7f536c098e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5384c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 0x55b44d0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48446/0 (socket says 172.21.15.52:48446) 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 -- 172.21.15.52:0/3769784190 learned_addr learned my addr 172.21.15.52:0/3769784190 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f5384c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 0x55b44d0b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45532/0 (socket says 172.21.15.52:45532) 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537fe00640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0a1820 0x55b44d0b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 -- 172.21.15.52:0/3769784190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0a1820 msgr2=0x55b44d0b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0a1820 0x55b44d0b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 -- 172.21.15.52:0/3769784190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 msgr2=0x55b44d0b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 0x55b44d0b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.921+0000 7f537f400640 1 -- 172.21.15.52:0/3769784190 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44d1222b0 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f537f400640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f536c0097d0 tx=0x7f536c005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4459381be9819727 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5369600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c0ce740 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f537005a070 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7f536c09bf60 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f536c09cb10 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5389a00740 10 client.26879 fetch_fsmap learned FSMap version 58 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55b44d123500 con 0x55b44d0a2140 2024-04-26T01:59:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.922+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537005a070 con 0x55b44d0a2140 2024-04-26T01:59:47.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f536c0b8030 con 0x55b44d0a2140 2024-04-26T01:59:47.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 10 client.26879 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T01:59:47.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 10 client.26879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T01:59:47.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 20 client.26879 populate_metadata read hostname 'smithi052' 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d123d10 0x55b44d126130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 4 client.26879 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b44d015320 con 0x55b44d123d10 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.923+0000 7f537fe00640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d123d10 0x55b44d126130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.924+0000 7f537fe00640 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d123d10 0x55b44d126130 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31e2e0a384c7a0d server_cookie=9a9aaf7c5885d99e in_seq=0 out_seq=0 2024-04-26T01:59:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:47.924+0000 7f5369600640 10 client.26879 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.039+0000 7f5369600640 1 -- 172.21.15.52:0/3769784190 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b44d015320 con 0x55b44d123d10 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.039+0000 7f5369600640 10 client.26879 handle_command_reply: tid=0 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.039+0000 7f5389a00740 1 client.26879 shutdown 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.039+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 msgr2=0x55b44d0b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.039+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f536c0097d0 tx=0x7f536c005780 comp rx=0 tx=0).stop 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 shutdown_connections 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b44d123d10 0x55b44d126130 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b44d0aca00 0x55b44d0b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b44d0a2140 0x55b44d0b28d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5369600640 0 client.26879 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 --2- 172.21.15.52:0/3769784190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b44d0a1820 0x55b44d0b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 >> 172.21.15.52:0/3769784190 conn(0x55b44d09dd30 msgr2=0x55b44d09c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 shutdown_connections 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 1 -- 172.21.15.52:0/3769784190 wait complete. 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.040+0000 7f5389a00740 20 client.26879 trim_cache size 0 max 16384 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T01:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 msgr2=0x7f53640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 0x7f53640a6300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5370005ab0 tx=0x7f5370005a20 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 msgr2=0x7f53801a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f536c00e9f0 tx=0x7f536c00edf0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 shutdown_connections 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f53640a3e40 0x7f53640a6300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f538010e910 0x7f538019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5380101680 0x7f53801a0bb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 --2- 172.21.15.52:0/1151111671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5380100d60 0x7f53801a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 >> 172.21.15.52:0/1151111671 conn(0x7f5380077fd0 msgr2=0x7f538010fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T01:59:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 shutdown_connections 2024-04-26T01:59:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T01:59:48.041+0000 7f5386c00640 1 -- 172.21.15.52:0/1151111671 wait complete. 2024-04-26T01:59:48.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 30.85 MB/sec warmup 65 sec latency 6586.206 ms 2024-04-26T01:59:49.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:49 smithi191 ceph-mon[32745]: pgmap v467: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:49.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:49 smithi183 ceph-mon[32752]: pgmap v467: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:49 smithi052 ceph-mon[27851]: pgmap v467: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:49.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 30.38 MB/sec warmup 66 sec latency 7586.355 ms 2024-04-26T01:59:50.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 29.93 MB/sec warmup 67 sec latency 8586.504 ms 2024-04-26T01:59:51.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:51 smithi191 ceph-mon[32745]: pgmap v468: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:51 smithi183 ceph-mon[32752]: pgmap v468: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:51 smithi052 ceph-mon[27851]: pgmap v468: 97 pgs: 97 active+clean; 454 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 508 op/s 2024-04-26T01:59:51.714 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 29.49 MB/sec warmup 68 sec latency 9586.643 ms 2024-04-26T01:59:52.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 29.06 MB/sec warmup 69 sec latency 10586.795 ms 2024-04-26T01:59:53.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:53 smithi191 ceph-mon[32745]: pgmap v469: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 581 op/s 2024-04-26T01:59:53.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:53 smithi183 ceph-mon[32752]: pgmap v469: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 581 op/s 2024-04-26T01:59:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:53 smithi052 ceph-mon[27851]: pgmap v469: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.8 MiB/s wr, 581 op/s 2024-04-26T01:59:53.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 28.65 MB/sec warmup 70 sec latency 11586.941 ms 2024-04-26T01:59:54.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 28.24 MB/sec warmup 71 sec latency 12587.086 ms 2024-04-26T01:59:55.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:55 smithi191 ceph-mon[32745]: pgmap v470: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 688 KiB/s wr, 181 op/s 2024-04-26T01:59:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:55 smithi183 ceph-mon[32752]: pgmap v470: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 688 KiB/s wr, 181 op/s 2024-04-26T01:59:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:55 smithi052 ceph-mon[27851]: pgmap v470: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 688 KiB/s wr, 181 op/s 2024-04-26T01:59:55.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 27.85 MB/sec warmup 72 sec latency 13587.218 ms 2024-04-26T01:59:56.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 27.47 MB/sec warmup 73 sec latency 14587.353 ms 2024-04-26T01:59:57.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:57 smithi191 ceph-mon[32745]: pgmap v471: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 185 op/s 2024-04-26T01:59:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:57 smithi183 ceph-mon[32752]: pgmap v471: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 185 op/s 2024-04-26T01:59:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:57 smithi052 ceph-mon[27851]: pgmap v471: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 185 op/s 2024-04-26T01:59:57.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 27.10 MB/sec warmup 74 sec latency 15587.491 ms 2024-04-26T01:59:58.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 26.74 MB/sec warmup 75 sec latency 16587.602 ms 2024-04-26T01:59:59.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 01:59:59 smithi191 ceph-mon[32745]: pgmap v472: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T01:59:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 01:59:59 smithi183 ceph-mon[32752]: pgmap v472: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T01:59:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 01:59:59 smithi052 ceph-mon[27851]: pgmap v472: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T01:59:59.715 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 26.38 MB/sec warmup 76 sec latency 17587.735 ms 2024-04-26T02:00:00.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:00 smithi191 ceph-mon[32745]: overall HEALTH_OK 2024-04-26T02:00:00.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:00 smithi183 ceph-mon[32752]: overall HEALTH_OK 2024-04-26T02:00:00.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:00 smithi052 ceph-mon[27851]: overall HEALTH_OK 2024-04-26T02:00:00.716 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 26.04 MB/sec warmup 77 sec latency 18587.879 ms 2024-04-26T02:00:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 -- 172.21.15.52:0/186728483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68104d50 msgr2=0x7fec68108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 --2- 172.21.15.52:0/186728483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68104d50 0x7fec68108d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fec50009900 tx=0x7fec5002a3f0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 -- 172.21.15.52:0/186728483 shutdown_connections 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 --2- 172.21.15.52:0/186728483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec68111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 --2- 172.21.15.52:0/186728483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec68104d50 0x7fec68108d90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 --2- 172.21.15.52:0/186728483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec681043a0 0x7fec68104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.189+0000 7fec6ee00640 1 -- 172.21.15.52:0/186728483 >> 172.21.15.52:0/186728483 conn(0x7fec680ffc40 msgr2=0x7fec68102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.190+0000 7fec6ee00640 1 -- 172.21.15.52:0/186728483 shutdown_connections 2024-04-26T02:00:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.190+0000 7fec6ee00640 1 -- 172.21.15.52:0/186728483 wait complete. 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 Processor -- start 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 -- start start 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec681a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 0x7fec6819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fec681143b0 con 0x7fec6819a6f0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec68114230 con 0x7fec68109470 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.191+0000 7fec6ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec68114530 con 0x7fec681043a0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec681a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:51934/0 (socket says 172.21.15.52:51934) 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 -- 172.21.15.52:0/2156399067 learned_addr learned my addr 172.21.15.52:0/2156399067 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 0x7fec6819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 -- 172.21.15.52:0/2156399067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 msgr2=0x7fec681a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec681a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 -- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 msgr2=0x7fec6819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 0x7fec6819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 -- 172.21.15.52:0/2156399067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6819b2f0 con 0x7fec681043a0 2024-04-26T02:00:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 0x7fec6819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec5fe00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec681a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.192+0000 7fec67e00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec5800e9f0 tx=0x7fec5800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.193+0000 7fec65600640 1 -- 172.21.15.52:0/2156399067 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5800cd80 con 0x7fec681043a0 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.193+0000 7fec65600640 1 -- 172.21.15.52:0/2156399067 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec580108e0 con 0x7fec681043a0 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.193+0000 7fec65600640 1 -- 172.21.15.52:0/2156399067 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58018980 con 0x7fec681043a0 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.193+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec681a73d0 con 0x7fec681043a0 2024-04-26T02:00:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.193+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec681a7780 con 0x7fec681043a0 2024-04-26T02:00:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.196+0000 7fec65600640 1 -- 172.21.15.52:0/2156399067 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fec58010430 con 0x7fec681043a0 2024-04-26T02:00:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.196+0000 7fec71d74740 1 Processor -- start 2024-04-26T02:00:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.196+0000 7fec71d74740 1 -- start start 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555925331900 con 0x5559253d6a00 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5559252d7ef0 con 0x5559253cc140 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec71d74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55592532d8c0 con 0x5559253cb820 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec6ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec6ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:51940/0 (socket says 172.21.15.52:51940) 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec6ce00640 1 -- 172.21.15.52:0/3416417791 learned_addr learned my addr 172.21.15.52:0/3416417791 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec67e00640 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.197+0000 7fec5fe00640 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 -- 172.21.15.52:0/3416417791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 msgr2=0x5559253d64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 -- 172.21.15.52:0/3416417791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 msgr2=0x5559253d8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 -- 172.21.15.52:0/3416417791 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555925044210 con 0x5559253cb820 2024-04-26T02:00:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec6ce00640 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec60009fd0 tx=0x7fec60010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33e1192f2f1c07b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec65600640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 0x7fec4c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec5d600640 1 -- 172.21.15.52:0/3416417791 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60011700 con 0x5559253cb820 2024-04-26T02:00:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec5d600640 1 -- 172.21.15.52:0/3416417791 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec60011d50 con 0x5559253cb820 2024-04-26T02:00:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec65600640 1 -- 172.21.15.52:0/2156399067 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fec580d0bd0 con 0x7fec681043a0 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec5d600640 1 -- 172.21.15.52:0/3416417791 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6000e3c0 con 0x5559253cb820 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.198+0000 7fec67e00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 0x7fec4c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec67e00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 0x7fec4c0a6280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5559253ccdc0 tx=0x7fec580d4680 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 -- 172.21.15.52:0/3416417791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 msgr2=0x5559253cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec60009fd0 tx=0x7fec60010860 comp rx=0 tx=0).stop 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 -- 172.21.15.52:0/3416417791 shutdown_connections 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 --2- 172.21.15.52:0/3416417791 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253cbc00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 -- 172.21.15.52:0/3416417791 >> 172.21.15.52:0/3416417791 conn(0x5559253c7d30 msgr2=0x5559253c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 -- 172.21.15.52:0/3416417791 shutdown_connections 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.199+0000 7fec71d74740 1 -- 172.21.15.52:0/3416417791 wait complete. 2024-04-26T02:00:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 -- start start 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 Processor -- start 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.200+0000 7fec71d74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5559253d9630 con 0x5559253d6a00 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec71d74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5559252eb310 con 0x5559253cc140 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec71d74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5559253d97b0 con 0x5559253cb820 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec71d74740 10 client.0 _ll_register_callbacks cb 0x5559252b9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec6ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253d3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41876/0 (socket says 172.21.15.52:41876) 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 -- 172.21.15.52:0/2979244563 learned_addr learned my addr 172.21.15.52:0/2979244563 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 -- 172.21.15.52:0/2979244563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 msgr2=0x5559253d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253d3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 -- 172.21.15.52:0/2979244563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 msgr2=0x5559253d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec5fe00640 1 -- 172.21.15.52:0/2979244563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555925110620 con 0x5559253cc140 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.201+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec5fe00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec50009900 tx=0x7fec500316d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bf5e9ecabcaa650 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec56000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50035040 con 0x5559253cc140 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fec50004240 con 0x5559253cc140 2024-04-26T02:00:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec60011d50 con 0x5559253cc140 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fec50031900 con 0x5559253cc140 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec71d74740 10 client.26891 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.202+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5559253e5970 con 0x5559253cc140 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec60011d50 con 0x5559253cc140 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fec50004ad0 con 0x5559253cc140 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 10 client.26891 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 10 client.26891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 20 client.26891 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5559253e6e80 0x5559253e9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 4 client.26891 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559253458b0 con 0x5559253e6e80 2024-04-26T02:00:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.203+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5559253e6e80 0x5559253e9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.204+0000 7fec6ce00640 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5559253e6e80 0x5559253e9290 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6bd38538688c174 server_cookie=a45aa108e29c9bbb in_seq=0 out_seq=0 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.204+0000 7fec56000640 10 client.26891 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec56000640 1 -- 172.21.15.52:0/2979244563 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559253458b0 con 0x5559253e6e80 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec56000640 10 client.26891 handle_command_reply: tid=0 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 client.26891 shutdown 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 msgr2=0x5559253d3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec50009900 tx=0x7fec500316d0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 shutdown_connections 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5559253e6e80 0x5559253e9290 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d3f70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec56000640 0 client.26891 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 --2- 172.21.15.52:0/2979244563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253d3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 >> 172.21.15.52:0/2979244563 conn(0x5559253c7d30 msgr2=0x5559253cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 shutdown_connections 2024-04-26T02:00:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 1 -- 172.21.15.52:0/2979244563 wait complete. 2024-04-26T02:00:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.205+0000 7fec71d74740 20 client.26891 trim_cache size 0 max 16384 2024-04-26T02:00:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 Processor -- start 2024-04-26T02:00:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 -- start start 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fec681b2640 con 0x5559253d6a00 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec681afe50 con 0x5559253cc140 2024-04-26T02:00:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.223+0000 7fec71d74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec681adcc0 con 0x5559253cb820 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41884/0 (socket says 172.21.15.52:41884) 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 -- 172.21.15.52:0/1703301858 learned_addr learned my addr 172.21.15.52:0/1703301858 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 -- 172.21.15.52:0/1703301858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 msgr2=0x5559253e24f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253e24f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 -- 172.21.15.52:0/1703301858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 msgr2=0x5559253eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec5fe00640 1 -- 172.21.15.52:0/1703301858 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559253cf630 con 0x5559253cc140 2024-04-26T02:00:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.224+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.225+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fec50003640 tx=0x7fec5002a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92086f1b074d5024 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.225+0000 7fec5cc00640 1 -- 172.21.15.52:0/1703301858 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50035070 con 0x5559253cc140 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.225+0000 7fec5cc00640 1 -- 172.21.15.52:0/1703301858 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec50005000 con 0x5559253cc140 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.225+0000 7fec5cc00640 1 -- 172.21.15.52:0/1703301858 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50003dc0 con 0x5559253cc140 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- 172.21.15.52:0/1703301858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 msgr2=0x5559253d42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fec50003640 tx=0x7fec5002a3f0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- 172.21.15.52:0/1703301858 shutdown_connections 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253d6a00 0x5559253eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253cc140 0x5559253d42d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 --2- 172.21.15.52:0/1703301858 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- 172.21.15.52:0/1703301858 >> 172.21.15.52:0/1703301858 conn(0x5559253c7d30 msgr2=0x5559253d5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- 172.21.15.52:0/1703301858 shutdown_connections 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- 172.21.15.52:0/1703301858 wait complete. 2024-04-26T02:00:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.226+0000 7fec71d74740 1 -- start start 2024-04-26T02:00:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 Processor -- start 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253dc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 0x5559253dc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fec681ae310 con 0x5559253cc140 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec681af780 con 0x5559253d6a00 2024-04-26T02:00:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fec681ae9e0 con 0x5559253cb820 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec71d74740 10 client.0 _ll_register_callbacks cb 0x7fec500095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec6ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec6ce00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41898/0 (socket says 172.21.15.52:41898) 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec6ce00640 1 -- 172.21.15.52:0/1499540790 learned_addr learned my addr 172.21.15.52:0/1499540790 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec5fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 0x5559253dc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.227+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 -- 172.21.15.52:0/1499540790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 msgr2=0x5559253dc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253dc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 -- 172.21.15.52:0/1499540790 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 msgr2=0x5559253dc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 0x5559253dc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 -- 172.21.15.52:0/1499540790 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592544c3e0 con 0x5559253d6a00 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 0x5559253dc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec6ce00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fec60005cc0 tx=0x7fec60010fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20169892569b746a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.228+0000 7fec56000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.229+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60024040 con 0x5559253d6a00 2024-04-26T02:00:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.229+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec50005000 con 0x5559253d6a00 2024-04-26T02:00:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.230+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8521+0+0 (secure 0 0 0) 0x7fec600049c0 con 0x5559253d6a00 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.230+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fec6003b050 con 0x5559253d6a00 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.230+0000 7fec71d74740 10 client.26897 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.230+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55592544d640 con 0x5559253d6a00 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.230+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec50005000 con 0x5559253d6a00 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fec6002a040 con 0x5559253d6a00 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 10 client.26897 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 10 client.26897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 20 client.26897 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55592544dc50 0x555925450070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 4 client.26897 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.231+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55592533f320 con 0x55592544dc50 2024-04-26T02:00:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.232+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55592544dc50 0x555925450070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.233+0000 7fec5fe00640 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55592544dc50 0x555925450070 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95e7731dc4d205fb server_cookie=a9c6b407c1929c01 in_seq=0 out_seq=0 2024-04-26T02:00:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.233+0000 7fec56000640 10 client.26897 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec56000640 1 -- 172.21.15.52:0/1499540790 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55592533f320 con 0x55592544dc50 2024-04-26T02:00:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec56000640 10 client.26897 handle_command_reply: tid=0 2024-04-26T02:00:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 client.26897 shutdown 2024-04-26T02:00:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 msgr2=0x5559253dcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fec60005cc0 tx=0x7fec60010fd0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 shutdown_connections 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55592544dc50 0x555925450070 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5559253d6a00 0x5559253dcec0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5559253cc140 0x5559253dc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 --2- 172.21.15.52:0/1499540790 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559253cb820 0x5559253dc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec56000640 0 client.26897 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 >> 172.21.15.52:0/1499540790 conn(0x5559253c7d30 msgr2=0x5559253d5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 shutdown_connections 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.295+0000 7fec71d74740 1 -- 172.21.15.52:0/1499540790 wait complete. 2024-04-26T02:00:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec71d74740 20 client.26897 trim_cache size 0 max 16384 2024-04-26T02:00:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:00:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:00:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:00:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 msgr2=0x7fec4c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 0x7fec4c0a6280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5559253ccdc0 tx=0x7fec580d4680 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 msgr2=0x7fec681a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec5800e9f0 tx=0x7fec5800eeb0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 shutdown_connections 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fec4c0a3dc0 0x7fec4c0a6280 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fec6819a6f0 0x7fec6819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec68109470 0x7fec681a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 --2- 172.21.15.52:0/2156399067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec681043a0 0x7fec681a0630 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.296+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 >> 172.21.15.52:0/2156399067 conn(0x7fec680ffc40 msgr2=0x7fec68100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.297+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 shutdown_connections 2024-04-26T02:00:01.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:01.297+0000 7fec6ee00640 1 -- 172.21.15.52:0/2156399067 wait complete. 2024-04-26T02:00:01.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:01 smithi191 ceph-mon[32745]: pgmap v473: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T02:00:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:01 smithi183 ceph-mon[32752]: pgmap v473: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T02:00:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:01 smithi052 ceph-mon[27851]: pgmap v473: 97 pgs: 97 active+clean; 443 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 73 KiB/s wr, 77 op/s 2024-04-26T02:00:01.716 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 25.71 MB/sec warmup 78 sec latency 19588.034 ms 2024-04-26T02:00:02.716 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 25.38 MB/sec warmup 79 sec latency 20588.202 ms 2024-04-26T02:00:03.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:03 smithi191 ceph-mon[32745]: pgmap v474: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-04-26T02:00:03.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:03 smithi183 ceph-mon[32752]: pgmap v474: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-04-26T02:00:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:03 smithi052 ceph-mon[27851]: pgmap v474: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1.7 MiB/s wr, 126 op/s 2024-04-26T02:00:03.716 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 25.06 MB/sec warmup 80 sec latency 21588.347 ms 2024-04-26T02:00:04.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 24.75 MB/sec warmup 81 sec latency 22588.485 ms 2024-04-26T02:00:05.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:05 smithi191 ceph-mon[32745]: pgmap v475: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-04-26T02:00:05.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:05 smithi183 ceph-mon[32752]: pgmap v475: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-04-26T02:00:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:05 smithi052 ceph-mon[27851]: pgmap v475: 97 pgs: 97 active+clean; 452 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-04-26T02:00:05.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 24.45 MB/sec warmup 82 sec latency 23588.977 ms 2024-04-26T02:00:06.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 24.16 MB/sec warmup 83 sec latency 24589.045 ms 2024-04-26T02:00:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:07 smithi191 ceph-mon[32745]: pgmap v476: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 3.8 MiB/s wr, 129 op/s 2024-04-26T02:00:07.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:07 smithi191 ceph-mon[32745]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T02:00:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:07 smithi183 ceph-mon[32752]: pgmap v476: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 3.8 MiB/s wr, 129 op/s 2024-04-26T02:00:07.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:07 smithi183 ceph-mon[32752]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T02:00:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:07 smithi052 ceph-mon[27851]: pgmap v476: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 3.8 MiB/s wr, 129 op/s 2024-04-26T02:00:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:07 smithi052 ceph-mon[27851]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T02:00:07.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 23.87 MB/sec warmup 84 sec latency 25589.133 ms 2024-04-26T02:00:07.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:00:08.528 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:08.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 23.59 MB/sec warmup 85 sec latency 26589.238 ms 2024-04-26T02:00:09.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:09 smithi191 ceph-mon[32745]: pgmap v478: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:09.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:09 smithi183 ceph-mon[32752]: pgmap v478: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:09 smithi052 ceph-mon[27851]: pgmap v478: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:09.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 23.32 MB/sec warmup 86 sec latency 27589.324 ms 2024-04-26T02:00:10.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 23.05 MB/sec warmup 87 sec latency 28589.423 ms 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 -- 172.21.15.52:0/2892367726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb406afe0 msgr2=0x7f2eb406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/2892367726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb406afe0 0x7f2eb406b440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009990 tx=0x7f2ea0009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 -- 172.21.15.52:0/2892367726 shutdown_connections 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/2892367726 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406b980 0x7f2eb410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/2892367726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb4108a20 0x7f2eb4108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/2892367726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb406afe0 0x7f2eb406b440 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 -- 172.21.15.52:0/2892367726 >> 172.21.15.52:0/2892367726 conn(0x7f2eb40788b0 msgr2=0x7f2eb4078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 -- 172.21.15.52:0/2892367726 shutdown_connections 2024-04-26T02:00:10.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.873+0000 7f2eb9c00700 1 -- 172.21.15.52:0/2892367726 wait complete. 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.874+0000 7f2eb9c00700 1 Processor -- start 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.874+0000 7f2eb9c00700 1 -- start start 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 0x7f2eb419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4112c10 con 0x7f2eb4108a20 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4112a90 con 0x7f2eb406b980 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb9c00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4112d90 con 0x7f2eb406afe0 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb2a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb2a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53496/0 (socket says 172.21.15.52:53496) 2024-04-26T02:00:10.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45358/0 (socket says 172.21.15.52:45358) 2024-04-26T02:00:10.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb2a00700 1 -- 172.21.15.52:0/427536163 learned_addr learned my addr 172.21.15.52:0/427536163 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:10.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.875+0000 7f2eb2000700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 0x7f2eb419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 -- 172.21.15.52:0/427536163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 msgr2=0x7f2eb419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 -- 172.21.15.52:0/427536163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 msgr2=0x7f2eb419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 0x7f2eb419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 -- 172.21.15.52:0/427536163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb41a0320 con 0x7f2eb4108a20 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb2000700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 0x7f2eb419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb2a00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb3400700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00efc0 tx=0x7f2eac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac0092e0 con 0x7f2eb4108a20 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eac004bb0 con 0x7f2eb4108a20 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb41a05b0 con 0x7f2eb4108a20 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.876+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac004020 con 0x7f2eb4108a20 2024-04-26T02:00:10.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.877+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb41a7e90 con 0x7f2eb4108a20 2024-04-26T02:00:10.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.878+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e94005330 con 0x7f2eb4108a20 2024-04-26T02:00:10.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.880+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2eac01b030 con 0x7f2eb4108a20 2024-04-26T02:00:10.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.881+0000 7f2eab400700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 0x7f2e980a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:10.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.881+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8565+0+0 (secure 0 0 0) 0x7f2eac0d0170 con 0x7f2eb4108a20 2024-04-26T02:00:10.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.882+0000 7f2eb2a00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 0x7f2e980a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:10.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.882+0000 7f2eb2a00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 0x7f2e980a62f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009680 tx=0x7f2ea4009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:10.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:10.886+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2eac022020 con 0x7f2eb4108a20 2024-04-26T02:00:11.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.171+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e940058f0 con 0x7f2eb4108a20 2024-04-26T02:00:11.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.172+0000 7f2eab400700 1 -- 172.21.15.52:0/427536163 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11891 (secure 0 0 0) 0x7f2eac010040 con 0x7f2eb4108a20 2024-04-26T02:00:11.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:00:11.171 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:59:07.564820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/3742874930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3742874930},{"type":"v1","addr":"172.21.15.183:6837","nonce":3742874930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T02:00:11.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 msgr2=0x7f2e980a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:11.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 0x7f2e980a62f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009680 tx=0x7f2ea4009260 comp rx=0 tx=0).stop 2024-04-26T02:00:11.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 msgr2=0x7f2eb419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:11.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00efc0 tx=0x7f2eac00c3a0 comp rx=0 tx=0).stop 2024-04-26T02:00:11.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 shutdown_connections 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2e980a3e30 0x7f2e980a62f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2eb4108a20 0x7f2eb419fbe0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2eb406afe0 0x7f2eb419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 --2- 172.21.15.52:0/427536163 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb406b980 0x7f2eb419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.175+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 >> 172.21.15.52:0/427536163 conn(0x7f2eb40788b0 msgr2=0x7f2eb410def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.176+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 shutdown_connections 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:11.176+0000 7f2eb9c00700 1 -- 172.21.15.52:0/427536163 wait complete. 2024-04-26T02:00:11.175 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 58 2024-04-26T02:00:11.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:11 smithi052 ceph-mon[27851]: pgmap v479: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:11.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:11 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/427536163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:11.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:11 smithi183 ceph-mon[32752]: pgmap v479: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:11.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:11 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/427536163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:11.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 22.79 MB/sec warmup 88 sec latency 29589.535 ms 2024-04-26T02:00:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:11 smithi191 ceph-mon[32745]: pgmap v479: 97 pgs: 97 active+clean; 466 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 4.5 MiB/s wr, 151 op/s 2024-04-26T02:00:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:11 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/427536163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:12.043 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id c5916cc --release --await' 2024-04-26T02:00:12.043 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id c5916cc --release --await 2024-04-26T02:00:12.488 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:12.717 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 22.53 MB/sec warmup 89 sec latency 30589.624 ms 2024-04-26T02:00:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:13 smithi183 ceph-mon[32752]: pgmap v480: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-26T02:00:13.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:13 smithi183 ceph-mon[32752]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:13.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:13 smithi052 ceph-mon[27851]: pgmap v480: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-26T02:00:13.704 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:13 smithi052 ceph-mon[27851]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:13.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 22.28 MB/sec warmup 90 sec latency 31589.753 ms 2024-04-26T02:00:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:13 smithi191 ceph-mon[32745]: pgmap v480: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-26T02:00:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:13 smithi191 ceph-mon[32745]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T02:00:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 -- 172.21.15.52:0/1600547503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c109490 msgr2=0x7fdf1c111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 --2- 172.21.15.52:0/1600547503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c109490 0x7fdf1c111060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009860 tx=0x7fdf1402a340 comp rx=0 tx=0).stop 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 -- 172.21.15.52:0/1600547503 shutdown_connections 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 --2- 172.21.15.52:0/1600547503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c109490 0x7fdf1c111060 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 --2- 172.21.15.52:0/1600547503 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c104d70 0x7fdf1c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 --2- 172.21.15.52:0/1600547503 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c1043c0 0x7fdf1c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.435+0000 7fdf21200640 1 -- 172.21.15.52:0/1600547503 >> 172.21.15.52:0/1600547503 conn(0x7fdf1c0ffc20 msgr2=0x7fdf1c102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.436+0000 7fdf21200640 1 -- 172.21.15.52:0/1600547503 shutdown_connections 2024-04-26T02:00:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.436+0000 7fdf21200640 1 -- 172.21.15.52:0/1600547503 wait complete. 2024-04-26T02:00:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 Processor -- start 2024-04-26T02:00:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 -- start start 2024-04-26T02:00:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c1043c0 0x7fdf1c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c19a680 0x7fdf1c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1143e0 con 0x7fdf1c19a680 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c114260 con 0x7fdf1c109490 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf21200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c114560 con 0x7fdf1c1043c0 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c19a680 0x7fdf1c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.437+0000 7fdf19400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c1043c0 0x7fdf1c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37230/0 (socket says 172.21.15.52:37230) 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 -- 172.21.15.52:0/389897717 learned_addr learned my addr 172.21.15.52:0/389897717 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 -- 172.21.15.52:0/389897717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c1043c0 msgr2=0x7fdf1c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c1043c0 0x7fdf1c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 -- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c19a680 msgr2=0x7fdf1c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c19a680 0x7fdf1c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 -- 172.21.15.52:0/389897717 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1c19b280 con 0x7fdf1c109490 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.438+0000 7fdf19400640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800efd0 tx=0x7fdf0800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.439+0000 7fdf12a00640 1 -- 172.21.15.52:0/389897717 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08003b70 con 0x7fdf1c109490 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.439+0000 7fdf12a00640 1 -- 172.21.15.52:0/389897717 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf08003d10 con 0x7fdf1c109490 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.439+0000 7fdf12a00640 1 -- 172.21.15.52:0/389897717 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf080189d0 con 0x7fdf1c109490 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.439+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1c1a7360 con 0x7fdf1c109490 2024-04-26T02:00:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.439+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1c1a7710 con 0x7fdf1c109490 2024-04-26T02:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.443+0000 7fdf241b3740 1 Processor -- start 2024-04-26T02:00:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 -- start start 2024-04-26T02:00:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563e33b93900 con 0x563e33c2d820 2024-04-26T02:00:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563e33b39ef0 con 0x563e33c2e140 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.444+0000 7fdf241b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563e33b8f8c0 con 0x563e33c38a00 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf12a00640 1 -- 172.21.15.52:0/389897717 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdf08018b70 con 0x7fdf1c109490 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45384/0 (socket says 172.21.15.52:45384) 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 -- 172.21.15.52:0/1769972783 learned_addr learned my addr 172.21.15.52:0/1769972783 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf19e00640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf19400640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 -- 172.21.15.52:0/1769972783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 msgr2=0x563e33c3adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 -- 172.21.15.52:0/1769972783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 msgr2=0x563e33c384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf1a800640 1 -- 172.21.15.52:0/1769972783 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e338a6210 con 0x563e33c2d820 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf19e00640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.445+0000 7fdf19400640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.446+0000 7fdf1a800640 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdf140057c0 tx=0x7fdf14005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bca3ba984fe603f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.446+0000 7fdf07400640 1 -- 172.21.15.52:0/1769972783 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14041030 con 0x563e33c2d820 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.446+0000 7fdf07400640 1 -- 172.21.15.52:0/1769972783 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf14030c90 con 0x563e33c2d820 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.446+0000 7fdf07400640 1 -- 172.21.15.52:0/1769972783 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14037040 con 0x563e33c2d820 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.446+0000 7fdf12a00640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 0x7fdf000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf12a00640 1 -- 172.21.15.52:0/389897717 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fdf080d0f60 con 0x7fdf1c109490 2024-04-26T02:00:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf19e00640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 0x7fdf000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 -- 172.21.15.52:0/1769972783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 msgr2=0x563e33c2dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdf140057c0 tx=0x7fdf14005b20 comp rx=0 tx=0).stop 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 -- 172.21.15.52:0/1769972783 shutdown_connections 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1769972783 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c2dc00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.447+0000 7fdf241b3740 1 -- 172.21.15.52:0/1769972783 >> 172.21.15.52:0/1769972783 conn(0x563e33c29d30 msgr2=0x563e33c2a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.448+0000 7fdf241b3740 1 -- 172.21.15.52:0/1769972783 shutdown_connections 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.448+0000 7fdf19e00640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 0x7fdf000a6300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x563e33c2edc0 tx=0x7fdf0c009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.448+0000 7fdf241b3740 1 -- 172.21.15.52:0/1769972783 wait complete. 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.448+0000 7fdf241b3740 1 -- start start 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.448+0000 7fdf241b3740 1 Processor -- start 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2e140 0x563e33c35e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 0x563e33c363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563e33c3b5d0 con 0x563e33c2d820 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563e33b4d310 con 0x563e33c38a00 2024-04-26T02:00:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563e33c3b750 con 0x563e33c2e140 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf241b3740 10 client.0 _ll_register_callbacks cb 0x563e33b1bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45386/0 (socket says 172.21.15.52:45386) 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.449+0000 7fdf19e00640 1 -- 172.21.15.52:0/71294882 learned_addr learned my addr 172.21.15.52:0/71294882 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf1a800640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 0x563e33c363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 -- 172.21.15.52:0/71294882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2e140 msgr2=0x563e33c35e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2e140 0x563e33c35e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 -- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 msgr2=0x563e33c363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 0x563e33c363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 -- 172.21.15.52:0/71294882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e33972620 con 0x563e33c2d820 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf1a800640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 0x563e33c363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdf19e00640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c005c80 tx=0x7fdf0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f180f38ccbff99e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.450+0000 7fdefbe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c02d060 con 0x563e33c2d820 2024-04-26T02:00:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fdf0c018030 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf14030c90 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c028420 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdf241b3740 10 client.16767 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563e33c477a0 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.451+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf14030c90 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fdf0c01c430 con 0x563e33c2d820 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 10 client.16767 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 20 client.16767 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33c48cb0 0x563e33c4b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 4 client.16767 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e33ba78b0 con 0x563e33c48cb0 2024-04-26T02:00:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.452+0000 7fdf19400640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33c48cb0 0x563e33c4b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.453+0000 7fdf19400640 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33c48cb0 0x563e33c4b100 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c824a6d61bfb19a5 server_cookie=eee77e1a047588b5 in_seq=0 out_seq=0 2024-04-26T02:00:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.453+0000 7fdefbe00640 10 client.16767 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdefbe00640 1 -- 172.21.15.52:0/71294882 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e33ba78b0 con 0x563e33c48cb0 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdefbe00640 10 client.16767 handle_command_reply: tid=0 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 client.16767 shutdown 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 msgr2=0x563e33c35920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c005c80 tx=0x7fdf0c004730 comp rx=0 tx=0).stop 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 shutdown_connections 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33c48cb0 0x563e33c4b100 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c38a00 0x563e33c363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2e140 0x563e33c35e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 --2- 172.21.15.52:0/71294882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c35920 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdefbe00640 0 client.16767 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 >> 172.21.15.52:0/71294882 conn(0x563e33c29d30 msgr2=0x563e33c2fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 shutdown_connections 2024-04-26T02:00:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 1 -- 172.21.15.52:0/71294882 wait complete. 2024-04-26T02:00:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.454+0000 7fdf241b3740 20 client.16767 trim_cache size 0 max 16384 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 Processor -- start 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 -- start start 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c2f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c38a00 0x563e33c50a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1b2680 con 0x563e33c38a00 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1afe90 con 0x563e33c2e140 2024-04-26T02:00:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.471+0000 7fdf241b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1add00 con 0x563e33c2d820 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf1a800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c38a00 0x563e33c50a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53532/0 (socket says 172.21.15.52:53532) 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19e00640 1 -- 172.21.15.52:0/1564256411 learned_addr learned my addr 172.21.15.52:0/1564256411 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c2f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 -- 172.21.15.52:0/1564256411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 msgr2=0x563e33c44490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 -- 172.21.15.52:0/1564256411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c38a00 msgr2=0x563e33c50a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c38a00 0x563e33c50a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19400640 1 -- 172.21.15.52:0/1564256411 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e33c25400 con 0x563e33c2e140 2024-04-26T02:00:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.472+0000 7fdf19e00640 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.473+0000 7fdf19400640 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c2f8e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdf080ce650 tx=0x7fdf080d3420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba010192c81b797a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.473+0000 7fdf06a00640 1 -- 172.21.15.52:0/1564256411 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf080b5070 con 0x563e33c2e140 2024-04-26T02:00:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.473+0000 7fdf06a00640 1 -- 172.21.15.52:0/1564256411 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf0809cea0 con 0x563e33c2e140 2024-04-26T02:00:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.473+0000 7fdf06a00640 1 -- 172.21.15.52:0/1564256411 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf080b6270 con 0x563e33c2e140 2024-04-26T02:00:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- 172.21.15.52:0/1564256411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 msgr2=0x563e33c2f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c2f8e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdf080ce650 tx=0x7fdf080d3420 comp rx=0 tx=0).stop 2024-04-26T02:00:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- 172.21.15.52:0/1564256411 shutdown_connections 2024-04-26T02:00:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c38a00 0x563e33c50a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c2f8e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 --2- 172.21.15.52:0/1564256411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c2d820 0x563e33c44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- 172.21.15.52:0/1564256411 >> 172.21.15.52:0/1564256411 conn(0x563e33c29d30 msgr2=0x563e33c2f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- 172.21.15.52:0/1564256411 shutdown_connections 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- 172.21.15.52:0/1564256411 wait complete. 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.474+0000 7fdf241b3740 1 -- start start 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 Processor -- start 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c3e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1ae350 con 0x563e33c2d820 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1af7c0 con 0x563e33c2e140 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1c1aea20 con 0x563e33c38a00 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf241b3740 10 client.0 _ll_register_callbacks cb 0x7fdf0c01d880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.475+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45412/0 (socket says 172.21.15.52:45412) 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 -- 172.21.15.52:0/2496418284 learned_addr learned my addr 172.21.15.52:0/2496418284 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19400640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c3e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf1a800640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 -- 172.21.15.52:0/2496418284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 msgr2=0x563e33c3e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 -- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 msgr2=0x563e33c3e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c3e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 -- 172.21.15.52:0/2496418284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e33cae290 con 0x563e33c2d820 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19400640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c3e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdf19e00640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c013010 tx=0x7fdf0c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b1a671ad88d80c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.476+0000 7fdefbe00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0287d0 con 0x563e33c2d820 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf0809cea0 con 0x563e33c2d820 2024-04-26T02:00:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fdf0c028970 con 0x563e33c2d820 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c049050 con 0x563e33c2d820 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 10 client.16773 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563e33caf420 con 0x563e33c2d820 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf0809cea0 con 0x563e33c2d820 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fdf0c03a050 con 0x563e33c2d820 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 10 client.16773 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 10 client.16773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.477+0000 7fdf241b3740 20 client.16773 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33cafc00 0x563e33cb2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdf241b3740 4 client.16773 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e33ba1320 con 0x563e33cafc00 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdf19400640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33cafc00 0x563e33cb2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdf19400640 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33cafc00 0x563e33cb2020 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba666430c6e0d58e server_cookie=b80002d93178e94d in_seq=0 out_seq=0 2024-04-26T02:00:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.478+0000 7fdefbe00640 10 client.16773 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdefbe00640 1 -- 172.21.15.52:0/2496418284 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e33ba1320 con 0x563e33cafc00 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdefbe00640 10 client.16773 handle_command_reply: tid=0 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 client.16773 shutdown 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 msgr2=0x563e33c40ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c013010 tx=0x7fdf0c000bc0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 shutdown_connections 2024-04-26T02:00:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563e33cafc00 0x563e33cb2020 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563e33c38a00 0x563e33c3e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563e33c2e140 0x563e33c3e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdefbe00640 0 client.16773 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 --2- 172.21.15.52:0/2496418284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563e33c2d820 0x563e33c40ed0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 >> 172.21.15.52:0/2496418284 conn(0x563e33c29d30 msgr2=0x563e33c2fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 shutdown_connections 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.515+0000 7fdf241b3740 1 -- 172.21.15.52:0/2496418284 wait complete. 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf241b3740 20 client.16773 trim_cache size 0 max 16384 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:00:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:00:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:00:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 msgr2=0x7fdf000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 0x7fdf000a6300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x563e33c2edc0 tx=0x7fdf0c009260 comp rx=0 tx=0).stop 2024-04-26T02:00:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 msgr2=0x7fdf1c1a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800efd0 tx=0x7fdf0800c6a0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 shutdown_connections 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdf000a3e40 0x7fdf000a6300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf1c19a680 0x7fdf1c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf1c109490 0x7fdf1c1a0b00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 --2- 172.21.15.52:0/389897717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf1c1043c0 0x7fdf1c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.516+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 >> 172.21.15.52:0/389897717 conn(0x7fdf1c0ffc20 msgr2=0x7fdf1c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.517+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 shutdown_connections 2024-04-26T02:00:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:14.517+0000 7fdf21200640 1 -- 172.21.15.52:0/389897717 wait complete. 2024-04-26T02:00:14.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34139 22.03 MB/sec warmup 91 sec latency 32589.894 ms 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.908+0000 7faa73000700 1 -- 172.21.15.52:0/3660580804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c101940 msgr2=0x7faa6c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.908+0000 7faa73000700 1 --2- 172.21.15.52:0/3660580804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c101940 0x7faa6c101dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7faa54009990 tx=0x7faa54009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 -- 172.21.15.52:0/3660580804 shutdown_connections 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 --2- 172.21.15.52:0/3660580804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 0x7faa6c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 --2- 172.21.15.52:0/3660580804 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c107990 0x7faa6c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 --2- 172.21.15.52:0/3660580804 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c101940 0x7faa6c101dc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 -- 172.21.15.52:0/3660580804 >> 172.21.15.52:0/3660580804 conn(0x7faa6c0788f0 msgr2=0x7faa6c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 -- 172.21.15.52:0/3660580804 shutdown_connections 2024-04-26T02:00:14.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.909+0000 7faa73000700 1 -- 172.21.15.52:0/3660580804 wait complete. 2024-04-26T02:00:14.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.910+0000 7faa73000700 1 Processor -- start 2024-04-26T02:00:14.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.910+0000 7faa73000700 1 -- start start 2024-04-26T02:00:14.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c101940 0x7faa6c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 0x7faa6c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c111ae0 con 0x7faa6c107990 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c111960 con 0x7faa6c101940 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa73000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c111c60 con 0x7faa6c102300 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa71000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa71000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51948/0 (socket says 172.21.15.52:51948) 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa71000700 1 -- 172.21.15.52:0/2747895874 learned_addr learned my addr 172.21.15.52:0/2747895874 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.911+0000 7faa6be00700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c101940 0x7faa6c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa63400700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 0x7faa6c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 -- 172.21.15.52:0/2747895874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 msgr2=0x7faa6c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 0x7faa6c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 -- 172.21.15.52:0/2747895874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c101940 msgr2=0x7faa6c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c101940 0x7faa6c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 -- 172.21.15.52:0/2747895874 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6c19d000 con 0x7faa6c107990 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.912+0000 7faa71000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faa6400e940 tx=0x7faa6400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.913+0000 7faa69600700 1 -- 172.21.15.52:0/2747895874 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64004a30 con 0x7faa6c107990 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.913+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6c19d2f0 con 0x7faa6c107990 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.913+0000 7faa69600700 1 -- 172.21.15.52:0/2747895874 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa640054d0 con 0x7faa6c107990 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.913+0000 7faa69600700 1 -- 172.21.15.52:0/2747895874 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6400ccf0 con 0x7faa6c107990 2024-04-26T02:00:14.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.913+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6c1a4b70 con 0x7faa6c107990 2024-04-26T02:00:14.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.918+0000 7faa69600700 1 -- 172.21.15.52:0/2747895874 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7faa64005020 con 0x7faa6c107990 2024-04-26T02:00:14.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.919+0000 7faa7d892b80 1 Processor -- start 2024-04-26T02:00:14.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.919+0000 7faa7d892b80 1 -- start start 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.919+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b1f50 0x5583bf0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b2870 0x5583bf0bd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa7d892b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5583bf002f90 con 0x5583bf0b2870 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa7d892b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5583befa9830 con 0x5583bf0bd6c0 2024-04-26T02:00:14.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa7d892b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5583beffef50 con 0x5583bf0b1f50 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa69600700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 0x7faa500a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa63400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa63400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57216/0 (socket says 172.21.15.52:57216) 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa6be00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b2870 0x5583bf0bd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa63400700 1 -- 172.21.15.52:0/493310441 learned_addr learned my addr 172.21.15.52:0/493310441 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa69600700 1 -- 172.21.15.52:0/2747895874 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7faa640d0930 con 0x7faa6c107990 2024-04-26T02:00:14.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa63400700 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b1f50 0x5583bf0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.920+0000 7faa71000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 0x7faa500a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 -- 172.21.15.52:0/493310441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b1f50 msgr2=0x5583bf0b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b1f50 0x5583bf0b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 -- 172.21.15.52:0/493310441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b2870 msgr2=0x5583bf0bd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b2870 0x5583bf0bd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 -- 172.21.15.52:0/493310441 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583bef9f780 con 0x5583bf0bd6c0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa63400700 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa54000c00 tx=0x7faa54005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1948a991fdc715ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa61600700 1 -- 172.21.15.52:0/493310441 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5400b920 con 0x5583bf0bd6c0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa61600700 1 -- 172.21.15.52:0/493310441 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa54003680 con 0x5583bf0bd6c0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa61600700 1 -- 172.21.15.52:0/493310441 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5400f660 con 0x5583bf0bd6c0 2024-04-26T02:00:14.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.921+0000 7faa71000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 0x7faa500a62f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa640a4fb0 tx=0x7faa64009840 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 -- 172.21.15.52:0/493310441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 msgr2=0x5583bf0bfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa54000c00 tx=0x7faa54005e80 comp rx=0 tx=0).stop 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 -- 172.21.15.52:0/493310441 shutdown_connections 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0bd6c0 0x5583bf0bfab0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b1f50 0x5583bf0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 --2- 172.21.15.52:0/493310441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b2870 0x5583bf0bd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.922+0000 7faa7d892b80 1 -- 172.21.15.52:0/493310441 >> 172.21.15.52:0/493310441 conn(0x5583befdf050 msgr2=0x5583bf0ac140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.923+0000 7faa7d892b80 1 -- 172.21.15.52:0/493310441 shutdown_connections 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.923+0000 7faa7d892b80 1 -- 172.21.15.52:0/493310441 wait complete. 2024-04-26T02:00:14.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.923+0000 7faa7d892b80 1 -- start start 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.923+0000 7faa7d892b80 1 Processor -- start 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0bce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0ba100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0ba640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5583bf0c0150 con 0x5583bf0b1f50 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5583befbc960 con 0x5583bf0b2870 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5583bf0c02d0 con 0x5583bf0bd6c0 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa7d892b80 10 client.0 _ll_register_callbacks cb 0x5583bef9b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa63400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0bce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa63400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0bce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51966/0 (socket says 172.21.15.52:51966) 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa63400700 1 -- 172.21.15.52:0/3107758799 learned_addr learned my addr 172.21.15.52:0/3107758799 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa6be00700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.924+0000 7faa63400700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0ba640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 -- 172.21.15.52:0/3107758799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 msgr2=0x5583bf0ba640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0ba640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 -- 172.21.15.52:0/3107758799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 msgr2=0x5583bf0bce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0bce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 -- 172.21.15.52:0/3107758799 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583bef509d0 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa63400700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0ba640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa6be00700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0ba100 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faa5c012e10 tx=0x7faa5c014600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9619e424a373631 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa5a000700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.925+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c014b90 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7faa5c014d30 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa54003680 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7faa5c007500 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa7d892b80 10 client.26933 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5583bef99bf0 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.926+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa54003680 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7faa5c01f690 con 0x5583bf0b2870 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 10 client.26933 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:14.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 10 client.26933 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 20 client.26933 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf0ccec0 0x5583bf0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 4 client.26933 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583bf010780 con 0x5583bf0ccec0 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.927+0000 7faa63400700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf0ccec0 0x5583bf0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.928+0000 7faa63400700 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf0ccec0 0x5583bf0cf2d0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c44f5bd75b16d1d server_cookie=614dca61ef0e10bc in_seq=0 out_seq=0 2024-04-26T02:00:14.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.928+0000 7faa5a000700 10 client.26933 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa5a000700 1 -- 172.21.15.52:0/3107758799 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583bf010780 con 0x5583bf0ccec0 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa5a000700 10 client.26933 handle_command_reply: tid=0 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 client.26933 shutdown 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 msgr2=0x5583bf0ba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0ba100 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faa5c012e10 tx=0x7faa5c014600 comp rx=0 tx=0).stop 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 shutdown_connections 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf0ccec0 0x5583bf0cf2d0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0ba640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0bce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 --2- 172.21.15.52:0/3107758799 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0ba100 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 >> 172.21.15.52:0/3107758799 conn(0x5583befdf050 msgr2=0x5583bf0abd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa5a000700 0 client.26933 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 shutdown_connections 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.929+0000 7faa7d892b80 1 -- 172.21.15.52:0/3107758799 wait complete. 2024-04-26T02:00:14.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.930+0000 7faa7d892b80 20 client.26933 trim_cache size 0 max 16384 2024-04-26T02:00:14.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.951+0000 7faa7d892b80 1 Processor -- start 2024-04-26T02:00:14.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.951+0000 7faa7d892b80 1 -- start start 2024-04-26T02:00:14.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b1f50 0x5583bf0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 0x5583bf0d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1ada20 con 0x5583bf0bd6c0 2024-04-26T02:00:14.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1ab1f0 con 0x5583bf0b1f50 2024-04-26T02:00:14.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa7d892b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1a9060 con 0x5583bf0b2870 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa6be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa6be00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35148/0 (socket says 172.21.15.52:35148) 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa6be00700 1 -- 172.21.15.52:0/239731906 learned_addr learned my addr 172.21.15.52:0/239731906 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa63400700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b1f50 0x5583bf0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.952+0000 7faa63400700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 0x5583bf0d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 -- 172.21.15.52:0/239731906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b1f50 msgr2=0x5583bf0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b1f50 0x5583bf0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 -- 172.21.15.52:0/239731906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 msgr2=0x5583bf0d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 0x5583bf0d5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 -- 172.21.15.52:0/239731906 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583bf0ca8b0 con 0x5583bf0b2870 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa63400700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 0x5583bf0d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa6be00700 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faa5c000f80 tx=0x7faa5c009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec5289960c474be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa60c00700 1 -- 172.21.15.52:0/239731906 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c035070 con 0x5583bf0b2870 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa60c00700 1 -- 172.21.15.52:0/239731906 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa5c005c20 con 0x5583bf0b2870 2024-04-26T02:00:14.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.953+0000 7faa60c00700 1 -- 172.21.15.52:0/239731906 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c02c040 con 0x5583bf0b2870 2024-04-26T02:00:14.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 -- 172.21.15.52:0/239731906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 msgr2=0x5583bf0d4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:14.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faa5c000f80 tx=0x7faa5c009ba0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 -- 172.21.15.52:0/239731906 shutdown_connections 2024-04-26T02:00:14.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0bd6c0 0x5583bf0d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b1f50 0x5583bf0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 --2- 172.21.15.52:0/239731906 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0b2870 0x5583bf0d4be0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 -- 172.21.15.52:0/239731906 >> 172.21.15.52:0/239731906 conn(0x5583befdf050 msgr2=0x5583bf0abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.954+0000 7faa7d892b80 1 -- 172.21.15.52:0/239731906 shutdown_connections 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.955+0000 7faa7d892b80 1 -- 172.21.15.52:0/239731906 wait complete. 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.955+0000 7faa7d892b80 1 -- start start 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.955+0000 7faa7d892b80 1 Processor -- start 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0c2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1a96b0 con 0x5583bf0b1f50 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1aab20 con 0x5583bf0b2870 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faa6c1a9d80 con 0x5583bf0bd6c0 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa7d892b80 10 client.0 _ll_register_callbacks cb 0x5583bef9b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:14.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa63400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa63400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51974/0 (socket says 172.21.15.52:51974) 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa63400700 1 -- 172.21.15.52:0/1063186756 learned_addr learned my addr 172.21.15.52:0/1063186756 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.956+0000 7faa6be00700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 -- 172.21.15.52:0/1063186756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 msgr2=0x5583bf0c2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0c2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 -- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 msgr2=0x5583bf0c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0c1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 -- 172.21.15.52:0/1063186756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583bf0c2c10 con 0x5583bf0b1f50 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa63400700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faa540054b0 tx=0x7faa54005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5584fa3c447ea94f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:14.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa5a000700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:00:14.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5403a040 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7faa54004df0 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.957+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa5c005c20 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7faa5400b720 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 10 client.16791 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5583bf1345a0 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa5c005c20 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7faa5400f8b0 con 0x5583bf0b1f50 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 10 client.16791 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 10 client.16791 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 20 client.16791 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf134880 0x5583bf136c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 4 client.16791 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c5916cc", "release": true, "await": true}] 2024-04-26T02:00:14.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.958+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c5916cc", "release": true, "await": true}) -- 0x5583bf00a820 con 0x5583bf134880 2024-04-26T02:00:14.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.959+0000 7faa6be00700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf134880 0x5583bf136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:14.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.959+0000 7faa6be00700 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf134880 0x5583bf136c70 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=990da25db80971b6 server_cookie=629619474e57a66 in_seq=0 out_seq=0 2024-04-26T02:00:14.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:14.959+0000 7faa5a000700 10 client.16791 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:15.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.001+0000 7faa5a000700 1 -- 172.21.15.52:0/1063186756 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5583bf00a820 con 0x5583bf134880 2024-04-26T02:00:15.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.001+0000 7faa5a000700 10 client.16791 handle_command_reply: tid=0 2024-04-26T02:00:15.000 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.001+0000 7faa7d892b80 1 client.16791 shutdown 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.001+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 msgr2=0x5583bf0c5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.001+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faa540054b0 tx=0x7faa54005e80 comp rx=0 tx=0).stop 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 shutdown_connections 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5583bf134880 0x5583bf136c70 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5583bf0bd6c0 0x5583bf0c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5583bf0b1f50 0x5583bf0c5a20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 --2- 172.21.15.52:0/1063186756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5583bf0b2870 0x5583bf0c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 >> 172.21.15.52:0/1063186756 conn(0x5583befdf050 msgr2=0x5583bf0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa5a000700 0 client.16791 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 shutdown_connections 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.002+0000 7faa7d892b80 1 -- 172.21.15.52:0/1063186756 wait complete. 2024-04-26T02:00:15.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.003+0000 7faa7d892b80 20 client.16791 trim_cache size 0 max 16384 2024-04-26T02:00:15.004 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:00:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 58, 2024-04-26T02:00:15.004 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 44, 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "c5916cc": { 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 44, 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T02:00:15.005 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 84.6, 2024-04-26T02:00:15.006 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:15.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:15.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:15.008 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:00:15.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.004+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 msgr2=0x7faa500a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:15.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 0x7faa500a62f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa640a4fb0 tx=0x7faa64009840 comp rx=0 tx=0).stop 2024-04-26T02:00:15.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 msgr2=0x7faa6c19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:15.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faa6400e940 tx=0x7faa6400ec50 comp rx=0 tx=0).stop 2024-04-26T02:00:15.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 shutdown_connections 2024-04-26T02:00:15.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7faa500a3e30 0x7faa500a62f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faa6c107990 0x7faa6c19c8c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa6c101940 0x7faa6c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 --2- 172.21.15.52:0/2747895874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa6c102300 0x7faa6c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:15.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 >> 172.21.15.52:0/2747895874 conn(0x7faa6c0788f0 msgr2=0x7faa6c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:15.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.005+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 shutdown_connections 2024-04-26T02:00:15.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:15.006+0000 7faa73000700 1 -- 172.21.15.52:0/2747895874 wait complete. 2024-04-26T02:00:15.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:15 smithi183 ceph-mon[32752]: pgmap v482: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 273 KiB/s wr, 50 op/s 2024-04-26T02:00:15.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:15 smithi052 ceph-mon[27851]: pgmap v482: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 273 KiB/s wr, 50 op/s 2024-04-26T02:00:15.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 34753 22.19 MB/sec warmup 92 sec latency 32882.146 ms 2024-04-26T02:00:15.905 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:15 smithi191 ceph-mon[32745]: pgmap v482: 97 pgs: 97 active+clean; 442 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 273 KiB/s wr, 50 op/s 2024-04-26T02:00:15.907 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c5916cc', seconds elapsed: 48.2 2024-04-26T02:00:15.907 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.6 2024-04-26T02:00:15.907 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:00:16.465 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:16.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 35382 22.42 MB/sec warmup 93 sec latency 45.039 ms 2024-04-26T02:00:17.418 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:17 smithi052 ceph-mon[27851]: pgmap v483: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 837 KiB/s wr, 120 op/s 2024-04-26T02:00:17.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:17 smithi183 ceph-mon[32752]: pgmap v483: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 837 KiB/s wr, 120 op/s 2024-04-26T02:00:17.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 35995 22.51 MB/sec warmup 94 sec latency 92.060 ms 2024-04-26T02:00:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:17 smithi191 ceph-mon[32745]: pgmap v483: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 837 KiB/s wr, 120 op/s 2024-04-26T02:00:18.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 36674 22.59 MB/sec warmup 95 sec latency 45.760 ms 2024-04-26T02:00:19.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 -- 172.21.15.52:0/1468457344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec105890 msgr2=0x7f31ec1098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 --2- 172.21.15.52:0/1468457344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec105890 0x7f31ec1098e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009920 tx=0x7f31d8009c30 comp rx=0 tx=0).stop 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 -- 172.21.15.52:0/1468457344 shutdown_connections 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 --2- 172.21.15.52:0/1468457344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec109fc0 0x7f31ec111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 --2- 172.21.15.52:0/1468457344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec104ee0 0x7f31ec1052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 --2- 172.21.15.52:0/1468457344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec105890 0x7f31ec1098e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.072+0000 7f31f1000700 1 -- 172.21.15.52:0/1468457344 >> 172.21.15.52:0/1468457344 conn(0x7f31ec100770 msgr2=0x7f31ec102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.073+0000 7f31f1000700 1 -- 172.21.15.52:0/1468457344 shutdown_connections 2024-04-26T02:00:19.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.073+0000 7f31f1000700 1 -- 172.21.15.52:0/1468457344 wait complete. 2024-04-26T02:00:19.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.075+0000 7f31f1000700 1 Processor -- start 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.075+0000 7f31f1000700 1 -- start start 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec104ee0 0x7f31ec076f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec109fc0 0x7f31ec0779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec114ed0 con 0x7f31ec104ee0 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec114d50 con 0x7f31ec105890 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31f1000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec115050 con 0x7f31ec109fc0 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31e9400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31e9400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57266/0 (socket says 172.21.15.52:57266) 2024-04-26T02:00:19.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.076+0000 7f31e9400700 1 -- 172.21.15.52:0/2215284226 learned_addr learned my addr 172.21.15.52:0/2215284226 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.077+0000 7f31e9400700 1 -- 172.21.15.52:0/2215284226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec109fc0 msgr2=0x7f31ec0779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.077+0000 7f31e9400700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec109fc0 0x7f31ec0779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.077+0000 7f31e9400700 1 -- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec104ee0 msgr2=0x7f31ec076f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.077+0000 7f31e9400700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec104ee0 0x7f31ec076f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.077+0000 7f31e9400700 1 -- 172.21.15.52:0/2215284226 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec19b280 con 0x7f31ec105890 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31e9400700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009230 tx=0x7f31d800fab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8027070 con 0x7f31ec105890 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31d8005650 con 0x7f31ec105890 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ec19b510 con 0x7f31ec105890 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8004930 con 0x7f31ec105890 2024-04-26T02:00:19.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.078+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ec19b9a0 con 0x7f31ec105890 2024-04-26T02:00:19.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.080+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ec1063c0 con 0x7f31ec105890 2024-04-26T02:00:19.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.081+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f31d802a070 con 0x7f31ec105890 2024-04-26T02:00:19.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.084+0000 7f31e2a00700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 0x7f31d00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:19.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.084+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f31d80d11c0 con 0x7f31ec105890 2024-04-26T02:00:19.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.084+0000 7f31e9e00700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 0x7f31d00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:19.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.085+0000 7f31e9e00700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 0x7f31d00a62f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31dc005950 tx=0x7f31dc0058e0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:19.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.088+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31d805eff0 con 0x7f31ec105890 2024-04-26T02:00:19.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.573+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31ec078430 con 0x7f31ec105890 2024-04-26T02:00:19.574 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:00:19.577 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:59:07.564820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/3742874930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3742874930},{"type":"v1","addr":"172.21.15.183:6837","nonce":3742874930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T02:00:19.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.575+0000 7f31e2a00700 1 -- 172.21.15.52:0/2215284226 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11891 (secure 0 0 0) 0x7f31d8013030 con 0x7f31ec105890 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 msgr2=0x7f31d00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 0x7f31d00a62f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31dc005950 tx=0x7f31dc0058e0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 msgr2=0x7f31ec077490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009230 tx=0x7f31d800fab0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 shutdown_connections 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f31d00a3e30 0x7f31d00a62f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec109fc0 0x7f31ec0779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31ec104ee0 0x7f31ec076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 --2- 172.21.15.52:0/2215284226 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31ec105890 0x7f31ec077490 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.579+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 >> 172.21.15.52:0/2215284226 conn(0x7f31ec100770 msgr2=0x7f31ec102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.580+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 shutdown_connections 2024-04-26T02:00:19.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:19.580+0000 7f31f1000700 1 -- 172.21.15.52:0/2215284226 wait complete. 2024-04-26T02:00:19.580 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 58 2024-04-26T02:00:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:19 smithi183 ceph-mon[32752]: pgmap v484: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:19 smithi052 ceph-mon[27851]: pgmap v484: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:19.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 37484 22.82 MB/sec warmup 96 sec latency 50.922 ms 2024-04-26T02:00:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:19 smithi191 ceph-mon[32745]: pgmap v484: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:20.444 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0' 2024-04-26T02:00:20.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 84.6 --await-for 42.0 2024-04-26T02:00:20.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:20 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2215284226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:20.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 38296 23.09 MB/sec warmup 97 sec latency 60.098 ms 2024-04-26T02:00:20.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:20 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2215284226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:20.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:20 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2215284226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:21.013 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:21.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:21 smithi052 ceph-mon[27851]: pgmap v485: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:21.718 INFO:tasks.workunit.client.0.smithi052.stdout: 10 38883 23.15 MB/sec warmup 98 sec latency 47.795 ms 2024-04-26T02:00:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:21 smithi191 ceph-mon[32745]: pgmap v485: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:21 smithi183 ceph-mon[32752]: pgmap v485: 97 pgs: 97 active+clean; 448 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 825 KiB/s wr, 118 op/s 2024-04-26T02:00:22.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 39528 23.28 MB/sec warmup 99 sec latency 66.505 ms 2024-04-26T02:00:23.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:23 smithi052 ceph-mon[27851]: pgmap v486: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 3.8 MiB/s wr, 354 op/s 2024-04-26T02:00:23.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40276 23.40 MB/sec warmup 100 sec latency 61.997 ms 2024-04-26T02:00:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:23 smithi191 ceph-mon[32745]: pgmap v486: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 3.8 MiB/s wr, 354 op/s 2024-04-26T02:00:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:23 smithi183 ceph-mon[32752]: pgmap v486: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 3.8 MiB/s wr, 354 op/s 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 -- 172.21.15.52:0/573087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84102300 msgr2=0x7f0f8410e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 --2- 172.21.15.52:0/573087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84102300 0x7f0f8410e800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00b0a0 tx=0x7f0f7c00b3b0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 -- 172.21.15.52:0/573087896 shutdown_connections 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 --2- 172.21.15.52:0/573087896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84102300 0x7f0f8410e800 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 --2- 172.21.15.52:0/573087896 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84107930 0x7f0f84107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 --2- 172.21.15.52:0/573087896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84101940 0x7f0f84101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.948+0000 7f0f8a600700 1 -- 172.21.15.52:0/573087896 >> 172.21.15.52:0/573087896 conn(0x7f0f840788e0 msgr2=0x7f0f84078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:23.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.950+0000 7f0f8a600700 1 -- 172.21.15.52:0/573087896 shutdown_connections 2024-04-26T02:00:23.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.950+0000 7f0f8a600700 1 -- 172.21.15.52:0/573087896 wait complete. 2024-04-26T02:00:23.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.951+0000 7f0f8a600700 1 Processor -- start 2024-04-26T02:00:23.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.952+0000 7f0f8a600700 1 -- start start 2024-04-26T02:00:23.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 0x7f0f8410fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84107930 0x7f0f8410ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f84114e20 con 0x7f0f84102300 2024-04-26T02:00:23.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f84114ca0 con 0x7f0f84107930 2024-04-26T02:00:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f8a600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f84114fa0 con 0x7f0f84101940 2024-04-26T02:00:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f83400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f83400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35184/0 (socket says 172.21.15.52:35184) 2024-04-26T02:00:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.953+0000 7f0f83400700 1 -- 172.21.15.52:0/1652535407 learned_addr learned my addr 172.21.15.52:0/1652535407 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:23.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.954+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84107930 0x7f0f8410ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.954+0000 7f0f83400700 1 -- 172.21.15.52:0/1652535407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84107930 msgr2=0x7f0f8410ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:23.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.954+0000 7f0f83400700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84107930 0x7f0f8410ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.954+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 0x7f0f8410fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.955+0000 7f0f83400700 1 -- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 msgr2=0x7f0f8410fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:23.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.955+0000 7f0f83400700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 0x7f0f8410fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.955+0000 7f0f83400700 1 -- 172.21.15.52:0/1652535407 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f841ab8c0 con 0x7f0f84101940 2024-04-26T02:00:23.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.955+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 0x7f0f8410fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:00:23.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f83400700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0f7400d9b0 tx=0x7f0f7400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:23.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f7be00700 1 -- 172.21.15.52:0/1652535407 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74016070 con 0x7f0f84101940 2024-04-26T02:00:23.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f7be00700 1 -- 172.21.15.52:0/1652535407 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f74004830 con 0x7f0f84101940 2024-04-26T02:00:23.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f7be00700 1 -- 172.21.15.52:0/1652535407 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74005500 con 0x7f0f84101940 2024-04-26T02:00:23.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f841abbb0 con 0x7f0f84101940 2024-04-26T02:00:23.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.956+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f841ac0a0 con 0x7f0f84101940 2024-04-26T02:00:23.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.962+0000 7f0f7be00700 1 -- 172.21.15.52:0/1652535407 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0f74004220 con 0x7f0f84101940 2024-04-26T02:00:23.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.963+0000 7f0f7be00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 0x7f0f680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.963+0000 7f0f7be00700 1 -- 172.21.15.52:0/1652535407 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f0f740d0580 con 0x7f0f84101940 2024-04-26T02:00:23.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.964+0000 7f0f94f09b80 1 Processor -- start 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.964+0000 7f0f94f09b80 1 -- start start 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a46f0 0x56293a5af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56293a4f7020 con 0x56293a5a3dd0 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56293a49d8c0 con 0x56293a5a46f0 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56293a4f2fe0 con 0x56293a5af560 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 0x7f0f680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f83400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f83e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f83400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5a41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51998/0 (socket says 172.21.15.52:51998) 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.965+0000 7f0f83400700 1 -- 172.21.15.52:0/1111462766 learned_addr learned my addr 172.21.15.52:0/1111462766 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a46f0 0x56293a5af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 -- 172.21.15.52:0/1111462766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a46f0 msgr2=0x56293a5af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a46f0 0x56293a5af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 -- 172.21.15.52:0/1111462766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 msgr2=0x56293a5a41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5a41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 -- 172.21.15.52:0/1111462766 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293a493810 con 0x56293a5af560 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b1950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0062a0 tx=0x7f0f7c009180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43aa3104a648c4ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.966+0000 7f0f78c00700 1 -- 172.21.15.52:0/1111462766 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c020070 con 0x56293a5af560 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.967+0000 7f0f78c00700 1 -- 172.21.15.52:0/1111462766 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f7c0074e0 con 0x56293a5af560 2024-04-26T02:00:23.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.967+0000 7f0f78c00700 1 -- 172.21.15.52:0/1111462766 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c005280 con 0x56293a5af560 2024-04-26T02:00:23.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.967+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1111462766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 msgr2=0x56293a5b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:23.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.967+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b1950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0062a0 tx=0x7f0f7c009180 comp rx=0 tx=0).stop 2024-04-26T02:00:23.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1111462766 shutdown_connections 2024-04-26T02:00:23.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 0x7f0f680a6390 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005d90 tx=0x7f0f70012670 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:23.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b1950 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5a41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1111462766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a46f0 0x56293a5af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.969+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1111462766 >> 172.21.15.52:0/1111462766 conn(0x56293a4d30e0 msgr2=0x56293a5a02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:23.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.970+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1111462766 shutdown_connections 2024-04-26T02:00:23.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.970+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1111462766 wait complete. 2024-04-26T02:00:23.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.971+0000 7f0f94f09b80 1 -- start start 2024-04-26T02:00:23.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.971+0000 7f0f94f09b80 1 Processor -- start 2024-04-26T02:00:23.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5abf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56293a5b1ff0 con 0x56293a5a3dd0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56293a4b09f0 con 0x56293a5af560 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56293a5b2170 con 0x56293a5a46f0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f94f09b80 10 client.0 _ll_register_callbacks cb 0x56293a48f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.972+0000 7f0f83400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57304/0 (socket says 172.21.15.52:57304) 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 -- 172.21.15.52:0/1884590615 learned_addr learned my addr 172.21.15.52:0/1884590615 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f82a00700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 -- 172.21.15.52:0/1884590615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 msgr2=0x56293a5abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:23.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 -- 172.21.15.52:0/1884590615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 msgr2=0x56293a5aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:23.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.973+0000 7f0f83e00700 1 -- 172.21.15.52:0/1884590615 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293a444a60 con 0x56293a5af560 2024-04-26T02:00:23.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.974+0000 7f0f83e00700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00b3f0 tx=0x7f0f7c0097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35adb06de57e0428 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:23.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.974+0000 7f0f6d600700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:23.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.974+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c020070 con 0x56293a5af560 2024-04-26T02:00:23.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.974+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f0f7c007960 con 0x56293a5af560 2024-04-26T02:00:23.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.974+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7c0074e0 con 0x56293a5af560 2024-04-26T02:00:23.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.975+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0f7c007b60 con 0x56293a5af560 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.975+0000 7f0f94f09b80 10 client.26954 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.975+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56293a48dc80 con 0x56293a5af560 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.975+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f7c0074e0 con 0x56293a5af560 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f0f7c011070 con 0x56293a5af560 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 10 client.26954 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 10 client.26954 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:23.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 20 client.26954 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:23.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a5beaf0 0x56293a5c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:23.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 4 client.26954 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:23.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.976+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56293a504810 con 0x56293a5beaf0 2024-04-26T02:00:23.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.977+0000 7f0f83400700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a5beaf0 0x56293a5c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:23.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.978+0000 7f0f83400700 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a5beaf0 0x56293a5c0ee0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41e025aabe467ba server_cookie=6cceb3f539d81bac in_seq=0 out_seq=0 2024-04-26T02:00:23.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.979+0000 7f0f6d600700 10 client.26954 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.980+0000 7f0f6d600700 1 -- 172.21.15.52:0/1884590615 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56293a504810 con 0x56293a5beaf0 2024-04-26T02:00:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.980+0000 7f0f6d600700 10 client.26954 handle_command_reply: tid=0 2024-04-26T02:00:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.980+0000 7f0f94f09b80 1 client.26954 shutdown 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.980+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 msgr2=0x56293a5ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.980+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00b3f0 tx=0x7f0f7c0097c0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 shutdown_connections 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a5beaf0 0x56293a5c0ee0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f6d600700 0 client.26954 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5ac470 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/1884590615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 >> 172.21.15.52:0/1884590615 conn(0x56293a4d30e0 msgr2=0x56293a59fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 shutdown_connections 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 1 -- 172.21.15.52:0/1884590615 wait complete. 2024-04-26T02:00:23.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:23.981+0000 7f0f94f09b80 20 client.26954 trim_cache size 0 max 16384 2024-04-26T02:00:24.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.022+0000 7f0f94f09b80 1 Processor -- start 2024-04-26T02:00:24.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.022+0000 7f0f94f09b80 1 -- start start 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5baef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b6200 con 0x56293a5a3dd0 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b39d0 con 0x56293a5af560 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b1840 con 0x56293a5a46f0 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57310/0 (socket says 172.21.15.52:57310) 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 -- 172.21.15.52:0/3357748671 learned_addr learned my addr 172.21.15.52:0/3357748671 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:24.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83400700 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5baef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 -- 172.21.15.52:0/3357748671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 msgr2=0x56293a5c64b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5c64b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 -- 172.21.15.52:0/3357748671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 msgr2=0x56293a5baef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5baef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.023+0000 7f0f83e00700 1 -- 172.21.15.52:0/3357748671 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293a5b1d80 con 0x56293a5af560 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.024+0000 7f0f83400700 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5baef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.024+0000 7f0f83e00700 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c005200 tx=0x7f0f7c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a4b3b33390ae46c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.024+0000 7f0f6fe00700 1 -- 172.21.15.52:0/3357748671 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c03a070 con 0x56293a5af560 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.024+0000 7f0f6fe00700 1 -- 172.21.15.52:0/3357748671 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f7c0074a0 con 0x56293a5af560 2024-04-26T02:00:24.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.024+0000 7f0f6fe00700 1 -- 172.21.15.52:0/3357748671 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c020070 con 0x56293a5af560 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 -- 172.21.15.52:0/3357748671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 msgr2=0x56293a5c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c005200 tx=0x7f0f7c000f40 comp rx=0 tx=0).stop 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 -- 172.21.15.52:0/3357748671 shutdown_connections 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5af560 0x56293a5c69f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a3dd0 0x56293a5baef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/3357748671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5a46f0 0x56293a5c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.025+0000 7f0f94f09b80 1 -- 172.21.15.52:0/3357748671 >> 172.21.15.52:0/3357748671 conn(0x56293a4d30e0 msgr2=0x56293a59fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.026+0000 7f0f94f09b80 1 -- 172.21.15.52:0/3357748671 shutdown_connections 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.026+0000 7f0f94f09b80 1 -- 172.21.15.52:0/3357748671 wait complete. 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.026+0000 7f0f94f09b80 1 -- start start 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 Processor -- start 2024-04-26T02:00:24.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a46f0 0x56293a5b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b1e90 con 0x56293a5a46f0 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b3300 con 0x56293a5a3dd0 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841b2560 con 0x56293a5af560 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.027+0000 7f0f94f09b80 10 client.0 _ll_register_callbacks cb 0x56293a48f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:24.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.028+0000 7f0f83400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.028+0000 7f0f83400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:57318/0 (socket says 172.21.15.52:57318) 2024-04-26T02:00:24.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.028+0000 7f0f83400700 1 -- 172.21.15.52:0/191643760 learned_addr learned my addr 172.21.15.52:0/191643760 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:24.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.028+0000 7f0f83e00700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.028+0000 7f0f82a00700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a46f0 0x56293a5b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83e00700 1 -- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 msgr2=0x56293a5b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83e00700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83e00700 1 -- 172.21.15.52:0/191643760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a46f0 msgr2=0x56293a5b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83e00700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a46f0 0x56293a5b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83e00700 1 -- 172.21.15.52:0/191643760 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293a624fb0 con 0x56293a5af560 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.029+0000 7f0f83400700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.030+0000 7f0f83e00700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b2e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00b3f0 tx=0x7f0f7c005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25f3cb43397cb498 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:24.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.030+0000 7f0f6d600700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:00:24.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.031+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c03a030 con 0x56293a5af560 2024-04-26T02:00:24.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.031+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7c0074a0 con 0x56293a5af560 2024-04-26T02:00:24.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.031+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f0f7c00f070 con 0x56293a5af560 2024-04-26T02:00:24.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.032+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0f7c044050 con 0x56293a5af560 2024-04-26T02:00:24.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.032+0000 7f0f94f09b80 10 client.26953 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:24.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.032+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56293a626230 con 0x56293a5af560 2024-04-26T02:00:24.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.032+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f7c0074a0 con 0x56293a5af560 2024-04-26T02:00:24.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.033+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f0f7c011040 con 0x56293a5af560 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 10 client.26953 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 10 client.26953 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 20 client.26953 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a626880 0x56293a628c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 4 client.26953 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}] 2024-04-26T02:00:24.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.6, "await_for": 42.0}) -- 0x56293a4fe8b0 con 0x56293a626880 2024-04-26T02:00:24.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.034+0000 7f0f83400700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a626880 0x56293a628c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:24.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.035+0000 7f0f83400700 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a626880 0x56293a628c70 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4261a10925dc6d7 server_cookie=6df4e2e698e74329 in_seq=0 out_seq=0 2024-04-26T02:00:24.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.035+0000 7f0f6d600700 10 client.26953 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.484+0000 7f0f6d600700 1 -- 172.21.15.52:0/191643760 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56293a4fe8b0 con 0x56293a626880 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.484+0000 7f0f6d600700 10 client.26953 handle_command_reply: tid=0 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.484+0000 7f0f94f09b80 1 client.26953 shutdown 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 msgr2=0x56293a5b2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b2e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00b3f0 tx=0x7f0f7c005cf0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 shutdown_connections 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56293a626880 0x56293a628c70 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56293a5af560 0x56293a5b2e40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56293a5a3dd0 0x56293a5b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 --2- 172.21.15.52:0/191643760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56293a5a46f0 0x56293a5b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 >> 172.21.15.52:0/191643760 conn(0x56293a4d30e0 msgr2=0x56293a5aa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f6d600700 0 client.26953 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 shutdown_connections 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 1 -- 172.21.15.52:0/191643760 wait complete. 2024-04-26T02:00:24.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.485+0000 7f0f94f09b80 20 client.26953 trim_cache size 0 max 16384 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 58, 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 46, 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "860f4752": { 2024-04-26T02:00:24.486 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 46, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 84.6, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCED", 2024-04-26T02:00:24.487 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 msgr2=0x7f0f680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:24.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 0x7f0f680a6390 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005d90 tx=0x7f0f70012670 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 msgr2=0x7f0f8410f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0f7400d9b0 tx=0x7f0f7400dcc0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 shutdown_connections 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0f680a3ed0 0x7f0f680a6390 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f84107930 0x7f0f8410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0f84101940 0x7f0f8410f4c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 --2- 172.21.15.52:0/1652535407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f84102300 0x7f0f8410fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 >> 172.21.15.52:0/1652535407 conn(0x7f0f840788e0 msgr2=0x7f0f840ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.486+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 shutdown_connections 2024-04-26T02:00:24.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:24.487+0000 7f0f8a600700 1 -- 172.21.15.52:0/1652535407 wait complete. 2024-04-26T02:00:24.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 23.33 MB/sec warmup 101 sec latency 665.931 ms 2024-04-26T02:00:25.354 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '860f4752', quiesce took 9.4 seconds. Will release after: 24.6 2024-04-26T02:00:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:25 smithi052 ceph-mon[27851]: pgmap v487: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 326 op/s 2024-04-26T02:00:25.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 23.10 MB/sec warmup 102 sec latency 1666.089 ms 2024-04-26T02:00:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:25 smithi191 ceph-mon[32745]: pgmap v487: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 326 op/s 2024-04-26T02:00:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:25 smithi183 ceph-mon[32752]: pgmap v487: 97 pgs: 97 active+clean; 476 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 326 op/s 2024-04-26T02:00:26.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 22.87 MB/sec warmup 103 sec latency 2666.223 ms 2024-04-26T02:00:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.666+0000 7f4e39000640 1 -- 172.21.15.52:0/1868805268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e341031e0 msgr2=0x7f4e341035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.666+0000 7f4e39000640 1 --2- 172.21.15.52:0/1868805268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e341031e0 0x7f4e341035c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009ca0 tx=0x7f4e2402a9d0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 -- 172.21.15.52:0/1868805268 shutdown_connections 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 --2- 172.21.15.52:0/1868805268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 0x7f4e3410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 --2- 172.21.15.52:0/1868805268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34103b00 0x7f4e3410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 --2- 172.21.15.52:0/1868805268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e341031e0 0x7f4e341035c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 -- 172.21.15.52:0/1868805268 >> 172.21.15.52:0/1868805268 conn(0x7f4e34077fd0 msgr2=0x7f4e340783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 -- 172.21.15.52:0/1868805268 shutdown_connections 2024-04-26T02:00:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.667+0000 7f4e39000640 1 -- 172.21.15.52:0/1868805268 wait complete. 2024-04-26T02:00:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 Processor -- start 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 -- start start 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 0x7f4e341a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 0x7f4e3419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341121b0 con 0x7f4e34103b00 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4e34112030 con 0x7f4e341031e0 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.668+0000 7f4e39000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4e34112330 con 0x7f4e3410c520 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e29400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 0x7f4e341a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e29400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38000/0 (socket says 172.21.15.52:38000) 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e29400640 1 -- 172.21.15.52:0/3081482714 learned_addr learned my addr 172.21.15.52:0/3081482714 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e32800640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 0x7f4e3419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 -- 172.21.15.52:0/3081482714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 msgr2=0x7f4e3419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 0x7f4e3419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 -- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 msgr2=0x7f4e341a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e31e00640 1 -- 172.21.15.52:0/3081482714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3419af90 con 0x7f4e341031e0 2024-04-26T02:00:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.669+0000 7f4e29400640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 0x7f4e341a0610 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006010 tx=0x7f4e2402efe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e2b400640 1 -- 172.21.15.52:0/3081482714 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24047070 con 0x7f4e341031e0 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3419b220 con 0x7f4e341031e0 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e2b400640 1 -- 172.21.15.52:0/3081482714 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2402e830 con 0x7f4e341031e0 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e2b400640 1 -- 172.21.15.52:0/3081482714 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24033050 con 0x7f4e341031e0 2024-04-26T02:00:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.670+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e341a7590 con 0x7f4e341031e0 2024-04-26T02:00:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.673+0000 7f4e3bf96740 1 Processor -- start 2024-04-26T02:00:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.673+0000 7f4e2b400640 1 -- 172.21.15.52:0/3081482714 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4e2404a070 con 0x7f4e341031e0 2024-04-26T02:00:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.673+0000 7f4e3bf96740 1 -- start start 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x565232bd4900 con 0x565232c79a00 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x565232b7aef0 con 0x565232c6e820 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x565232bd08c0 con 0x565232c6f140 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e31e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e32800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e29400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e32800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:52600/0 (socket says 172.21.15.52:52600) 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e31e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:42356/0 (socket says 172.21.15.52:42356) 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e29400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c7bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38004/0 (socket says 172.21.15.52:38004) 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.674+0000 7f4e32800640 1 -- 172.21.15.52:0/3915855274 learned_addr learned my addr 172.21.15.52:0/3915855274 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 -- 172.21.15.52:0/3915855274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 msgr2=0x565232c794c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 -- 172.21.15.52:0/3915855274 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 msgr2=0x565232c7bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c7bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 -- 172.21.15.52:0/3915855274 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652328e7210 con 0x565232c6e820 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e32800640 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009c20 tx=0x7f4e2c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7590a959034dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e2b400640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 0x7f4e180a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e23400640 1 -- 172.21.15.52:0/3915855274 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01e070 con 0x565232c6e820 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e2b400640 1 -- 172.21.15.52:0/3081482714 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f4e240f0d90 con 0x7f4e341031e0 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e23400640 1 -- 172.21.15.52:0/3915855274 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c019ae0 con 0x565232c6e820 2024-04-26T02:00:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e23400640 1 -- 172.21.15.52:0/3915855274 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c00e960 con 0x565232c6e820 2024-04-26T02:00:27.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.675+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 0x7f4e180a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3915855274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 msgr2=0x565232c6ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009c20 tx=0x7f4e2c011810 comp rx=0 tx=0).stop 2024-04-26T02:00:27.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e31e00640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 0x7f4e180a6300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x565232c6fdc0 tx=0x7f4e240f4840 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3915855274 shutdown_connections 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c7bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6f140 0x565232c794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3915855274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c6ec00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3915855274 >> 172.21.15.52:0/3915855274 conn(0x565232c6ad30 msgr2=0x565232c6b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3915855274 shutdown_connections 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.676+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3915855274 wait complete. 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 -- start start 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 Processor -- start 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c769b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c77430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x565232c7c630 con 0x565232c79a00 2024-04-26T02:00:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x565232b8e310 con 0x565232c6f140 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.677+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x565232c7c7b0 con 0x565232c6e820 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e3bf96740 10 client.0 _ll_register_callbacks cb 0x565232b5ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:52610/0 (socket says 172.21.15.52:52610) 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e32800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 -- 172.21.15.52:0/1233543271 learned_addr learned my addr 172.21.15.52:0/1233543271 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e32800640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c77430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 -- 172.21.15.52:0/1233543271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 msgr2=0x565232c769b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c769b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 -- 172.21.15.52:0/1233543271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 msgr2=0x565232c77430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c77430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e29400640 1 -- 172.21.15.52:0/1233543271 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652329b3620 con 0x565232c6f140 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.678+0000 7f4e32800640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c77430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e29400640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c012d20 tx=0x7f4e1c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c567a9b09bcfb97f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e17e00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c012110 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f4e1c004050 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e2c019ae0 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f4e1c0122b0 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e3bf96740 10 client.26972 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.679+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x565232c88950 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c019ae0 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f4e1c010450 con 0x565232c6f140 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 10 client.26972 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 10 client.26972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 20 client.26972 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232c89ed0 0x565232c8c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 4 client.26972 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565232be88b0 con 0x565232c89ed0 2024-04-26T02:00:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.680+0000 7f4e32800640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232c89ed0 0x565232c8c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.681+0000 7f4e32800640 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232c89ed0 0x565232c8c2c0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deecaa610b9c99d7 server_cookie=24ad56474d41ee71 in_seq=0 out_seq=0 2024-04-26T02:00:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.681+0000 7f4e17e00640 10 client.26972 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:27.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e17e00640 1 -- 172.21.15.52:0/1233543271 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565232be88b0 con 0x565232c89ed0 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e17e00640 10 client.26972 handle_command_reply: tid=0 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 client.26972 shutdown 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 msgr2=0x565232c76ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c012d20 tx=0x7f4e1c00f790 comp rx=0 tx=0).stop 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 shutdown_connections 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232c89ed0 0x565232c8c2c0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c77430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c76ef0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/1233543271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c769b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 >> 172.21.15.52:0/1233543271 conn(0x565232c6ad30 msgr2=0x565232c70a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.682+0000 7f4e17e00640 0 client.26972 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.683+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 shutdown_connections 2024-04-26T02:00:27.684 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.683+0000 7f4e3bf96740 1 -- 172.21.15.52:0/1233543271 wait complete. 2024-04-26T02:00:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.683+0000 7f4e3bf96740 20 client.26972 trim_cache size 0 max 16384 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.699+0000 7f4e3bf96740 1 Processor -- start 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.699+0000 7f4e3bf96740 1 -- start start 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c91bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341b2550 con 0x565232c79a00 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341afd60 con 0x565232c6f140 2024-04-26T02:00:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341adbd0 con 0x565232c6e820 2024-04-26T02:00:27.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38026/0 (socket says 172.21.15.52:38026) 2024-04-26T02:00:27.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 -- 172.21.15.52:0/3036137398 learned_addr learned my addr 172.21.15.52:0/3036137398 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:27.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e29400640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c91bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 -- 172.21.15.52:0/3036137398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 msgr2=0x565232c854f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c854f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 -- 172.21.15.52:0/3036137398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 msgr2=0x565232c91bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c91bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e32800640 1 -- 172.21.15.52:0/3036137398 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565232c717d0 con 0x565232c79a00 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.700+0000 7f4e29400640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c91bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.701+0000 7f4e32800640 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c011d10 tx=0x7f4e2c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7efc8163ee0e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.701+0000 7f4e22a00640 1 -- 172.21.15.52:0/3036137398 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01e040 con 0x565232c79a00 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.701+0000 7f4e22a00640 1 -- 172.21.15.52:0/3036137398 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c005580 con 0x565232c79a00 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.701+0000 7f4e22a00640 1 -- 172.21.15.52:0/3036137398 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c00eac0 con 0x565232c79a00 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3036137398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 msgr2=0x565232c92110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c011d10 tx=0x7f4e2c005f90 comp rx=0 tx=0).stop 2024-04-26T02:00:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3036137398 shutdown_connections 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c79a00 0x565232c92110 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6f140 0x565232c91bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/3036137398 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c6e820 0x565232c854f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3036137398 >> 172.21.15.52:0/3036137398 conn(0x565232c6ad30 msgr2=0x565232c71ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3036137398 shutdown_connections 2024-04-26T02:00:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- 172.21.15.52:0/3036137398 wait complete. 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.702+0000 7f4e3bf96740 1 -- start start 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 Processor -- start 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c81de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 0x565232c7f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 0x565232c7f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341ae220 con 0x565232c6f140 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341af690 con 0x565232c6e820 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341ae8f0 con 0x565232c79a00 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e3bf96740 10 client.0 _ll_register_callbacks cb 0x7f4e1c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.703+0000 7f4e32800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c81de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e32800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c81de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:52624/0 (socket says 172.21.15.52:52624) 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e32800640 1 -- 172.21.15.52:0/4176261165 learned_addr learned my addr 172.21.15.52:0/4176261165 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:27.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 0x565232c7f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e32800640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 0x565232c7f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 -- 172.21.15.52:0/4176261165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 msgr2=0x565232c7f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 0x565232c7f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 -- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 msgr2=0x565232c81de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c81de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 -- 172.21.15.52:0/4176261165 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565232c7ff00 con 0x565232c6f140 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e32800640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 0x565232c7f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e29400640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 0x565232c7f1c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c012cf0 tx=0x7f4e1c010640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611e0fdff4d1a0f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.704+0000 7f4e17e00640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c032060 con 0x565232c6f140 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e2c005580 con 0x565232c6f140 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f4e1c005110 con 0x565232c6f140 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f4e1c029040 con 0x565232c6f140 2024-04-26T02:00:27.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e3bf96740 10 client.16827 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x565232cf8ca0 con 0x565232c6f140 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c005580 con 0x565232c6f140 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.705+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7f4e1c02c030 con 0x565232c6f140 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 10 client.16827 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 10 client.16827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 20 client.16827 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232cf94b0 0x565232cfb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 4 client.16827 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565232be2320 con 0x565232cf94b0 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e32800640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232cf94b0 0x565232cfb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e32800640 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232cf94b0 0x565232cfb8d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d3bc13f0744e94 server_cookie=d615250fb41752b6 in_seq=0 out_seq=0 2024-04-26T02:00:27.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.706+0000 7f4e17e00640 10 client.16827 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:27.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:27 smithi052 ceph-mon[27851]: pgmap v488: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 459 op/s 2024-04-26T02:00:27.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 22.65 MB/sec warmup 104 sec latency 3666.316 ms 2024-04-26T02:00:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.877+0000 7f4e17e00640 1 -- 172.21.15.52:0/4176261165 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565232be2320 con 0x565232cf94b0 2024-04-26T02:00:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.877+0000 7f4e17e00640 10 client.16827 handle_command_reply: tid=0 2024-04-26T02:00:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 client.16827 shutdown 2024-04-26T02:00:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 msgr2=0x565232c7f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 0x565232c7f1c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c012cf0 tx=0x7f4e1c010640 comp rx=0 tx=0).stop 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 shutdown_connections 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x565232cf94b0 0x565232cfb8d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565232c79a00 0x565232c7f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x565232c6f140 0x565232c7f1c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 --2- 172.21.15.52:0/4176261165 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x565232c6e820 0x565232c81de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 >> 172.21.15.52:0/4176261165 conn(0x565232c6ad30 msgr2=0x565232c69600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e17e00640 0 client.16827 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 shutdown_connections 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 1 -- 172.21.15.52:0/4176261165 wait complete. 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.878+0000 7f4e3bf96740 20 client.16827 trim_cache size 0 max 16384 2024-04-26T02:00:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:00:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:00:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:00:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.879+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 msgr2=0x7f4e180a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.879+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 0x7f4e180a6300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x565232c6fdc0 tx=0x7f4e240f4840 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.879+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 msgr2=0x7f4e341a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.879+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 0x7f4e341a0610 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006010 tx=0x7f4e2402efe0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 shutdown_connections 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4e180a3e40 0x7f4e180a6300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e3410c520 0x7f4e3419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4e34103b00 0x7f4e341a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 --2- 172.21.15.52:0/3081482714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e341031e0 0x7f4e341a0610 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 >> 172.21.15.52:0/3081482714 conn(0x7f4e34077fd0 msgr2=0x7f4e34102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 shutdown_connections 2024-04-26T02:00:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:27.880+0000 7f4e39000640 1 -- 172.21.15.52:0/3081482714 wait complete. 2024-04-26T02:00:27.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:27 smithi191 ceph-mon[32745]: pgmap v488: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 459 op/s 2024-04-26T02:00:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:27 smithi183 ceph-mon[32752]: pgmap v488: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 459 op/s 2024-04-26T02:00:28.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 22.44 MB/sec warmup 105 sec latency 4666.462 ms 2024-04-26T02:00:29.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:29 smithi052 ceph-mon[27851]: pgmap v489: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:29.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 22.23 MB/sec warmup 106 sec latency 5666.615 ms 2024-04-26T02:00:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:29 smithi191 ceph-mon[32745]: pgmap v489: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:29 smithi183 ceph-mon[32752]: pgmap v489: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:30.720 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 22.02 MB/sec warmup 107 sec latency 6666.764 ms 2024-04-26T02:00:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:31 smithi052 ceph-mon[27851]: pgmap v490: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:31.719 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 21.81 MB/sec warmup 108 sec latency 7666.903 ms 2024-04-26T02:00:31.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:31 smithi191 ceph-mon[32745]: pgmap v490: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:31 smithi183 ceph-mon[32752]: pgmap v490: 97 pgs: 97 active+clean; 495 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 394 op/s 2024-04-26T02:00:32.720 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 21.61 MB/sec warmup 109 sec latency 8667.062 ms 2024-04-26T02:00:33.720 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 21.42 MB/sec warmup 110 sec latency 9667.224 ms 2024-04-26T02:00:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:33 smithi191 ceph-mon[32745]: pgmap v491: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-04-26T02:00:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:33 smithi183 ceph-mon[32752]: pgmap v491: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-04-26T02:00:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:33 smithi052 ceph-mon[27851]: pgmap v491: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.2 MiB/s wr, 483 op/s 2024-04-26T02:00:34.720 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 21.22 MB/sec warmup 111 sec latency 10667.390 ms 2024-04-26T02:00:35.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 21.03 MB/sec warmup 112 sec latency 11667.537 ms 2024-04-26T02:00:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:35 smithi191 ceph-mon[32745]: pgmap v492: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-04-26T02:00:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:35 smithi183 ceph-mon[32752]: pgmap v492: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-04-26T02:00:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:35 smithi052 ceph-mon[27851]: pgmap v492: 97 pgs: 97 active+clean; 491 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-04-26T02:00:36.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 20.85 MB/sec warmup 113 sec latency 12667.679 ms 2024-04-26T02:00:37.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 20.67 MB/sec warmup 114 sec latency 13667.806 ms 2024-04-26T02:00:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:37 smithi191 ceph-mon[32745]: pgmap v493: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-26T02:00:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:37 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:37 smithi183 ceph-mon[32752]: pgmap v493: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-26T02:00:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:37 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:37 smithi052 ceph-mon[27851]: pgmap v493: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-26T02:00:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:37 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:00:38.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 20.49 MB/sec warmup 115 sec latency 14667.956 ms 2024-04-26T02:00:38.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:38 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:38.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:38 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:38.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:38 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:00:38.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:38 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:38.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:38 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:38.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:38 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:00:38.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:38 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:00:38.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:38 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:00:38.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:38 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:00:39.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 20.31 MB/sec warmup 116 sec latency 15668.120 ms 2024-04-26T02:00:39.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:39 smithi191 ceph-mon[32745]: pgmap v494: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:39 smithi183 ceph-mon[32752]: pgmap v494: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:39 smithi052 ceph-mon[27851]: pgmap v494: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:40.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 20.14 MB/sec warmup 117 sec latency 16668.281 ms 2024-04-26T02:00:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.033+0000 7fb1bd600640 1 -- 172.21.15.52:0/4236056853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b8100eb0 msgr2=0x7fb1b8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 --2- 172.21.15.52:0/4236056853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b8100eb0 0x7fb1b8101290 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009be0 tx=0x7fb1a802a6c0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 -- 172.21.15.52:0/4236056853 shutdown_connections 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 --2- 172.21.15.52:0/4236056853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8105f80 0x7fb1b810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 --2- 172.21.15.52:0/4236056853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8101860 0x7fb1b81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 --2- 172.21.15.52:0/4236056853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b8100eb0 0x7fb1b8101290 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 -- 172.21.15.52:0/4236056853 >> 172.21.15.52:0/4236056853 conn(0x7fb1b80fc770 msgr2=0x7fb1b80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 -- 172.21.15.52:0/4236056853 shutdown_connections 2024-04-26T02:00:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.034+0000 7fb1bd600640 1 -- 172.21.15.52:0/4236056853 wait complete. 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 Processor -- start 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 -- start start 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8105f80 0x7fb1b81a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b819f0b0 0x7fb1b819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b810fec0 con 0x7fb1b8100eb0 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b810fd40 con 0x7fb1b8105f80 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.035+0000 7fb1bd600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8110040 con 0x7fb1b819f0b0 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52714/0 (socket says 172.21.15.52:52714) 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 -- 172.21.15.52:0/2648400976 learned_addr learned my addr 172.21.15.52:0/2648400976 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1af400640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8105f80 0x7fb1b81a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b819f0b0 0x7fb1b819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 -- 172.21.15.52:0/2648400976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b819f0b0 msgr2=0x7fb1b819f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b819f0b0 0x7fb1b819f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 -- 172.21.15.52:0/2648400976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8105f80 msgr2=0x7fb1b81a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8105f80 0x7fb1b81a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.036+0000 7fb1b7e00640 1 -- 172.21.15.52:0/2648400976 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b819fb70 con 0x7fb1b8100eb0 2024-04-26T02:00:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004af0 tx=0x7fb1a8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1b5600640 1 -- 172.21.15.52:0/2648400976 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8040020 con 0x7fb1b8100eb0 2024-04-26T02:00:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b819fe00 con 0x7fb1b8100eb0 2024-04-26T02:00:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1b5600640 1 -- 172.21.15.52:0/2648400976 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a8030970 con 0x7fb1b8100eb0 2024-04-26T02:00:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1b5600640 1 -- 172.21.15.52:0/2648400976 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8036040 con 0x7fb1b8100eb0 2024-04-26T02:00:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.037+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b81a75b0 con 0x7fb1b8100eb0 2024-04-26T02:00:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.040+0000 7fb1c1bed740 1 Processor -- start 2024-04-26T02:00:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.040+0000 7fb1c1bed740 1 -- start start 2024-04-26T02:00:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1b5600640 1 -- 172.21.15.52:0/2648400976 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb1a8043020 con 0x7fb1b8100eb0 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b99650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e826af4900 con 0x55e826b8e9b0 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e826a9aef0 con 0x55e826b8f2d0 2024-04-26T02:00:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e826af08c0 con 0x55e826b99b90 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1bcc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1bcc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52730/0 (socket says 172.21.15.52:52730) 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1bcc00640 1 -- 172.21.15.52:0/1498186057 learned_addr learned my addr 172.21.15.52:0/1498186057 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1af400640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.041+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b99650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 -- 172.21.15.52:0/1498186057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 msgr2=0x55e826b9bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 -- 172.21.15.52:0/1498186057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 msgr2=0x55e826b99650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b99650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 -- 172.21.15.52:0/1498186057 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e826807210 con 0x55e826b8e9b0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1af400640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005f60 tx=0x7fb1b00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d3ae4bab640eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1ad200640 1 -- 172.21.15.52:0/1498186057 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0009e50 con 0x55e826b8e9b0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1b5600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 0x7fb19c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1ad200640 1 -- 172.21.15.52:0/1498186057 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b0012900 con 0x55e826b8e9b0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.042+0000 7fb1ad200640 1 -- 172.21.15.52:0/1498186057 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b000fa30 con 0x55e826b8e9b0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 0x7fb19c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1b5600640 1 -- 172.21.15.52:0/2648400976 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fb1a80f1380 con 0x7fb1b8100eb0 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1498186057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 msgr2=0x55e826b8ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005f60 tx=0x7fb1b00117b0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1498186057 shutdown_connections 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b99650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1498186057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8e9b0 0x55e826b8ed90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1498186057 >> 172.21.15.52:0/1498186057 conn(0x55e826b8aec0 msgr2=0x55e826b8b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.043+0000 7fb1b7e00640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 0x7fb19c0a6250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55e826b8ff50 tx=0x7fb1a80f4e30 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.044+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1498186057 shutdown_connections 2024-04-26T02:00:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.044+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1498186057 wait complete. 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.044+0000 7fb1c1bed740 1 -- start start 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.044+0000 7fb1c1bed740 1 Processor -- start 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 0x55e826b96ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b97000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 0x55e826b97540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e826b9c7c0 con 0x55e826b99b90 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55e826aae310 con 0x55e826b8f2d0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e826b9c940 con 0x55e826b8e9b0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1c1bed740 10 client.0 _ll_register_callbacks cb 0x55e826a7ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1bcc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 0x55e826b96ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1af400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b97000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1bcc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 0x55e826b96ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56142/0 (socket says 172.21.15.52:56142) 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1bcc00640 1 -- 172.21.15.52:0/2235048260 learned_addr learned my addr 172.21.15.52:0/2235048260 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.045+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 0x55e826b97540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 -- 172.21.15.52:0/2235048260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 msgr2=0x55e826b96ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 0x55e826b96ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 -- 172.21.15.52:0/2235048260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 msgr2=0x55e826b97540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 0x55e826b97540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 -- 172.21.15.52:0/2235048260 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8268d3620 con 0x55e826b8f2d0 2024-04-26T02:00:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 0x55e826b97540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1af400640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b97000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8107900 tx=0x7fb1a0011b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4e7b6a8dd789f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1a5600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a001e020 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b0012900 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.046+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fb1a000fbc0 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb1a000fdc0 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 10 client.26990 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55e826ba8a70 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b0012900 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a0019070 con 0x55e826b8f2d0 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 10 client.26990 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 10 client.26990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 20 client.26990 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826ba9f80 0x55e826bac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 4 client.26990 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.047+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e826b088b0 con 0x55e826ba9f80 2024-04-26T02:00:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.048+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826ba9f80 0x55e826bac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.048+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826ba9f80 0x55e826bac390 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b86705a40903cd server_cookie=309c99ef300a813f in_seq=0 out_seq=0 2024-04-26T02:00:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.048+0000 7fb1a5600640 10 client.26990 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.049+0000 7fb1a5600640 1 -- 172.21.15.52:0/2235048260 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e826b088b0 con 0x55e826ba9f80 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.049+0000 7fb1a5600640 10 client.26990 handle_command_reply: tid=0 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.049+0000 7fb1c1bed740 1 client.26990 shutdown 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.049+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 msgr2=0x55e826b97000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.049+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b97000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8107900 tx=0x7fb1a0011b30 comp rx=0 tx=0).stop 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 shutdown_connections 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826ba9f80 0x55e826bac390 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b99b90 0x55e826b97540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8f2d0 0x55e826b97000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2235048260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b8e9b0 0x55e826b96ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1a5600640 0 client.26990 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 >> 172.21.15.52:0/2235048260 conn(0x55e826b8aec0 msgr2=0x55e826b90c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 shutdown_connections 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2235048260 wait complete. 2024-04-26T02:00:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.050+0000 7fb1c1bed740 20 client.26990 trim_cache size 0 max 16384 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.066+0000 7fb1c1bed740 1 Processor -- start 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.066+0000 7fb1c1bed740 1 -- start start 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826bb1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81b25b0 con 0x55e826b8f2d0 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81afdc0 con 0x55e826b8e9b0 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81adc30 con 0x55e826b99b90 2024-04-26T02:00:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52748/0 (socket says 172.21.15.52:52748) 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 -- 172.21.15.52:0/2903252792 learned_addr learned my addr 172.21.15.52:0/2903252792 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826bb1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 -- 172.21.15.52:0/2903252792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 msgr2=0x55e826bb1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826bb1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.067+0000 7fb1af400640 1 -- 172.21.15.52:0/2903252792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 msgr2=0x55e826ba5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1af400640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1af400640 1 -- 172.21.15.52:0/2903252792 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e826b85490 con 0x55e826b8f2d0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1af400640 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0006010 tx=0x7fb1a000fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e62aef5a820775 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1a7e00640 1 -- 172.21.15.52:0/2903252792 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a001e070 con 0x55e826b8f2d0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1a7e00640 1 -- 172.21.15.52:0/2903252792 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a00196d0 con 0x55e826b8f2d0 2024-04-26T02:00:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.068+0000 7fb1a7e00640 1 -- 172.21.15.52:0/2903252792 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00052d0 con 0x55e826b8f2d0 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2903252792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 msgr2=0x55e826b97360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0006010 tx=0x7fb1a000fd50 comp rx=0 tx=0).stop 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2903252792 shutdown_connections 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826bb1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b97360 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/2903252792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2903252792 >> 172.21.15.52:0/2903252792 conn(0x55e826b8aec0 msgr2=0x55e826b90360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2903252792 shutdown_connections 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- 172.21.15.52:0/2903252792 wait complete. 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.069+0000 7fb1c1bed740 1 -- start start 2024-04-26T02:00:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 Processor -- start 2024-04-26T02:00:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ae280 con 0x55e826b8f2d0 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81af6f0 con 0x55e826b8e9b0 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81ae950 con 0x55e826b99b90 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1c1bed740 10 client.0 _ll_register_callbacks cb 0x7fb1a0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1af400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1af400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52762/0 (socket says 172.21.15.52:52762) 2024-04-26T02:00:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1bcc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.070+0000 7fb1af400640 1 -- 172.21.15.52:0/1625079402 learned_addr learned my addr 172.21.15.52:0/1625079402 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 -- 172.21.15.52:0/1625079402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 msgr2=0x55e826b9f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 -- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 msgr2=0x55e826ba1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 -- 172.21.15.52:0/1625079402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e826b9fc00 con 0x55e826b8f2d0 2024-04-26T02:00:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1af400640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00195d0 tx=0x7fb1a0004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ff0709d4559f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1a5600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.071+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a001e040 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fb1a0004020 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a00196d0 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fb1a003b050 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 10 client.16845 fetch_fsmap learned FSMap version 58 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55e826c10180 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a00196d0 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a0033020 con 0x55e826b8f2d0 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 10 client.16845 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 20 client.16845 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826c10860 0x55e826c12c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 4 client.16845 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.072+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e826b02320 con 0x55e826c10860 2024-04-26T02:00:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.073+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826c10860 0x55e826c12c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.074+0000 7fb1bcc00640 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826c10860 0x55e826c12c80 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7153557566d390f4 server_cookie=b266cf46adcbc210 in_seq=0 out_seq=0 2024-04-26T02:00:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.074+0000 7fb1a5600640 10 client.16845 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1a5600640 1 -- 172.21.15.52:0/1625079402 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e826b02320 con 0x55e826c10860 2024-04-26T02:00:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1a5600640 10 client.16845 handle_command_reply: tid=0 2024-04-26T02:00:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 client.16845 shutdown 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 msgr2=0x55e826b9f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00195d0 tx=0x7fb1a0004330 comp rx=0 tx=0).stop 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 shutdown_connections 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55e826c10860 0x55e826c12c80 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e826b99b90 0x55e826b9f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e826b8f2d0 0x55e826b9f180 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1a5600640 0 client.16845 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 --2- 172.21.15.52:0/1625079402 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55e826b8e9b0 0x55e826ba1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 >> 172.21.15.52:0/1625079402 conn(0x55e826b8aec0 msgr2=0x55e826b90bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 shutdown_connections 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.108+0000 7fb1c1bed740 1 -- 172.21.15.52:0/1625079402 wait complete. 2024-04-26T02:00:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.109+0000 7fb1c1bed740 20 client.16845 trim_cache size 0 max 16384 2024-04-26T02:00:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:00:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:00:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:00:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.109+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 msgr2=0x7fb19c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.109+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 0x7fb19c0a6250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55e826b8ff50 tx=0x7fb1a80f4e30 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.109+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 msgr2=0x7fb1b81a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.109+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004af0 tx=0x7fb1a8003c20 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 shutdown_connections 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb19c0a3d90 0x7fb19c0a6250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1b819f0b0 0x7fb1b819f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1b8105f80 0x7fb1b81a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 --2- 172.21.15.52:0/2648400976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb1b8100eb0 0x7fb1b81a0310 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 >> 172.21.15.52:0/2648400976 conn(0x7fb1b80fc770 msgr2=0x7fb1b8104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 shutdown_connections 2024-04-26T02:00:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:41.110+0000 7fb1bd600640 1 -- 172.21.15.52:0/2648400976 wait complete. 2024-04-26T02:00:41.721 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 19.97 MB/sec warmup 118 sec latency 17668.455 ms 2024-04-26T02:00:41.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:41 smithi191 ceph-mon[32745]: pgmap v495: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:41 smithi183 ceph-mon[32752]: pgmap v495: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:41 smithi052 ceph-mon[27851]: pgmap v495: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 830 KiB/s wr, 98 op/s 2024-04-26T02:00:42.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 19.80 MB/sec warmup 119 sec latency 18668.623 ms 2024-04-26T02:00:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:43 smithi191 ceph-mon[32745]: pgmap v496: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 856 KiB/s wr, 103 op/s 2024-04-26T02:00:43.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:43 smithi183 ceph-mon[32752]: pgmap v496: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 856 KiB/s wr, 103 op/s 2024-04-26T02:00:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:43 smithi052 ceph-mon[27851]: pgmap v496: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 856 KiB/s wr, 103 op/s 2024-04-26T02:00:44.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 1 sec latency 20668.891 ms 2024-04-26T02:00:45.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 2 sec latency 21668.992 ms 2024-04-26T02:00:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:45 smithi191 ceph-mon[32745]: pgmap v497: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-26T02:00:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:45 smithi183 ceph-mon[32752]: pgmap v497: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-26T02:00:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:45 smithi052 ceph-mon[27851]: pgmap v497: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-26T02:00:46.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 3 sec latency 22669.129 ms 2024-04-26T02:00:47.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 4 sec latency 23669.291 ms 2024-04-26T02:00:47.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:47 smithi052 ceph-mon[27851]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 188 KiB/s wr, 27 op/s 2024-04-26T02:00:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:47 smithi191 ceph-mon[32745]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 188 KiB/s wr, 27 op/s 2024-04-26T02:00:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:47 smithi183 ceph-mon[32752]: pgmap v498: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 188 KiB/s wr, 27 op/s 2024-04-26T02:00:48.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 5 sec latency 24669.410 ms 2024-04-26T02:00:49.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 6 sec latency 25669.529 ms 2024-04-26T02:00:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:49 smithi191 ceph-mon[32745]: pgmap v499: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:49 smithi183 ceph-mon[32752]: pgmap v499: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:49.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:00:49.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:49 smithi052 ceph-mon[27851]: pgmap v499: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:50.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:50.722 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 7 sec latency 26669.622 ms 2024-04-26T02:00:51.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 8 sec latency 27669.770 ms 2024-04-26T02:00:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:51 smithi191 ceph-mon[32745]: pgmap v500: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:51.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:51 smithi183 ceph-mon[32752]: pgmap v500: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:51 smithi052 ceph-mon[27851]: pgmap v500: 97 pgs: 97 active+clean; 493 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 67 KiB/s wr, 17 op/s 2024-04-26T02:00:52.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 9 sec latency 28669.904 ms 2024-04-26T02:00:53.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 -- 172.21.15.52:0/2198289573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 msgr2=0x7f31b41098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:53.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 --2- 172.21.15.52:0/2198289573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b41098e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009990 tx=0x7f31a0009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 -- 172.21.15.52:0/2198289573 shutdown_connections 2024-04-26T02:00:53.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 --2- 172.21.15.52:0/2198289573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4109fc0 0x7f31b4111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 --2- 172.21.15.52:0/2198289573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4104ee0 0x7f31b41052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 --2- 172.21.15.52:0/2198289573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b41098e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 -- 172.21.15.52:0/2198289573 >> 172.21.15.52:0/2198289573 conn(0x7f31b4100770 msgr2=0x7f31b4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 -- 172.21.15.52:0/2198289573 shutdown_connections 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.969+0000 7f31bb800700 1 -- 172.21.15.52:0/2198289573 wait complete. 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.970+0000 7f31bb800700 1 Processor -- start 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.970+0000 7f31bb800700 1 -- start start 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4109fc0 0x7f31b419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f31b4114e50 con 0x7f31b4109fc0 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f31b4114cd0 con 0x7f31b4105890 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31bb800700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f31b4114fd0 con 0x7f31b4104ee0 2024-04-26T02:00:53.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37340/0 (socket says 172.21.15.52:37340) 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 -- 172.21.15.52:0/3527421599 learned_addr learned my addr 172.21.15.52:0/3527421599 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b9800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4109fc0 0x7f31b419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b3e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 -- 172.21.15.52:0/3527421599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 msgr2=0x7f31b419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 -- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4109fc0 msgr2=0x7f31b419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4109fc0 0x7f31b419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.971+0000 7f31b8e00700 1 -- 172.21.15.52:0/3527421599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b41a0330 con 0x7f31b4104ee0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31b8e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f31a800eb30 tx=0x7f31a800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a800cbd0 con 0x7f31b4104ee0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a80044d0 con 0x7f31b4104ee0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8003e30 con 0x7f31b4104ee0 2024-04-26T02:00:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b41a0620 con 0x7f31b4104ee0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.972+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b41a7ea0 con 0x7f31b4104ee0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.973+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31b4069730 con 0x7f31b4104ee0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.975+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f31a8004000 con 0x7f31b4104ee0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.977+0000 7f31b1600700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 0x7f319c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.977+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f31a80d0bc0 con 0x7f31b4104ee0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.977+0000 7f31b3e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 0x7f319c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.978+0000 7f31b3e00700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 0x7f319c0a6270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b2e0 tx=0x7f31a0005960 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:53.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:52.981+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31a80042f0 con 0x7f31b4104ee0 2024-04-26T02:00:53.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.309+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31b402df80 con 0x7f31b4104ee0 2024-04-26T02:00:53.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.310+0000 7f31b1600700 1 -- 172.21.15.52:0/3527421599 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11891 (secure 0 0 0) 0x7f31a809d200 con 0x7f31b4104ee0 2024-04-26T02:00:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:00:53.310 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T01:59:07.564820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/3742874930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3742874930},{"type":"v1","addr":"172.21.15.183:6837","nonce":3742874930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T02:00:53.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 msgr2=0x7f319c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:53.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 0x7f319c0a6270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b2e0 tx=0x7f31a0005960 comp rx=0 tx=0).stop 2024-04-26T02:00:53.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 msgr2=0x7f31b419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f31a800eb30 tx=0x7f31a800ee40 comp rx=0 tx=0).stop 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 shutdown_connections 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f319c0a3db0 0x7f319c0a6270 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31b4109fc0 0x7f31b419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31b4104ee0 0x7f31b419b320 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 --2- 172.21.15.52:0/3527421599 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31b4105890 0x7f31b419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 >> 172.21.15.52:0/3527421599 conn(0x7f31b4100770 msgr2=0x7f31b4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:53.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 shutdown_connections 2024-04-26T02:00:53.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:53.314+0000 7f31bb800700 1 -- 172.21.15.52:0/3527421599 wait complete. 2024-04-26T02:00:53.316 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 58 2024-04-26T02:00:53.698 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:53 smithi052 ceph-mon[27851]: pgmap v501: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 752 KiB/s wr, 43 op/s 2024-04-26T02:00:53.699 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:53 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3527421599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:53.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 10 sec latency 29669.984 ms 2024-04-26T02:00:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:53 smithi191 ceph-mon[32745]: pgmap v501: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 752 KiB/s wr, 43 op/s 2024-04-26T02:00:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:53 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3527421599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:53 smithi183 ceph-mon[32752]: pgmap v501: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 752 KiB/s wr, 43 op/s 2024-04-26T02:00:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:53 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3527421599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:00:54.167 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id 860f4752 --release --await' 2024-04-26T02:00:54.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id 860f4752 --release --await 2024-04-26T02:00:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 -- 172.21.15.52:0/1279566671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f4109470 msgr2=0x7fe3f4111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/1279566671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f4109470 0x7fe3f4111040 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009ca0 tx=0x7fe3ec02a8c0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 -- 172.21.15.52:0/1279566671 shutdown_connections 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/1279566671 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f4109470 0x7fe3f4111040 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/1279566671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4104d50 0x7fe3f4108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/1279566671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f41043a0 0x7fe3f4104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 -- 172.21.15.52:0/1279566671 >> 172.21.15.52:0/1279566671 conn(0x7fe3f40ffc40 msgr2=0x7fe3f4102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 -- 172.21.15.52:0/1279566671 shutdown_connections 2024-04-26T02:00:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.257+0000 7fe3f8c00640 1 -- 172.21.15.52:0/1279566671 wait complete. 2024-04-26T02:00:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 Processor -- start 2024-04-26T02:00:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 -- start start 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4109470 0x7fe3f41a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f419a670 0x7fe3f419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f4114410 con 0x7fe3f419a670 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f4114290 con 0x7fe3f41043a0 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.258+0000 7fe3f8c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f4114590 con 0x7fe3f4109470 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36770/0 (socket says 172.21.15.52:36770) 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 -- 172.21.15.52:0/284555690 learned_addr learned my addr 172.21.15.52:0/284555690 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f2800640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f419a670 0x7fe3f419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1400640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4109470 0x7fe3f41a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 -- 172.21.15.52:0/284555690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4109470 msgr2=0x7fe3f41a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4109470 0x7fe3f41a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 -- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f419a670 msgr2=0x7fe3f419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f419a670 0x7fe3f419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 -- 172.21.15.52:0/284555690 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f419b270 con 0x7fe3f41043a0 2024-04-26T02:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.259+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400d950 tx=0x7fe3e400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.260+0000 7fe3eaa00640 1 -- 172.21.15.52:0/284555690 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4017030 con 0x7fe3f41043a0 2024-04-26T02:00:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.260+0000 7fe3eaa00640 1 -- 172.21.15.52:0/284555690 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e4004020 con 0x7fe3f41043a0 2024-04-26T02:00:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.260+0000 7fe3eaa00640 1 -- 172.21.15.52:0/284555690 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40047a0 con 0x7fe3f41043a0 2024-04-26T02:00:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.260+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f41a7350 con 0x7fe3f41043a0 2024-04-26T02:00:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.260+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f41a7700 con 0x7fe3f41043a0 2024-04-26T02:00:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.263+0000 7fe3eaa00640 1 -- 172.21.15.52:0/284555690 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe3e4009a30 con 0x7fe3f41043a0 2024-04-26T02:00:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.264+0000 7fe3fb97d740 1 Processor -- start 2024-04-26T02:00:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 -- start start 2024-04-26T02:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f2e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f2ea00 0x5563a8f30df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5563a8e89900 con 0x5563a8f2ea00 2024-04-26T02:00:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5563a8e2fef0 con 0x5563a8f23820 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5563a8e858c0 con 0x5563a8f24140 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.265+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36784/0 (socket says 172.21.15.52:36784) 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f2e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 -- 172.21.15.52:0/1779116724 learned_addr learned my addr 172.21.15.52:0/1779116724 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f1400640 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f2ea00 0x5563a8f30df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 -- 172.21.15.52:0/1779116724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 msgr2=0x5563a8f2e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f2e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 -- 172.21.15.52:0/1779116724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f2ea00 msgr2=0x5563a8f30df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f2ea00 0x5563a8f30df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3eaa00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 0x7fe3d80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 -- 172.21.15.52:0/1779116724 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a8b9c210 con 0x5563a8f23820 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3eaa00640 1 -- 172.21.15.52:0/284555690 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fe3e40d01a0 con 0x7fe3f41043a0 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.266+0000 7fe3f2800640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 0x7fe3d80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.267+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009ca0 tx=0x7fe3ec02f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f08375facb635466 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.267+0000 7fe3df000640 1 -- 172.21.15.52:0/1779116724 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec031920 con 0x5563a8f23820 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.267+0000 7fe3df000640 1 -- 172.21.15.52:0/1779116724 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec036030 con 0x5563a8f23820 2024-04-26T02:00:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.267+0000 7fe3df000640 1 -- 172.21.15.52:0/1779116724 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec0047b0 con 0x5563a8f23820 2024-04-26T02:00:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1779116724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 msgr2=0x5563a8f23c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009ca0 tx=0x7fe3ec02f3c0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3f2800640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 0x7fe3d80a6300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec005fe0 tx=0x7fe3ec04b000 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1779116724 shutdown_connections 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f2ea00 0x5563a8f30df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f2e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1779116724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f23820 0x5563a8f23c00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1779116724 >> 172.21.15.52:0/1779116724 conn(0x5563a8f1fd30 msgr2=0x5563a8f20110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1779116724 shutdown_connections 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.268+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1779116724 wait complete. 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.269+0000 7fe3fb97d740 1 -- start start 2024-04-26T02:00:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.269+0000 7fe3fb97d740 1 Processor -- start 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f2b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f2ea00 0x5563a8f2c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5563a8f314b0 con 0x5563a8f23820 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5563a8e43310 con 0x5563a8f24140 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5563a8f31630 con 0x5563a8f2ea00 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3fb97d740 10 client.0 _ll_register_callbacks cb 0x5563a8e11cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f2b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36798/0 (socket says 172.21.15.52:36798) 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 -- 172.21.15.52:0/2515465175 learned_addr learned my addr 172.21.15.52:0/2515465175 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f2800640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f2ea00 0x5563a8f2c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 -- 172.21.15.52:0/2515465175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f2ea00 msgr2=0x5563a8f2c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f2ea00 0x5563a8f2c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 -- 172.21.15.52:0/2515465175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 msgr2=0x5563a8f2b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f2b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.270+0000 7fe3f1400640 1 -- 172.21.15.52:0/2515465175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a8c68620 con 0x5563a8f24140 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3f1400640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00124a0 tx=0x7fe3e0012960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4196c7dd1e14079 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3d3400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0004050 con 0x5563a8f24140 2024-04-26T02:00:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fe3e00041f0 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3ec036030 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe3e00043f0 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3fb97d740 10 client.27014 fetch_fsmap learned FSMap version 59 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.271+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x5563a8f3d790 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec036030 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 59) ==== 8358+0+0 (secure 0 0 0) 0x7fe3e000f730 con 0x5563a8f24140 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 10 client.27014 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 10 client.27014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 20 client.27014 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8f3ecc0 0x5563a8f41110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 4 client.27014 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563a8e9d8b0 con 0x5563a8f3ecc0 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.272+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8f3ecc0 0x5563a8f41110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.273+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8f3ecc0 0x5563a8f41110 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5be234cfd06b18f9 server_cookie=317a5f82ec405aeb in_seq=0 out_seq=0 2024-04-26T02:00:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.273+0000 7fe3d3400640 10 client.27014 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.273+0000 7fe3d3400640 1 -- 172.21.15.52:0/2515465175 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563a8e9d8b0 con 0x5563a8f3ecc0 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.273+0000 7fe3d3400640 10 client.27014 handle_command_reply: tid=0 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.273+0000 7fe3fb97d740 1 client.27014 shutdown 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 msgr2=0x5563a8f2be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00124a0 tx=0x7fe3e0012960 comp rx=0 tx=0).stop 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 shutdown_connections 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8f3ecc0 0x5563a8f41110 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f2ea00 0x5563a8f2c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f24140 0x5563a8f2be80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/2515465175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f2b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3d3400640 0 client.27014 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 >> 172.21.15.52:0/2515465175 conn(0x5563a8f1fd30 msgr2=0x5563a8f25ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 shutdown_connections 2024-04-26T02:00:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 1 -- 172.21.15.52:0/2515465175 wait complete. 2024-04-26T02:00:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.274+0000 7fe3fb97d740 20 client.27014 trim_cache size 0 max 16384 2024-04-26T02:00:54.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 Processor -- start 2024-04-26T02:00:54.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 -- start start 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f3a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f46a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41b2680 con 0x5563a8f23820 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41afe90 con 0x5563a8f2ea00 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41add00 con 0x5563a8f24140 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.291+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:36808/0 (socket says 172.21.15.52:36808) 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 -- 172.21.15.52:0/1582745673 learned_addr learned my addr 172.21.15.52:0/1582745673 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f3a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f1400640 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f46a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 -- 172.21.15.52:0/1582745673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 msgr2=0x5563a8f46a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f46a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 -- 172.21.15.52:0/1582745673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 msgr2=0x5563a8f3a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f3a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 -- 172.21.15.52:0/1582745673 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a8f23f70 con 0x5563a8f2ea00 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3f2800640 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec005e30 tx=0x7fe3ec051000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fdd4072edf0e32c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.292+0000 7fe3de600640 1 -- 172.21.15.52:0/1582745673 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec0318a0 con 0x5563a8f2ea00 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3de600640 1 -- 172.21.15.52:0/1582745673 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec059070 con 0x5563a8f2ea00 2024-04-26T02:00:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3de600640 1 -- 172.21.15.52:0/1582745673 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec02fa10 con 0x5563a8f2ea00 2024-04-26T02:00:54.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1582745673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 msgr2=0x5563a8f46fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec005e30 tx=0x7fe3ec051000 comp rx=0 tx=0).stop 2024-04-26T02:00:54.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1582745673 shutdown_connections 2024-04-26T02:00:54.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f46fb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f46a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/1582745673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f3a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1582745673 >> 172.21.15.52:0/1582745673 conn(0x5563a8f1fd30 msgr2=0x5563a8f20110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1582745673 shutdown_connections 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.293+0000 7fe3fb97d740 1 -- 172.21.15.52:0/1582745673 wait complete. 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.294+0000 7fe3fb97d740 1 -- start start 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.294+0000 7fe3fb97d740 1 Processor -- start 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.294+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f34500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41ae350 con 0x5563a8f23820 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41af7c0 con 0x5563a8f2ea00 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f41aea20 con 0x5563a8f24140 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3fb97d740 10 client.0 _ll_register_callbacks cb 0x7fe3e0009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37384/0 (socket says 172.21.15.52:37384) 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 -- 172.21.15.52:0/3053578076 learned_addr learned my addr 172.21.15.52:0/3053578076 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f2800640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f34500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 -- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 msgr2=0x5563a8f34f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f34f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 -- 172.21.15.52:0/3053578076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 msgr2=0x5563a8f34500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f34500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f1400640 1 -- 172.21.15.52:0/3053578076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a8fa4540 con 0x5563a8f24140 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.295+0000 7fe3f2800640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3f1400640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009bc0 tx=0x7fe3e0005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1f6285ce7fddf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3d3400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0012e60 con 0x5563a8f24140 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3ec059070 con 0x5563a8f24140 2024-04-26T02:00:54.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7fe3e0005040 con 0x5563a8f24140 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fe3e003c050 con 0x5563a8f24140 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3fb97d740 10 client.27013 fetch_fsmap learned FSMap version 59 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.296+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x5563a8fa5810 con 0x5563a8f24140 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3ec059070 con 0x5563a8f24140 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 59) ==== 8358+0+0 (secure 0 0 0) 0x7fe3e0004c50 con 0x5563a8f24140 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 10 client.27013 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 10 client.27013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 20 client.27013 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8fa5fe0 0x5563a8fa8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 4 client.27013 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563a8e97320 con 0x5563a8fa5fe0 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.297+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8fa5fe0 0x5563a8fa8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.298+0000 7fe3f1e00640 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8fa5fe0 0x5563a8fa8400 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2be7f3e26314d0d server_cookie=d5a66e456c8cdc3b in_seq=0 out_seq=0 2024-04-26T02:00:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.298+0000 7fe3d3400640 10 client.27013 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.340+0000 7fe3d3400640 1 -- 172.21.15.52:0/3053578076 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563a8e97320 con 0x5563a8fa5fe0 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.340+0000 7fe3d3400640 10 client.27013 handle_command_reply: tid=0 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 client.27013 shutdown 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 msgr2=0x5563a8f34a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009bc0 tx=0x7fe3e0005c50 comp rx=0 tx=0).stop 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 shutdown_connections 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5563a8fa5fe0 0x5563a8fa8400 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5563a8f2ea00 0x5563a8f34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5563a8f24140 0x5563a8f34a40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 --2- 172.21.15.52:0/3053578076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5563a8f23820 0x5563a8f34500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3d3400640 0 client.27013 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 >> 172.21.15.52:0/3053578076 conn(0x5563a8f1fd30 msgr2=0x5563a8f25220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 shutdown_connections 2024-04-26T02:00:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 1 -- 172.21.15.52:0/3053578076 wait complete. 2024-04-26T02:00:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.341+0000 7fe3fb97d740 20 client.27013 trim_cache size 0 max 16384 2024-04-26T02:00:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:00:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:00:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:00:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 msgr2=0x7fe3d80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 0x7fe3d80a6300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec005fe0 tx=0x7fe3ec04b000 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 msgr2=0x7fe3f41a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400d950 tx=0x7fe3e400de10 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 shutdown_connections 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3d80a3e40 0x7fe3d80a6300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f419a670 0x7fe3f419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3f4109470 0x7fe3f41a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 --2- 172.21.15.52:0/284555690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3f41043a0 0x7fe3f41a05b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.342+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 >> 172.21.15.52:0/284555690 conn(0x7fe3f40ffc40 msgr2=0x7fe3f4100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.343+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 shutdown_connections 2024-04-26T02:00:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:00:54.343+0000 7fe3f8c00640 1 -- 172.21.15.52:0/284555690 wait complete. 2024-04-26T02:00:54.679 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:00:54.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 11 sec latency 30670.055 ms 2024-04-26T02:00:54.839 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:54 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:54.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:54 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:54.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:54 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:00:55.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 12 sec latency 31670.214 ms 2024-04-26T02:00:55.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:55 smithi052 ceph-mon[27851]: pgmap v502: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 727 KiB/s wr, 38 op/s 2024-04-26T02:00:55.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:55 smithi191 ceph-mon[32745]: pgmap v502: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 727 KiB/s wr, 38 op/s 2024-04-26T02:00:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:55 smithi183 ceph-mon[32752]: pgmap v502: 97 pgs: 97 active+clean; 492 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 727 KiB/s wr, 38 op/s 2024-04-26T02:00:56.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40545 0.00 MB/sec execute 13 sec latency 32670.350 ms 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 -- 172.21.15.52:0/49938496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc107980 msgr2=0x7f2edc107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 --2- 172.21.15.52:0/49938496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc107980 0x7f2edc107d60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009990 tx=0x7f2ecc009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 -- 172.21.15.52:0/49938496 shutdown_connections 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 --2- 172.21.15.52:0/49938496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc1022f0 0x7f2edc10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 --2- 172.21.15.52:0/49938496 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc107980 0x7f2edc107d60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 --2- 172.21.15.52:0/49938496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc101930 0x7f2edc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.194+0000 7f2ee3000700 1 -- 172.21.15.52:0/49938496 >> 172.21.15.52:0/49938496 conn(0x7f2edc0788b0 msgr2=0x7f2edc078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.195+0000 7f2ee3000700 1 -- 172.21.15.52:0/49938496 shutdown_connections 2024-04-26T02:00:57.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.195+0000 7f2ee3000700 1 -- 172.21.15.52:0/49938496 wait complete. 2024-04-26T02:00:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.196+0000 7f2ee3000700 1 Processor -- start 2024-04-26T02:00:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.196+0000 7f2ee3000700 1 -- start start 2024-04-26T02:00:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc101930 0x7f2edc1034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 0x7f2edc103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc111ad0 con 0x7f2edc101930 2024-04-26T02:00:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc111950 con 0x7f2edc1022f0 2024-04-26T02:00:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee3000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc111c50 con 0x7f2edc107980 2024-04-26T02:00:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ed3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc101930 0x7f2edc1034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ee1000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 0x7f2edc103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ed3400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:50410/0 (socket says 172.21.15.52:50410) 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.197+0000 7f2ed3400700 1 -- 172.21.15.52:0/3300996312 learned_addr learned my addr 172.21.15.52:0/3300996312 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 -- 172.21.15.52:0/3300996312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 msgr2=0x7f2edc103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 0x7f2edc103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 -- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc101930 msgr2=0x7f2edc1034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc101930 0x7f2edc1034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 -- 172.21.15.52:0/3300996312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc1a7650 con 0x7f2edc1022f0 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ee1000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 0x7f2edc103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:57.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed3400700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400e9a0 tx=0x7f2ec400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed9600700 1 -- 172.21.15.52:0/3300996312 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec400ca10 con 0x7f2edc1022f0 2024-04-26T02:00:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed9600700 1 -- 172.21.15.52:0/3300996312 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec400cbb0 con 0x7f2edc1022f0 2024-04-26T02:00:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ed9600700 1 -- 172.21.15.52:0/3300996312 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4018680 con 0x7f2edc1022f0 2024-04-26T02:00:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2edc198f10 con 0x7f2edc1022f0 2024-04-26T02:00:57.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.198+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2edc1994a0 con 0x7f2edc1022f0 2024-04-26T02:00:57.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.202+0000 7f2ed9600700 1 -- 172.21.15.52:0/3300996312 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2ec4010430 con 0x7f2edc1022f0 2024-04-26T02:00:57.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.203+0000 7f2ed9600700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 0x7f2ec00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.203+0000 7f2eed9bdb80 1 Processor -- start 2024-04-26T02:00:57.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.203+0000 7f2ed9600700 1 -- 172.21.15.52:0/3300996312 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f2ec40d09a0 con 0x7f2edc1022f0 2024-04-26T02:00:57.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.203+0000 7f2eed9bdb80 1 -- start start 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9cefd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9cfabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9cfd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557ae9c42f90 con 0x557ae9cfb0f0 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557ae9be9830 con 0x557ae9cf0240 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557ae9c3ef50 con 0x557ae9cef920 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2edbe00700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 0x7f2ec00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9cefd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9cefd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:35570/0 (socket says 172.21.15.52:35570) 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2edbe00700 1 -- 172.21.15.52:0/2086160509 learned_addr learned my addr 172.21.15.52:0/2086160509 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.204+0000 7f2ed3400700 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9cfabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ee1000700 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9cfd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 -- 172.21.15.52:0/2086160509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 msgr2=0x557ae9cefd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9cefd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 -- 172.21.15.52:0/2086160509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 msgr2=0x557ae9cfd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9cfd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 -- 172.21.15.52:0/2086160509 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae9bdf780 con 0x557ae9cf0240 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2edbe00700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 0x7f2ec00a62f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc000c00 tx=0x7f2ecc009530 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.205+0000 7f2ed3400700 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9cfabb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40ce090 tx=0x7f2ec409c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a347ac8260011840 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2ed1600700 1 -- 172.21.15.52:0/2086160509 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec40a5550 con 0x557ae9cf0240 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2ed1600700 1 -- 172.21.15.52:0/2086160509 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec409abe0 con 0x557ae9cf0240 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2ed1600700 1 -- 172.21.15.52:0/2086160509 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec40a5320 con 0x557ae9cf0240 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/2086160509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 msgr2=0x557ae9cfabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9cfabb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40ce090 tx=0x7f2ec409c550 comp rx=0 tx=0).stop 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/2086160509 shutdown_connections 2024-04-26T02:00:57.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9cfd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9cefd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/2086160509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9cfabb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.206+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/2086160509 >> 172.21.15.52:0/2086160509 conn(0x557ae9c1e6a0 msgr2=0x557ae9cebdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.207+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/2086160509 shutdown_connections 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.207+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/2086160509 wait complete. 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.207+0000 7f2eed9bdb80 1 -- start start 2024-04-26T02:00:57.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.207+0000 7f2eed9bdb80 1 Processor -- start 2024-04-26T02:00:57.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 0x557ae9cf7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557ae9cfdc10 con 0x557ae9cef920 2024-04-26T02:00:57.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557ae9bfc960 con 0x557ae9cfb0f0 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557ae9cfdd90 con 0x557ae9cf0240 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2eed9bdb80 10 client.0 _ll_register_callbacks cb 0x557ae9bdb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37470/0 (socket says 172.21.15.52:37470) 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2edbe00700 1 -- 172.21.15.52:0/90814670 learned_addr learned my addr 172.21.15.52:0/90814670 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2ed3400700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 0x557ae9cf7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.208+0000 7f2ee1000700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 -- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cfb0f0 msgr2=0x557ae9cf8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 -- 172.21.15.52:0/90814670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 msgr2=0x557ae9cfa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 -- 172.21.15.52:0/90814670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae9b909d0 con 0x557ae9cf0240 2024-04-26T02:00:57.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2edbe00700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2ed3400700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 0x557ae9cf7bc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40ce0c0 tx=0x7f2ec40a0d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afcd50d961bb6ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2eca000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec40a50c0 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f2ec40a5260 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.209+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ec409abe0 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f2ec409bd30 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 10 client.27025 fetch_fsmap learned FSMap version 59 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x557ae9bd9bf0 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec409abe0 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 59) ==== 8358+0+0 (secure 0 0 0) 0x7f2ec409b7a0 con 0x557ae9cf0240 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 10 client.27025 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 10 client.27025 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 20 client.27025 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d0a810 0x557ae9d0cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 4 client.27025 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:00:57.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.210+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ae9c50780 con 0x557ae9d0a810 2024-04-26T02:00:57.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.211+0000 7f2edbe00700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d0a810 0x557ae9d0cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.211+0000 7f2edbe00700 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d0a810 0x557ae9d0cc00 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4a668baffb55142 server_cookie=54de72505ccd9a6c in_seq=0 out_seq=0 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.211+0000 7f2eca000700 10 client.27025 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eca000700 1 -- 172.21.15.52:0/90814670 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ae9c50780 con 0x557ae9d0a810 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eca000700 10 client.27025 handle_command_reply: tid=0 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 client.27025 shutdown 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 msgr2=0x557ae9cf7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 0x557ae9cf7bc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2ec40ce0c0 tx=0x7f2ec40a0d50 comp rx=0 tx=0).stop 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 shutdown_connections 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d0a810 0x557ae9d0cc00 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.212+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cef920 0x557ae9cfa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eca000700 0 client.27025 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/90814670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cf0240 0x557ae9cf7bc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 >> 172.21.15.52:0/90814670 conn(0x557ae9c1e6a0 msgr2=0x557ae9ceb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 shutdown_connections 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/90814670 wait complete. 2024-04-26T02:00:57.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.213+0000 7f2eed9bdb80 20 client.27025 trim_cache size 0 max 16384 2024-04-26T02:00:57.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 Processor -- start 2024-04-26T02:00:57.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 -- start start 2024-04-26T02:00:57.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 0x557ae9cebf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1adb20 con 0x557ae9cf0240 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1ab2f0 con 0x557ae9cef920 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc19d010 con 0x557ae9cfb0f0 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.235+0000 7f2ed3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2edbe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 0x557ae9cebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37482/0 (socket says 172.21.15.52:37482) 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 -- 172.21.15.52:0/349001381 learned_addr learned my addr 172.21.15.52:0/349001381 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ee1000700 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 -- 172.21.15.52:0/349001381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cfb0f0 msgr2=0x557ae9cf4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 -- 172.21.15.52:0/349001381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 msgr2=0x557ae9cebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 0x557ae9cebf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 -- 172.21.15.52:0/349001381 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae9cf6150 con 0x557ae9cf0240 2024-04-26T02:00:57.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2edbe00700 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 0x557ae9cebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.236+0000 7f2ed3400700 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2ec409ab90 tx=0x7f2ec4009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0a0aa5be5a6466 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2ed0c00700 1 -- 172.21.15.52:0/349001381 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec409abe0 con 0x557ae9cf0240 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2ed0c00700 1 -- 172.21.15.52:0/349001381 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec40a5550 con 0x557ae9cf0240 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2ed0c00700 1 -- 172.21.15.52:0/349001381 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec40a50c0 con 0x557ae9cf0240 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/349001381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 msgr2=0x557ae9cf9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2ec409ab90 tx=0x7f2ec4009820 comp rx=0 tx=0).stop 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/349001381 shutdown_connections 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cfb0f0 0x557ae9cf4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cef920 0x557ae9cebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/349001381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cf0240 0x557ae9cf9b00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.237+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/349001381 >> 172.21.15.52:0/349001381 conn(0x557ae9c1e6a0 msgr2=0x557ae9cf93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.238+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/349001381 shutdown_connections 2024-04-26T02:00:57.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.238+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/349001381 wait complete. 2024-04-26T02:00:57.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.238+0000 7f2eed9bdb80 1 -- start start 2024-04-26T02:00:57.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.238+0000 7f2eed9bdb80 1 Processor -- start 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9d0ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9d0e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1a97b0 con 0x557ae9cfb0f0 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1aac20 con 0x557ae9cf0240 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1a9e80 con 0x557ae9cef920 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2eed9bdb80 10 client.0 _ll_register_callbacks cb 0x557ae9bdb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2ee1000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2ee1000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37490/0 (socket says 172.21.15.52:37490) 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2ee1000700 1 -- 172.21.15.52:0/866261869 learned_addr learned my addr 172.21.15.52:0/866261869 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2edbe00700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9d0ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.239+0000 7f2ed3400700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9d0e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 -- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 msgr2=0x557ae9d0e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9d0e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 -- 172.21.15.52:0/866261869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 msgr2=0x557ae9d0e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 -- 172.21.15.52:0/866261869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae9d0efb0 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2ed3400700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9d0e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2ee1000700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2edbe00700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9d0ff20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x557ae9cfde40 tx=0x7f2ecc009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2dda16cddaed94b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2eca000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc003950 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8521+0+0 (secure 0 0 0) 0x7f2ecc029040 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.240+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ec40a5550 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f2ecc003af0 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 10 client.27031 fetch_fsmap learned FSMap version 59 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x557ae9d71a10 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec40a5550 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 59) ==== 8358+0+0 (secure 0 0 0) 0x7f2ecc047070 con 0x557ae9cef920 2024-04-26T02:00:57.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 10 client.27031 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 10 client.27031 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 20 client.27031 populate_metadata read hostname 'smithi052' 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d72070 0x557ae9d74460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 4 client.27031 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "860f4752", "release": true, "await": true}] 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.241+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "860f4752", "release": true, "await": true}) -- 0x557ae9c4a820 con 0x557ae9d72070 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.242+0000 7f2ed3400700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d72070 0x557ae9d74460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:00:57.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.242+0000 7f2ed3400700 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d72070 0x557ae9d74460 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754fd6fec41a634a server_cookie=4e78744c52561ab4 in_seq=0 out_seq=0 2024-04-26T02:00:57.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.242+0000 7f2eca000700 10 client.27031 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:57.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.275+0000 7f2eca000700 1 -- 172.21.15.52:0/866261869 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557ae9c4a820 con 0x557ae9d72070 2024-04-26T02:00:57.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.275+0000 7f2eca000700 10 client.27031 handle_command_reply: tid=0 2024-04-26T02:00:57.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.275+0000 7f2eed9bdb80 1 client.27031 shutdown 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 msgr2=0x557ae9d0ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9d0ff20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x557ae9cfde40 tx=0x7f2ecc009ca0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 shutdown_connections 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ae9d72070 0x557ae9d74460 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ae9cfb0f0 0x557ae9d0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ae9cef920 0x557ae9d0ff20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 --2- 172.21.15.52:0/866261869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ae9cf0240 0x557ae9d0e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 >> 172.21.15.52:0/866261869 conn(0x557ae9c1e6a0 msgr2=0x557ae9cf9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eca000700 0 client.27031 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 shutdown_connections 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.276+0000 7f2eed9bdb80 1 -- 172.21.15.52:0/866261869 wait complete. 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.277+0000 7f2eed9bdb80 20 client.27031 trim_cache size 0 max 16384 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 59, 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 48, 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "860f4752": { 2024-04-26T02:00:57.276 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 48, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 84.6, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RELEASED", 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:57.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:57.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:57.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:00:57.278 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:00:57.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 msgr2=0x7f2ec00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 0x7f2ec00a62f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc000c00 tx=0x7f2ecc009530 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 msgr2=0x7f2edc103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400e9a0 tx=0x7f2ec400ecb0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 shutdown_connections 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2ec00a3e30 0x7f2ec00a62f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2edc107980 0x7f2edc103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2edc101930 0x7f2edc1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 --2- 172.21.15.52:0/3300996312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2edc1022f0 0x7f2edc103a20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.278+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 >> 172.21.15.52:0/3300996312 conn(0x7f2edc0788b0 msgr2=0x7f2edc06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.279+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 shutdown_connections 2024-04-26T02:00:57.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:00:57.279+0000 7f2ee3000700 1 -- 172.21.15.52:0/3300996312 wait complete. 2024-04-26T02:00:57.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 40816 0.99 MB/sec execute 14 sec latency 33262.973 ms 2024-04-26T02:00:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:57 smithi191 ceph-mon[32745]: pgmap v503: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 3.4 MiB/s wr, 144 op/s 2024-04-26T02:00:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:57 smithi183 ceph-mon[32752]: pgmap v503: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 3.4 MiB/s wr, 144 op/s 2024-04-26T02:00:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:57 smithi052 ceph-mon[27851]: pgmap v503: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 3.4 MiB/s wr, 144 op/s 2024-04-26T02:00:58.263 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '860f4752', seconds elapsed: 42.4 2024-04-26T02:00:58.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 41547 3.87 MB/sec execute 15 sec latency 61.923 ms 2024-04-26T02:00:59.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 42151 5.57 MB/sec execute 16 sec latency 79.363 ms 2024-04-26T02:00:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:00:59 smithi191 ceph-mon[32745]: pgmap v504: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:00:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:00:59 smithi183 ceph-mon[32752]: pgmap v504: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:00:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:00:59 smithi052 ceph-mon[27851]: pgmap v504: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:01:00.723 INFO:tasks.workunit.client.0.smithi052.stdout: 10 42847 7.53 MB/sec execute 17 sec latency 68.733 ms 2024-04-26T02:01:01.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 42847 7.11 MB/sec execute 18 sec latency 1025.061 ms 2024-04-26T02:01:01.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:01 smithi052 ceph-mon[27851]: pgmap v505: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:01:01.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:01 smithi191 ceph-mon[32745]: pgmap v505: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:01:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:01 smithi183 ceph-mon[32752]: pgmap v505: 97 pgs: 97 active+clean; 486 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 3.3 MiB/s wr, 131 op/s 2024-04-26T02:01:02.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 42847 6.73 MB/sec execute 19 sec latency 2025.152 ms 2024-04-26T02:01:03.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 43230 7.35 MB/sec execute 20 sec latency 2419.326 ms 2024-04-26T02:01:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:03 smithi191 ceph-mon[32745]: pgmap v506: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 333 op/s 2024-04-26T02:01:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:03 smithi183 ceph-mon[32752]: pgmap v506: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 333 op/s 2024-04-26T02:01:03.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:03 smithi052 ceph-mon[27851]: pgmap v506: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 6.4 MiB/s wr, 333 op/s 2024-04-26T02:01:04.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 43544 7.62 MB/sec execute 21 sec latency 135.649 ms 2024-04-26T02:01:04.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:04 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:04.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:04 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:04 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:05.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 44039 8.95 MB/sec execute 22 sec latency 90.939 ms 2024-04-26T02:01:05.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:05 smithi191 ceph-mon[32745]: pgmap v507: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.7 MiB/s wr, 307 op/s 2024-04-26T02:01:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:05 smithi183 ceph-mon[32752]: pgmap v507: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.7 MiB/s wr, 307 op/s 2024-04-26T02:01:05.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:05 smithi052 ceph-mon[27851]: pgmap v507: 97 pgs: 97 active+clean; 505 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.7 MiB/s wr, 307 op/s 2024-04-26T02:01:06.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 44571 9.73 MB/sec execute 23 sec latency 68.243 ms 2024-04-26T02:01:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 -- 172.21.15.52:0/1728735880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e1010c650 msgr2=0x7f2e10108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 --2- 172.21.15.52:0/1728735880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e1010c650 0x7f2e10108f50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009900 tx=0x7f2dfc02a420 comp rx=0 tx=0).stop 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 -- 172.21.15.52:0/1728735880 shutdown_connections 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 --2- 172.21.15.52:0/1728735880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10116fe0 0x7f2e101193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 --2- 172.21.15.52:0/1728735880 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e10109490 0x7f2e101098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 --2- 172.21.15.52:0/1728735880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e1010c650 0x7f2e10108f50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 -- 172.21.15.52:0/1728735880 >> 172.21.15.52:0/1728735880 conn(0x7f2e1006d4b0 msgr2=0x7f2e1006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 -- 172.21.15.52:0/1728735880 shutdown_connections 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.551+0000 7f2e14a00640 1 -- 172.21.15.52:0/1728735880 wait complete. 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.552+0000 7f2e14a00640 1 Processor -- start 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.552+0000 7f2e14a00640 1 -- start start 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 0x7f2e10116880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1010c650 0x7f2e10113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011ccb0 con 0x7f2e10109490 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011cb30 con 0x7f2e1010c650 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e14a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011ce30 con 0x7f2e10116fe0 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1010c650 0x7f2e10113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e0e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e0e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36438/0 (socket says 172.21.15.52:36438) 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.553+0000 7f2e0e800640 1 -- 172.21.15.52:0/40887641 learned_addr learned my addr 172.21.15.52:0/40887641 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.554+0000 7f2e0e800640 1 -- 172.21.15.52:0/40887641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1010c650 msgr2=0x7f2e10113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.554+0000 7f2e0e800640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1010c650 0x7f2e10113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.554+0000 7f2e0e800640 1 -- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 msgr2=0x7f2e10116880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e0de00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 0x7f2e10116880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e0e800640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 0x7f2e10116880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e0e800640 1 -- 172.21.15.52:0/40887641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e10114740 con 0x7f2e10116fe0 2024-04-26T02:01:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e0de00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 0x7f2e10116880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e0e800640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800ccb0 tx=0x7f2e0800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e06a00640 1 -- 172.21.15.52:0/40887641 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800f450 con 0x7f2e10116fe0 2024-04-26T02:01:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e06a00640 1 -- 172.21.15.52:0/40887641 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e0800faa0 con 0x7f2e10116fe0 2024-04-26T02:01:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e06a00640 1 -- 172.21.15.52:0/40887641 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e08018b00 con 0x7f2e10116fe0 2024-04-26T02:01:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.555+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e101b3ab0 con 0x7f2e10116fe0 2024-04-26T02:01:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.556+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e101b3fb0 con 0x7f2e10116fe0 2024-04-26T02:01:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.560+0000 7f2e17993740 1 Processor -- start 2024-04-26T02:01:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.560+0000 7f2e06a00640 1 -- 172.21.15.52:0/40887641 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2e0800fc50 con 0x7f2e10116fe0 2024-04-26T02:01:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.560+0000 7f2e17993740 1 -- start start 2024-04-26T02:01:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b954c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b95a00 0x55ac93b97df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ac93af0900 con 0x55ac93b8b140 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ac93a96ef0 con 0x55ac93b8a820 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e17993740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ac93aec8c0 con 0x55ac93b95a00 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60890/0 (socket says 172.21.15.52:60890) 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e0d400640 1 -- 172.21.15.52:0/3029821120 learned_addr learned my addr 172.21.15.52:0/3029821120 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.561+0000 7f2e0d400640 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b95a00 0x55ac93b97df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 -- 172.21.15.52:0/3029821120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b95a00 msgr2=0x55ac93b97df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b95a00 0x55ac93b97df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 -- 172.21.15.52:0/3029821120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 msgr2=0x55ac93b954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 -- 172.21.15.52:0/3029821120 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac93803210 con 0x55ac93b8a820 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e06a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 0x7f2df40a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.562+0000 7f2e0d400640 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009300 tx=0x7f2e000094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fa054512f7e7627 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.563+0000 7f2e06a00640 1 -- 172.21.15.52:0/40887641 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8565+0+0 (secure 0 0 0) 0x7f2e080d0fc0 con 0x7f2e10116fe0 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.563+0000 7f2e0de00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 0x7f2df40a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.564+0000 7f2dfb000640 1 -- 172.21.15.52:0/3029821120 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00026770 con 0x55ac93b8a820 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.564+0000 7f2dfb000640 1 -- 172.21.15.52:0/3029821120 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e00026dc0 con 0x55ac93b8a820 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.564+0000 7f2dfb000640 1 -- 172.21.15.52:0/3029821120 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00013b40 con 0x55ac93b8a820 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e17993740 1 -- 172.21.15.52:0/3029821120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 msgr2=0x55ac93b8ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e17993740 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009300 tx=0x7f2e000094c0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e0de00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 0x7f2df40a6350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009850 tx=0x7f2dfc031400 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e17993740 1 -- 172.21.15.52:0/3029821120 shutdown_connections 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e17993740 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b95a00 0x55ac93b97df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.565+0000 7f2e17993740 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.566+0000 7f2e17993740 1 --2- 172.21.15.52:0/3029821120 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b8ac00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.566+0000 7f2e17993740 1 -- 172.21.15.52:0/3029821120 >> 172.21.15.52:0/3029821120 conn(0x55ac93b86d30 msgr2=0x55ac93b87110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.567+0000 7f2e17993740 1 -- 172.21.15.52:0/3029821120 shutdown_connections 2024-04-26T02:01:07.565 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.567+0000 7f2e17993740 1 -- 172.21.15.52:0/3029821120 wait complete. 2024-04-26T02:01:07.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.567+0000 7f2e17993740 1 -- start start 2024-04-26T02:01:07.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 Processor -- start 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93b93400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ac93b98630 con 0x55ac93b95a00 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ac93aaa310 con 0x55ac93b8b140 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ac93b987b0 con 0x55ac93b8a820 2024-04-26T02:01:07.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.568+0000 7f2e17993740 10 client.0 _ll_register_callbacks cb 0x55ac93a78cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:60904/0 (socket says 172.21.15.52:60904) 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36468/0 (socket says 172.21.15.52:36468) 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0de00640 1 -- 172.21.15.52:0/1770332916 learned_addr learned my addr 172.21.15.52:0/1770332916 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.569+0000 7f2e0e800640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93b93400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0de00640 1 -- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 msgr2=0x55ac93b92ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0de00640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0de00640 1 -- 172.21.15.52:0/1770332916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 msgr2=0x55ac93b93400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0de00640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93b93400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0de00640 1 -- 172.21.15.52:0/1770332916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac938cf620 con 0x55ac93b8a820 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0d400640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.570+0000 7f2e0e800640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93b93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.571+0000 7f2e0de00640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0092a0 tx=0x7f2dfc02a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad503ccca850f794 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.571+0000 7f2def400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.571+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc041070 con 0x55ac93b8a820 2024-04-26T02:01:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.571+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e00026dc0 con 0x55ac93b8a820 2024-04-26T02:01:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.571+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8565+0+0 (secure 0 0 0) 0x7f2dfc004c10 con 0x55ac93b8a820 2024-04-26T02:01:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.578+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f2dfc035050 con 0x55ac93b8a820 2024-04-26T02:01:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.578+0000 7f2e17993740 10 client.27043 fetch_fsmap learned FSMap version 60 2024-04-26T02:01:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.578+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55ac93ba4900 con 0x55ac93b8a820 2024-04-26T02:01:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.578+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e00026dc0 con 0x55ac93b8a820 2024-04-26T02:01:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.578+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 60) ==== 8354+0+0 (secure 0 0 0) 0x7f2dfc032640 con 0x55ac93b8a820 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 10 client.27043 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 10 client.27043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 20 client.27043 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93ba5e10 0x55ac93ba8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 4 client.27043 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.579+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac93b048b0 con 0x55ac93ba5e10 2024-04-26T02:01:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.580+0000 7f2e0d400640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93ba5e10 0x55ac93ba8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.581+0000 7f2def400640 10 client.27043 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.581+0000 7f2e0d400640 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93ba5e10 0x55ac93ba8230 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f5c7ea085896fc6 server_cookie=41a7c5f0d4ca92f6 in_seq=0 out_seq=0 2024-04-26T02:01:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.582+0000 7f2def400640 1 -- 172.21.15.52:0/1770332916 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac93b048b0 con 0x55ac93ba5e10 2024-04-26T02:01:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.582+0000 7f2def400640 10 client.27043 handle_command_reply: tid=0 2024-04-26T02:01:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.582+0000 7f2e17993740 1 client.27043 shutdown 2024-04-26T02:01:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.582+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 msgr2=0x55ac93b92980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.582+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0092a0 tx=0x7f2dfc02a420 comp rx=0 tx=0).stop 2024-04-26T02:01:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 shutdown_connections 2024-04-26T02:01:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93ba5e10 0x55ac93ba8230 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93b93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8b140 0x55ac93b92ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 --2- 172.21.15.52:0/1770332916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93b92980 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 >> 172.21.15.52:0/1770332916 conn(0x55ac93b86d30 msgr2=0x55ac93b8cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2def400640 0 client.27043 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.583+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 shutdown_connections 2024-04-26T02:01:07.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.584+0000 7f2e17993740 1 -- 172.21.15.52:0/1770332916 wait complete. 2024-04-26T02:01:07.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.584+0000 7f2e17993740 20 client.27043 trim_cache size 0 max 16384 2024-04-26T02:01:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.616+0000 7f2e17993740 1 Processor -- start 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 -- start start 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93ba14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b93240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 0x55ac93b85d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2e101b6c40 con 0x55ac93b8b140 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2e101b4480 con 0x55ac93b95a00 2024-04-26T02:01:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e17993740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1004fd20 con 0x55ac93b8a820 2024-04-26T02:01:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.617+0000 7f2e0de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93ba14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.618+0000 7f2e0e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 0x55ac93b85d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93ba14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36476/0 (socket says 172.21.15.52:36476) 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0de00640 1 -- 172.21.15.52:0/2308532363 learned_addr learned my addr 172.21.15.52:0/2308532363 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.618+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b93240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0e800640 1 -- 172.21.15.52:0/2308532363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 msgr2=0x55ac93ba14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0e800640 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93ba14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0e800640 1 -- 172.21.15.52:0/2308532363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 msgr2=0x55ac93b93240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0e800640 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b93240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.619+0000 7f2e0e800640 1 -- 172.21.15.52:0/2308532363 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac93b82690 con 0x55ac93b95a00 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.620+0000 7f2e0e800640 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 0x55ac93b85d10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2e080ce680 tx=0x7f2e0809d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eddf39dc7d26555f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.620+0000 7f2dfa600640 1 -- 172.21.15.52:0/2308532363 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e080b5020 con 0x55ac93b95a00 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.620+0000 7f2dfa600640 1 -- 172.21.15.52:0/2308532363 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e08099540 con 0x55ac93b95a00 2024-04-26T02:01:07.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.620+0000 7f2dfa600640 1 -- 172.21.15.52:0/2308532363 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e080a5910 con 0x55ac93b95a00 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 -- 172.21.15.52:0/2308532363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 msgr2=0x55ac93b85d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 0x55ac93b85d10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2e080ce680 tx=0x7f2e0809d720 comp rx=0 tx=0).stop 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 -- 172.21.15.52:0/2308532363 shutdown_connections 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b95a00 0x55ac93b85d10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b8b140 0x55ac93b93240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 --2- 172.21.15.52:0/2308532363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8a820 0x55ac93ba14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 -- 172.21.15.52:0/2308532363 >> 172.21.15.52:0/2308532363 conn(0x55ac93b86d30 msgr2=0x55ac93b85600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 -- 172.21.15.52:0/2308532363 shutdown_connections 2024-04-26T02:01:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.621+0000 7f2e17993740 1 -- 172.21.15.52:0/2308532363 wait complete. 2024-04-26T02:01:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.622+0000 7f2e17993740 1 -- start start 2024-04-26T02:01:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.622+0000 7f2e17993740 1 Processor -- start 2024-04-26T02:01:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e17993740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b9a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e17993740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 0x55ac93b9af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e17993740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93c0a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e17993740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2e101a9090 con 0x55ac93b95a00 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e17993740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2e101aa500 con 0x55ac93b8a820 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 0x55ac93b9af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e0d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 0x55ac93b9af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36480/0 (socket says 172.21.15.52:36480) 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e0d400640 1 -- 172.21.15.52:0/877557317 learned_addr learned my addr 172.21.15.52:0/877557317 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:07.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.623+0000 7f2e0e800640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93c0a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2e101a9760 con 0x55ac93b8b140 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b9a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e17993740 10 client.0 _ll_register_callbacks cb 0x7f2dfc0044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 -- 172.21.15.52:0/877557317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 msgr2=0x55ac93b9af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 0x55ac93b9af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 -- 172.21.15.52:0/877557317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 msgr2=0x55ac93c0a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:07.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93c0a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.624+0000 7f2e0de00640 1 -- 172.21.15.52:0/877557317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac93c0a8f0 con 0x55ac93b8a820 2024-04-26T02:01:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.625+0000 7f2e0de00640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b9a9d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc000c00 tx=0x7f2dfc004a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cee1398310eed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.625+0000 7f2def400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:01:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc035050 con 0x55ac93b8a820 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8565+0+0 (secure 0 0 0) 0x7f2dfc0035f0 con 0x55ac93b8a820 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e08099540 con 0x55ac93b8a820 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f2dfc06a050 con 0x55ac93b8a820 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2e17993740 10 client.27053 fetch_fsmap learned FSMap version 60 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55ac93c0ba60 con 0x55ac93b8a820 2024-04-26T02:01:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e08099540 con 0x55ac93b8a820 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.626+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 60) ==== 8354+0+0 (secure 0 0 0) 0x7f2dfc048600 con 0x55ac93b8a820 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 10 client.27053 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 10 client.27053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 20 client.27053 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93c0c230 0x55ac93c0e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 4 client.27053 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac93afe320 con 0x55ac93c0c230 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.627+0000 7f2e0d400640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93c0c230 0x55ac93c0e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.628+0000 7f2e0d400640 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93c0c230 0x55ac93c0e650 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0403a215053bac2 server_cookie=b6180022eb04bb17 in_seq=0 out_seq=0 2024-04-26T02:01:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.628+0000 7f2def400640 10 client.27053 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:07.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 44923 9.87 MB/sec execute 24 sec latency 95.653 ms 2024-04-26T02:01:07.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:01:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:01:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:01:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:01:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.776+0000 7f2def400640 1 -- 172.21.15.52:0/877557317 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac93afe320 con 0x55ac93c0c230 2024-04-26T02:01:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.776+0000 7f2def400640 10 client.27053 handle_command_reply: tid=0 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 client.27053 shutdown 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 msgr2=0x55ac93b9a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b9a9d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc000c00 tx=0x7f2dfc004a90 comp rx=0 tx=0).stop 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 shutdown_connections 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ac93c0c230 0x55ac93c0e650 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ac93b95a00 0x55ac93c0a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac93b8b140 0x55ac93b9af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 --2- 172.21.15.52:0/877557317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ac93b8a820 0x55ac93b9a9d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 >> 172.21.15.52:0/877557317 conn(0x55ac93b86d30 msgr2=0x55ac93b94a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2def400640 0 client.27053 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.777+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 shutdown_connections 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.778+0000 7f2e17993740 1 -- 172.21.15.52:0/877557317 wait complete. 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.778+0000 7f2e17993740 20 client.27053 trim_cache size 0 max 16384 2024-04-26T02:01:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 msgr2=0x7f2df40a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 0x7f2df40a6350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009850 tx=0x7f2dfc031400 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 msgr2=0x7f2e10113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800ccb0 tx=0x7f2e0800a620 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 shutdown_connections 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2df40a3e90 0x7f2df40a6350 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e10116fe0 0x7f2e10113eb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1010c650 0x7f2e10113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 --2- 172.21.15.52:0/40887641 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e10109490 0x7f2e10116880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 >> 172.21.15.52:0/40887641 conn(0x7f2e1006d4b0 msgr2=0x7f2e10118600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 shutdown_connections 2024-04-26T02:01:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:07.779+0000 7f2e14a00640 1 -- 172.21.15.52:0/40887641 wait complete. 2024-04-26T02:01:07.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:07 smithi191 ceph-mon[32745]: pgmap v508: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.6 MiB/s wr, 605 op/s 2024-04-26T02:01:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:07 smithi191 ceph-mon[32745]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:01:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:07 smithi183 ceph-mon[32752]: pgmap v508: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.6 MiB/s wr, 605 op/s 2024-04-26T02:01:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:07 smithi183 ceph-mon[32752]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:01:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:07 smithi052 ceph-mon[27851]: pgmap v508: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.6 MiB/s wr, 605 op/s 2024-04-26T02:01:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:07 smithi052 ceph-mon[27851]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T02:01:08.724 INFO:tasks.workunit.client.0.smithi052.stdout: 10 45767 11.20 MB/sec execute 25 sec latency 97.106 ms 2024-04-26T02:01:09.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 46672 12.84 MB/sec execute 26 sec latency 103.441 ms 2024-04-26T02:01:09.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:09 smithi191 ceph-mon[32745]: pgmap v510: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:09 smithi183 ceph-mon[32752]: pgmap v510: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:09.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:09 smithi052 ceph-mon[27851]: pgmap v510: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:10.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 47401 14.19 MB/sec execute 27 sec latency 59.463 ms 2024-04-26T02:01:11.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 47861 14.42 MB/sec execute 28 sec latency 64.021 ms 2024-04-26T02:01:11.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:11 smithi191 ceph-mon[32745]: pgmap v511: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:11 smithi183 ceph-mon[32752]: pgmap v511: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:11.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:11 smithi052 ceph-mon[27851]: pgmap v511: 97 pgs: 97 active+clean; 515 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.9 MiB/s wr, 599 op/s 2024-04-26T02:01:12.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 48339 14.36 MB/sec execute 29 sec latency 267.242 ms 2024-04-26T02:01:13.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 49238 15.49 MB/sec execute 30 sec latency 47.790 ms 2024-04-26T02:01:13.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:13 smithi191 ceph-mon[32745]: pgmap v512: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 5.4 MiB/s wr, 667 op/s 2024-04-26T02:01:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:13 smithi191 ceph-mon[32745]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:01:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:13 smithi183 ceph-mon[32752]: pgmap v512: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 5.4 MiB/s wr, 667 op/s 2024-04-26T02:01:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:13 smithi183 ceph-mon[32752]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:01:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:13 smithi052 ceph-mon[27851]: pgmap v512: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 5.4 MiB/s wr, 667 op/s 2024-04-26T02:01:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:13 smithi052 ceph-mon[27851]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T02:01:14.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 50170 16.73 MB/sec execute 31 sec latency 80.396 ms 2024-04-26T02:01:15.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 50822 17.59 MB/sec execute 32 sec latency 54.325 ms 2024-04-26T02:01:15.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:15 smithi191 ceph-mon[32745]: pgmap v514: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-04-26T02:01:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:15 smithi183 ceph-mon[32752]: pgmap v514: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-04-26T02:01:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:15 smithi052 ceph-mon[27851]: pgmap v514: 97 pgs: 97 active+clean; 532 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-04-26T02:01:16.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 51345 17.91 MB/sec execute 33 sec latency 139.145 ms 2024-04-26T02:01:17.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 52010 18.30 MB/sec execute 34 sec latency 80.878 ms 2024-04-26T02:01:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:17 smithi191 ceph-mon[32745]: pgmap v515: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 572 op/s 2024-04-26T02:01:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:17 smithi183 ceph-mon[32752]: pgmap v515: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 572 op/s 2024-04-26T02:01:17.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:17 smithi052 ceph-mon[27851]: pgmap v515: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.7 MiB/s wr, 572 op/s 2024-04-26T02:01:18.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 52624 18.75 MB/sec execute 35 sec latency 113.143 ms 2024-04-26T02:01:19.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 53375 19.19 MB/sec execute 36 sec latency 50.849 ms 2024-04-26T02:01:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:19 smithi191 ceph-mon[32745]: pgmap v516: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:19 smithi183 ceph-mon[32752]: pgmap v516: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:19 smithi052 ceph-mon[27851]: pgmap v516: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:20.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 54000 19.82 MB/sec execute 37 sec latency 51.367 ms 2024-04-26T02:01:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.958+0000 7f27ce000640 1 -- 172.21.15.52:0/541435548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 msgr2=0x7f27c8106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.958+0000 7f27ce000640 1 --2- 172.21.15.52:0/541435548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8106590 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009ca0 tx=0x7f27c002a950 comp rx=0 tx=0).stop 2024-04-26T02:01:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 -- 172.21.15.52:0/541435548 shutdown_connections 2024-04-26T02:01:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 --2- 172.21.15.52:0/541435548 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8106590 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 --2- 172.21.15.52:0/541435548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 0x7f27c8105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 --2- 172.21.15.52:0/541435548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c81010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 -- 172.21.15.52:0/541435548 >> 172.21.15.52:0/541435548 conn(0x7f27c80fc770 msgr2=0x7f27c80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.960+0000 7f27ce000640 1 -- 172.21.15.52:0/541435548 shutdown_connections 2024-04-26T02:01:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.961+0000 7f27ce000640 1 -- 172.21.15.52:0/541435548 wait complete. 2024-04-26T02:01:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.962+0000 7f27ce000640 1 Processor -- start 2024-04-26T02:01:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.963+0000 7f27ce000640 1 -- start start 2024-04-26T02:01:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.964+0000 7f27ce000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c810ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.965+0000 7f27ce000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 0x7f27c8109030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.965+0000 7f27ce000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.965+0000 7f27ce000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81185a0 con 0x7f27c8100d00 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.965+0000 7f27ce000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27c8118420 con 0x7f27c8106100 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.965+0000 7f27ce000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27c8118720 con 0x7f27c8101620 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.966+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c810ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.966+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c810ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33796/0 (socket says 172.21.15.52:33796) 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.966+0000 7f27ccc00640 1 -- 172.21.15.52:0/2969381132 learned_addr learned my addr 172.21.15.52:0/2969381132 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27c7e00640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 0x7f27c8109030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 -- 172.21.15.52:0/2969381132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 msgr2=0x7f27c8109030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:20.966 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 0x7f27c8109030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 -- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 msgr2=0x7f27c810ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c810ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.967+0000 7f27cd600640 1 -- 172.21.15.52:0/2969381132 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c8109be0 con 0x7f27c8106100 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.968+0000 7f27cd600640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8109570 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005e30 tx=0x7f27c002fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.968+0000 7f27c5600640 1 -- 172.21.15.52:0/2969381132 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0034040 con 0x7f27c8106100 2024-04-26T02:01:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.968+0000 7f27c5600640 1 -- 172.21.15.52:0/2969381132 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c00308f0 con 0x7f27c8106100 2024-04-26T02:01:20.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.968+0000 7f27c5600640 1 -- 172.21.15.52:0/2969381132 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0004540 con 0x7f27c8106100 2024-04-26T02:01:20.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.969+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c8109e70 con 0x7f27c8106100 2024-04-26T02:01:20.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.970+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c810de00 con 0x7f27c8106100 2024-04-26T02:01:20.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.974+0000 7f27c5600640 1 -- 172.21.15.52:0/2969381132 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f27c0037020 con 0x7f27c8106100 2024-04-26T02:01:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.976+0000 7f27c5600640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 0x7f27b00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.976+0000 7f27c5600640 1 -- 172.21.15.52:0/2969381132 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f27c00f16d0 con 0x7f27c8106100 2024-04-26T02:01:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.976+0000 7f27d23fe740 1 Processor -- start 2024-04-26T02:01:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 -- start start 2024-04-26T02:01:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4379b0 0x562e9a437d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a442b90 0x562e9a444f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562e9a39d900 con 0x562e9a4379b0 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562e9a343ef0 con 0x562e9a4382d0 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.977+0000 7f27d23fe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562e9a3998c0 con 0x562e9a442b90 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27ccc00640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 0x7f27b00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4379b0 0x562e9a437d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48062/0 (socket says 172.21.15.52:48062) 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 -- 172.21.15.52:0/3893696825 learned_addr learned my addr 172.21.15.52:0/3893696825 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27cd600640 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a442b90 0x562e9a444f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 -- 172.21.15.52:0/3893696825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a442b90 msgr2=0x562e9a444f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a442b90 0x562e9a444f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 -- 172.21.15.52:0/3893696825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4379b0 msgr2=0x562e9a437d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4379b0 0x562e9a437d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.978+0000 7f27c7e00640 1 -- 172.21.15.52:0/3893696825 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e9a0b0210 con 0x562e9a4382d0 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.979+0000 7f27c7e00640 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009b60 tx=0x7f27b40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32cb30102854cc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.979+0000 7f27b9c00640 1 -- 172.21.15.52:0/3893696825 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4009e50 con 0x562e9a4382d0 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.979+0000 7f27b9c00640 1 -- 172.21.15.52:0/3893696825 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27b4016070 con 0x562e9a4382d0 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.979+0000 7f27b9c00640 1 -- 172.21.15.52:0/3893696825 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b401b470 con 0x562e9a4382d0 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 -- 172.21.15.52:0/3893696825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 msgr2=0x562e9a442650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009b60 tx=0x7f27b40117b0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 -- 172.21.15.52:0/3893696825 shutdown_connections 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a442b90 0x562e9a444f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a442650 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 --2- 172.21.15.52:0/3893696825 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4379b0 0x562e9a437d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.980+0000 7f27d23fe740 1 -- 172.21.15.52:0/3893696825 >> 172.21.15.52:0/3893696825 conn(0x562e9a433ec0 msgr2=0x562e9a4342a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.981+0000 7f27d23fe740 1 -- 172.21.15.52:0/3893696825 shutdown_connections 2024-04-26T02:01:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.981+0000 7f27d23fe740 1 -- 172.21.15.52:0/3893696825 wait complete. 2024-04-26T02:01:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.981+0000 7f27ccc00640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 0x7f27b00a6250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097f0 tx=0x7f27bc011a00 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.981+0000 7f27d23fe740 1 -- start start 2024-04-26T02:01:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.981+0000 7f27d23fe740 1 Processor -- start 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a440020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a442b90 0x562e9a440560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27d23fe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562e9a4457c0 con 0x562e9a442b90 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27d23fe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562e9a357310 con 0x562e9a4382d0 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58022/0 (socket says 172.21.15.52:58022) 2024-04-26T02:01:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27ccc00640 1 -- 172.21.15.52:0/99672988 learned_addr learned my addr 172.21.15.52:0/99672988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.982+0000 7f27c7e00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a440020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562e9a445940 con 0x562e9a4379b0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27d23fe740 10 client.0 _ll_register_callbacks cb 0x562e9a325cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27cd600640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a442b90 0x562e9a440560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27ccc00640 1 -- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 msgr2=0x562e9a440020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27ccc00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a440020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27ccc00640 1 -- 172.21.15.52:0/99672988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a442b90 msgr2=0x562e9a440560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27ccc00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a442b90 0x562e9a440560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.983+0000 7f27ccc00640 1 -- 172.21.15.52:0/99672988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e9a17c620 con 0x562e9a4379b0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27ccc00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00fa40 tx=0x7f27bc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b55ba88453d293e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27ae000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc025070 con 0x562e9a4379b0 2024-04-26T02:01:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f27bc01dbc0 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27b4016070 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f27bc01ddc0 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27d23fe740 10 client.27061 fetch_fsmap learned FSMap version 60 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.984+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x562e9a4517c0 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27b4016070 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 60) ==== 8354+0+0 (secure 0 0 0) 0x7f27bc01e050 con 0x562e9a4379b0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 10 client.27061 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 10 client.27061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 20 client.27061 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a452cf0 0x562e9a455140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 4 client.27061 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e9a3b18b0 con 0x562e9a452cf0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.985+0000 7f27c7e00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a452cf0 0x562e9a455140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.986+0000 7f27c7e00640 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a452cf0 0x562e9a455140 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd31321ddce0a5ed server_cookie=3feaa1608a680de7 in_seq=0 out_seq=0 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.986+0000 7f27ae000640 10 client.27061 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.987+0000 7f27ae000640 1 -- 172.21.15.52:0/99672988 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e9a3b18b0 con 0x562e9a452cf0 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.987+0000 7f27ae000640 10 client.27061 handle_command_reply: tid=0 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.987+0000 7f27d23fe740 1 client.27061 shutdown 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 msgr2=0x562e9a43fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:20.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00fa40 tx=0x7f27bc005f90 comp rx=0 tx=0).stop 2024-04-26T02:01:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 shutdown_connections 2024-04-26T02:01:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a452cf0 0x562e9a455140 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a442b90 0x562e9a440560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a4382d0 0x562e9a440020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 --2- 172.21.15.52:0/99672988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a43fae0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:20.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 >> 172.21.15.52:0/99672988 conn(0x562e9a433ec0 msgr2=0x562e9a439c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:20.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27ae000640 0 client.27061 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:20.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 shutdown_connections 2024-04-26T02:01:20.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.988+0000 7f27d23fe740 1 -- 172.21.15.52:0/99672988 wait complete. 2024-04-26T02:01:20.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:20.989+0000 7f27d23fe740 20 client.27061 trim_cache size 0 max 16384 2024-04-26T02:01:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.011+0000 7f27d23fe740 1 Processor -- start 2024-04-26T02:01:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.011+0000 7f27d23fe740 1 -- start start 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.011+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a454c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a43a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27d23fe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81bf090 con 0x562e9a4382d0 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27d23fe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81bc8d0 con 0x562e9a442b90 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27d23fe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81ba740 con 0x562e9a4379b0 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58028/0 (socket says 172.21.15.52:58028) 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27ccc00640 1 -- 172.21.15.52:0/4128847466 learned_addr learned my addr 172.21.15.52:0/4128847466 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.012+0000 7f27c7e00640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a454c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.013+0000 7f27cd600640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a43a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.013+0000 7f27ccc00640 1 -- 172.21.15.52:0/4128847466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 msgr2=0x562e9a43a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.013+0000 7f27ccc00640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a43a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.013+0000 7f27ccc00640 1 -- 172.21.15.52:0/4128847466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 msgr2=0x562e9a454c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.013+0000 7f27ccc00640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a454c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.014+0000 7f27ccc00640 1 -- 172.21.15.52:0/4128847466 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e9a438100 con 0x562e9a4379b0 2024-04-26T02:01:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.014+0000 7f27c7e00640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a454c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.014+0000 7f27ccc00640 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005820 tx=0x7f27bc00f440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7512522037541f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.014+0000 7f27b9200640 1 -- 172.21.15.52:0/4128847466 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc025070 con 0x562e9a4379b0 2024-04-26T02:01:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.015+0000 7f27b9200640 1 -- 172.21.15.52:0/4128847466 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27bc01dbc0 con 0x562e9a4379b0 2024-04-26T02:01:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.015+0000 7f27b9200640 1 -- 172.21.15.52:0/4128847466 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc033620 con 0x562e9a4379b0 2024-04-26T02:01:21.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 -- 172.21.15.52:0/4128847466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 msgr2=0x562e9a44e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005820 tx=0x7f27bc00f440 comp rx=0 tx=0).stop 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 -- 172.21.15.52:0/4128847466 shutdown_connections 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a43a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a454c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 --2- 172.21.15.52:0/4128847466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a44e680 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 -- 172.21.15.52:0/4128847466 >> 172.21.15.52:0/4128847466 conn(0x562e9a433ec0 msgr2=0x562e9a43e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.016+0000 7f27d23fe740 1 -- 172.21.15.52:0/4128847466 shutdown_connections 2024-04-26T02:01:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.017+0000 7f27d23fe740 1 -- 172.21.15.52:0/4128847466 wait complete. 2024-04-26T02:01:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.017+0000 7f27d23fe740 1 -- start start 2024-04-26T02:01:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.017+0000 7f27d23fe740 1 Processor -- start 2024-04-26T02:01:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a4b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a4b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a447f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81bad90 con 0x562e9a4382d0 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81bc200 con 0x562e9a442b90 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f27c81bb460 con 0x562e9a4379b0 2024-04-26T02:01:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27d23fe740 10 client.0 _ll_register_callbacks cb 0x7f27bc00f640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27cd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a447f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27ccc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a4b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27cd600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a447f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:48084/0 (socket says 172.21.15.52:48084) 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.018+0000 7f27cd600640 1 -- 172.21.15.52:0/1758048688 learned_addr learned my addr 172.21.15.52:0/1758048688 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27c7e00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a4b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 -- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 msgr2=0x562e9a447f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a447f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 -- 172.21.15.52:0/1758048688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 msgr2=0x562e9a4b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a4b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 -- 172.21.15.52:0/1758048688 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e9a448660 con 0x562e9a4379b0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ccc00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a4b1690 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f27bc029530 tx=0x7f27bc0055e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ab640429b92506d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:21.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ae000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:21.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.019+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc025070 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f27bc0298b0 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27bc01dbc0 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f27bc029ab0 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27d23fe740 10 client.27067 fetch_fsmap learned FSMap version 60 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x562e9a4b8e40 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.020+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27bc01dbc0 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 60) ==== 8354+0+0 (secure 0 0 0) 0x7f27bc0334c0 con 0x562e9a4379b0 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 10 client.27067 fetch_fsmap finished waiting for FSMap version 60 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 10 client.27067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 20 client.27067 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a4b9450 0x562e9a4bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 4 client.27067 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e9a3ab320 con 0x562e9a4b9450 2024-04-26T02:01:21.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.021+0000 7f27c7e00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a4b9450 0x562e9a4bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.022+0000 7f27c7e00640 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a4b9450 0x562e9a4bb870 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1590335705f4f5a server_cookie=9faa54d950aa42b1 in_seq=0 out_seq=0 2024-04-26T02:01:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.022+0000 7f27ae000640 10 client.27067 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.220+0000 7f27ae000640 1 -- 172.21.15.52:0/1758048688 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e9a3ab320 con 0x562e9a4b9450 2024-04-26T02:01:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.220+0000 7f27ae000640 10 client.27067 handle_command_reply: tid=0 2024-04-26T02:01:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.220+0000 7f27d23fe740 1 client.27067 shutdown 2024-04-26T02:01:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 msgr2=0x562e9a4b1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a4b1690 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f27bc029530 tx=0x7f27bc0055e0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 shutdown_connections 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562e9a4b9450 0x562e9a4bb870 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27ae000640 0 client.27067 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562e9a442b90 0x562e9a447f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562e9a4382d0 0x562e9a4b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 --2- 172.21.15.52:0/1758048688 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e9a4379b0 0x562e9a4b1690 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 >> 172.21.15.52:0/1758048688 conn(0x562e9a433ec0 msgr2=0x562e9a432790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 shutdown_connections 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.221+0000 7f27d23fe740 1 -- 172.21.15.52:0/1758048688 wait complete. 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.222+0000 7f27d23fe740 20 client.27067 trim_cache size 0 max 16384 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:01:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:01:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:01:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 msgr2=0x7f27b00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 0x7f27b00a6250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097f0 tx=0x7f27bc011a00 comp rx=0 tx=0).stop 2024-04-26T02:01:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 msgr2=0x7f27c8109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8109570 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005e30 tx=0x7f27c002fc50 comp rx=0 tx=0).stop 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 shutdown_connections 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f27b00a3d90 0x7f27b00a6250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f27c8106100 0x7f27c8109570 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27c8101620 0x7f27c8109030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 --2- 172.21.15.52:0/2969381132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f27c8100d00 0x7f27c810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 >> 172.21.15.52:0/2969381132 conn(0x7f27c80fc770 msgr2=0x7f27c80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 shutdown_connections 2024-04-26T02:01:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:21.223+0000 7f27ce000640 1 -- 172.21.15.52:0/2969381132 wait complete. 2024-04-26T02:01:21.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 54588 19.86 MB/sec execute 38 sec latency 202.243 ms 2024-04-26T02:01:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:21 smithi191 ceph-mon[32745]: pgmap v517: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:21 smithi183 ceph-mon[32752]: pgmap v517: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:21 smithi052 ceph-mon[27851]: pgmap v517: 97 pgs: 97 active+clean; 562 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 555 op/s 2024-04-26T02:01:22.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 55457 20.70 MB/sec execute 39 sec latency 48.171 ms 2024-04-26T02:01:23.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 55626 20.34 MB/sec execute 40 sec latency 478.984 ms 2024-04-26T02:01:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:23 smithi191 ceph-mon[32745]: pgmap v518: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.6 MiB/s wr, 506 op/s 2024-04-26T02:01:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:23 smithi183 ceph-mon[32752]: pgmap v518: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.6 MiB/s wr, 506 op/s 2024-04-26T02:01:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:23 smithi052 ceph-mon[27851]: pgmap v518: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.6 MiB/s wr, 506 op/s 2024-04-26T02:01:24.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 56285 20.91 MB/sec execute 41 sec latency 59.286 ms 2024-04-26T02:01:24.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:24 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:24.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:24 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:24.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:24 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:25.725 INFO:tasks.workunit.client.0.smithi052.stdout: 10 56878 21.11 MB/sec execute 42 sec latency 65.574 ms 2024-04-26T02:01:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:25 smithi191 ceph-mon[32745]: pgmap v519: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 465 op/s 2024-04-26T02:01:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:25 smithi183 ceph-mon[32752]: pgmap v519: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 465 op/s 2024-04-26T02:01:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:25 smithi052 ceph-mon[27851]: pgmap v519: 97 pgs: 97 active+clean; 600 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 465 op/s 2024-04-26T02:01:26.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 57561 21.34 MB/sec execute 43 sec latency 53.626 ms 2024-04-26T02:01:26.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:26 smithi191 ceph-mon[32745]: pgmap v520: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 12 MiB/s wr, 791 op/s 2024-04-26T02:01:26.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:26 smithi183 ceph-mon[32752]: pgmap v520: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 12 MiB/s wr, 791 op/s 2024-04-26T02:01:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:26 smithi052 ceph-mon[27851]: pgmap v520: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 12 MiB/s wr, 791 op/s 2024-04-26T02:01:27.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 58363 21.85 MB/sec execute 44 sec latency 93.630 ms 2024-04-26T02:01:28.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 58579 21.58 MB/sec execute 45 sec latency 496.937 ms 2024-04-26T02:01:29.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:29 smithi191 ceph-mon[32745]: pgmap v521: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:29.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:29 smithi183 ceph-mon[32752]: pgmap v521: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:29.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:29 smithi052 ceph-mon[27851]: pgmap v521: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:29.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 59219 21.89 MB/sec execute 46 sec latency 675.195 ms 2024-04-26T02:01:30.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 59747 22.08 MB/sec execute 47 sec latency 102.644 ms 2024-04-26T02:01:31.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:31 smithi052 ceph-mon[27851]: pgmap v522: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:31.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:31 smithi191 ceph-mon[32745]: pgmap v522: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:31.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:31 smithi183 ceph-mon[32752]: pgmap v522: 97 pgs: 97 active+clean; 614 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.8 MiB/s wr, 586 op/s 2024-04-26T02:01:31.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 60458 22.46 MB/sec execute 48 sec latency 48.044 ms 2024-04-26T02:01:32.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 61206 22.76 MB/sec execute 49 sec latency 71.142 ms 2024-04-26T02:01:33.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:33 smithi191 ceph-mon[32745]: pgmap v523: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-26T02:01:33.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:33 smithi183 ceph-mon[32752]: pgmap v523: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-26T02:01:33.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:33 smithi052 ceph-mon[27851]: pgmap v523: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-26T02:01:33.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 61860 23.05 MB/sec execute 50 sec latency 66.833 ms 2024-04-26T02:01:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 -- 172.21.15.52:0/3565379435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 msgr2=0x7f0990105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 --2- 172.21.15.52:0/3565379435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f0990105db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f097c009860 tx=0x7f097c02a460 comp rx=0 tx=0).stop 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 -- 172.21.15.52:0/3565379435 shutdown_connections 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 --2- 172.21.15.52:0/3565379435 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f0990106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 --2- 172.21.15.52:0/3565379435 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f0990105db0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 --2- 172.21.15.52:0/3565379435 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 0x7f09901013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.424+0000 7f0997400640 1 -- 172.21.15.52:0/3565379435 >> 172.21.15.52:0/3565379435 conn(0x7f0990077fd0 msgr2=0x7f09900783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.425+0000 7f0997400640 1 -- 172.21.15.52:0/3565379435 shutdown_connections 2024-04-26T02:01:34.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.425+0000 7f0997400640 1 -- 172.21.15.52:0/3565379435 wait complete. 2024-04-26T02:01:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.426+0000 7f0997400640 1 Processor -- start 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.427+0000 7f0997400640 1 -- start start 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 0x7f09901a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f09901a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09901187e0 con 0x7f09901062f0 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0990118660 con 0x7f0990100fd0 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.428+0000 7f0997400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0990118960 con 0x7f09901018f0 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.429+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f09901a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.429+0000 7f0995400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.429+0000 7f0995400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34480/0 (socket says 172.21.15.52:34480) 2024-04-26T02:01:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.429+0000 7f0995400640 1 -- 172.21.15.52:0/3822610685 learned_addr learned my addr 172.21.15.52:0/3822610685 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.429+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f09901a4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:42984/0 (socket says 172.21.15.52:42984) 2024-04-26T02:01:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0994a00640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 0x7f09901a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 -- 172.21.15.52:0/3822610685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 msgr2=0x7f09901a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f09901a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 -- 172.21.15.52:0/3822610685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 msgr2=0x7f09901a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 0x7f09901a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 -- 172.21.15.52:0/3822610685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099019f250 con 0x7f09901062f0 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.430+0000 7f0995400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f098800cce0 tx=0x7f098800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.432+0000 7f098d600640 1 -- 172.21.15.52:0/3822610685 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098800f450 con 0x7f09901062f0 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.432+0000 7f098d600640 1 -- 172.21.15.52:0/3822610685 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f098800faa0 con 0x7f09901062f0 2024-04-26T02:01:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.432+0000 7f098d600640 1 -- 172.21.15.52:0/3822610685 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988018b70 con 0x7f09901062f0 2024-04-26T02:01:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.432+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099019f540 con 0x7f09901062f0 2024-04-26T02:01:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.432+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099006b970 con 0x7f09901062f0 2024-04-26T02:01:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.437+0000 7f098d600640 1 -- 172.21.15.52:0/3822610685 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0988018d10 con 0x7f09901062f0 2024-04-26T02:01:34.436 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 Processor -- start 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 -- start start 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd7cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd89df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebce2900 con 0x55d7ebd7d140 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebc88ef0 con 0x55d7ebd7c820 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.438+0000 7f099a26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebcde8c0 con 0x55d7ebd87a00 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34490/0 (socket says 172.21.15.52:34490) 2024-04-26T02:01:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 -- 172.21.15.52:0/3690008407 learned_addr learned my addr 172.21.15.52:0/3690008407 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 -- 172.21.15.52:0/3690008407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 msgr2=0x55d7ebd89df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd89df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.439+0000 7f0994a00640 1 -- 172.21.15.52:0/3690008407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 msgr2=0x55d7ebd7cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.440+0000 7f098fe00640 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd7cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.440+0000 7f0994a00640 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd7cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.440+0000 7f0994a00640 1 -- 172.21.15.52:0/3690008407 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7eb9f5210 con 0x55d7ebd7d140 2024-04-26T02:01:34.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.440+0000 7f098d600640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 0x7f09780a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.440+0000 7f098d600640 1 -- 172.21.15.52:0/3822610685 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f09880d0d80 con 0x7f09901062f0 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.441+0000 7f0994a00640 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f098400bc70 tx=0x7f098400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46827629a42eb36b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.441+0000 7f098fe00640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 0x7f09780a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.441+0000 7f0981c00640 1 -- 172.21.15.52:0/3690008407 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984015070 con 0x55d7ebd7d140 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.441+0000 7f0981c00640 1 -- 172.21.15.52:0/3690008407 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09840054d0 con 0x55d7ebd7d140 2024-04-26T02:01:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.441+0000 7f0981c00640 1 -- 172.21.15.52:0/3690008407 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984011440 con 0x55d7ebd7d140 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f098fe00640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 0x7f09780a63e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09901a02a0 tx=0x7f097c005c90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 -- 172.21.15.52:0/3690008407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 msgr2=0x55d7ebd874c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f098400bc70 tx=0x7f098400f730 comp rx=0 tx=0).stop 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 -- 172.21.15.52:0/3690008407 shutdown_connections 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd89df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd874c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 --2- 172.21.15.52:0/3690008407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd7cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.442+0000 7f099a26e740 1 -- 172.21.15.52:0/3690008407 >> 172.21.15.52:0/3690008407 conn(0x55d7ebd78d30 msgr2=0x55d7ebd79110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.443+0000 7f099a26e740 1 -- 172.21.15.52:0/3690008407 shutdown_connections 2024-04-26T02:01:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.443+0000 7f099a26e740 1 -- 172.21.15.52:0/3690008407 wait complete. 2024-04-26T02:01:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.443+0000 7f099a26e740 1 -- start start 2024-04-26T02:01:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.443+0000 7f099a26e740 1 Processor -- start 2024-04-26T02:01:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd85450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebd8a630 con 0x55d7ebd7d140 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebc9c310 con 0x55d7ebd7c820 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.444+0000 7f099a26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7ebd8a7b0 con 0x55d7ebd87a00 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f099a26e740 10 client.0 _ll_register_callbacks cb 0x55d7ebc6acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f0994a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f0994a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:52756/0 (socket says 172.21.15.52:52756) 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f0994a00640 1 -- 172.21.15.52:0/4005263507 learned_addr learned my addr 172.21.15.52:0/4005263507 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34494/0 (socket says 172.21.15.52:34494) 2024-04-26T02:01:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f0995400640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd85450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 -- 172.21.15.52:0/4005263507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 msgr2=0x55d7ebd85450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd85450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 -- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 msgr2=0x55d7ebd849d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f098fe00640 1 -- 172.21.15.52:0/4005263507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ebac1620 con 0x55d7ebd7d140 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.445+0000 7f0994a00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.446+0000 7f098fe00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f097c009830 tx=0x7f097c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6854dc759580885 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.446+0000 7f0976000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.446+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c004230 con 0x55d7ebd7d140 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.446+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f097c0043d0 con 0x55d7ebd7d140 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.446+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09840054d0 con 0x55d7ebd7d140 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.447+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f097c0046c0 con 0x55d7ebd7d140 2024-04-26T02:01:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.447+0000 7f099a26e740 10 client.16923 fetch_fsmap learned FSMap version 62 2024-04-26T02:01:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.447+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d7ebd968e0 con 0x55d7ebd7d140 2024-04-26T02:01:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.447+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09840054d0 con 0x55d7ebd7d140 2024-04-26T02:01:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 62) ==== 8346+0+0 (secure 0 0 0) 0x7f097c044070 con 0x55d7ebd7d140 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 10 client.16923 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 10 client.16923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 20 client.16923 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebd97da0 0x55d7ebd9a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 4 client.16923 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7ebcf68b0 con 0x55d7ebd97da0 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.448+0000 7f0994a00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebd97da0 0x55d7ebd9a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.449+0000 7f0994a00640 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebd97da0 0x55d7ebd9a230 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad5bdc731e5f5141 server_cookie=1e86946b30da4a8a in_seq=0 out_seq=0 2024-04-26T02:01:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.449+0000 7f0976000640 10 client.16923 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.450+0000 7f0976000640 1 -- 172.21.15.52:0/4005263507 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7ebcf68b0 con 0x55d7ebd97da0 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.450+0000 7f0976000640 10 client.16923 handle_command_reply: tid=0 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.450+0000 7f099a26e740 1 client.16923 shutdown 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.450+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 msgr2=0x55d7ebd84f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.450+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f097c009830 tx=0x7f097c005f90 comp rx=0 tx=0).stop 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 shutdown_connections 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebd97da0 0x55d7ebd9a230 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd87a00 0x55d7ebd85450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7d140 0x55d7ebd84f10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 --2- 172.21.15.52:0/4005263507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd7c820 0x55d7ebd849d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f0976000640 0 client.16923 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 >> 172.21.15.52:0/4005263507 conn(0x55d7ebd78d30 msgr2=0x55d7ebd7eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 shutdown_connections 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 1 -- 172.21.15.52:0/4005263507 wait complete. 2024-04-26T02:01:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.451+0000 7f099a26e740 20 client.16923 trim_cache size 0 max 16384 2024-04-26T02:01:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.472+0000 7f099a26e740 1 Processor -- start 2024-04-26T02:01:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 -- start start 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd934f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd79110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09901bf230 con 0x55d7ebd7c820 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f09901bcaa0 con 0x55d7ebd87a00 2024-04-26T02:01:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f099a26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f099004fce0 con 0x55d7ebd7d140 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43010/0 (socket says 172.21.15.52:43010) 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.473+0000 7f098fe00640 1 -- 172.21.15.52:0/3517365926 learned_addr learned my addr 172.21.15.52:0/3517365926 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0994a00640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd934f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 -- 172.21.15.52:0/3517365926 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 msgr2=0x55d7ebd79110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0995400640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd79110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd79110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 -- 172.21.15.52:0/3517365926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 msgr2=0x55d7ebd934f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd934f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 -- 172.21.15.52:0/3517365926 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ebd7c020 con 0x55d7ebd7d140 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0994a00640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd934f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0995400640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd79110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f098fe00640 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f097c003ca0 tx=0x7f097c003850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79bb87ae26a8e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0981200640 1 -- 172.21.15.52:0/3517365926 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c004a60 con 0x55d7ebd7d140 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0981200640 1 -- 172.21.15.52:0/3517365926 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f097c04b030 con 0x55d7ebd7d140 2024-04-26T02:01:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.474+0000 7f0981200640 1 -- 172.21.15.52:0/3517365926 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c058430 con 0x55d7ebd7d140 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 -- 172.21.15.52:0/3517365926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 msgr2=0x55d7ebd73f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f097c003ca0 tx=0x7f097c003850 comp rx=0 tx=0).stop 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 -- 172.21.15.52:0/3517365926 shutdown_connections 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd79110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd73f60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 --2- 172.21.15.52:0/3517365926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd934f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.475+0000 7f099a26e740 1 -- 172.21.15.52:0/3517365926 >> 172.21.15.52:0/3517365926 conn(0x55d7ebd78d30 msgr2=0x55d7ebd84260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.476+0000 7f099a26e740 1 -- 172.21.15.52:0/3517365926 shutdown_connections 2024-04-26T02:01:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.476+0000 7f099a26e740 1 -- 172.21.15.52:0/3517365926 wait complete. 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.476+0000 7f099a26e740 1 -- start start 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.477+0000 7f099a26e740 1 Processor -- start 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.477+0000 7f099a26e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd8c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.477+0000 7f099a26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.477+0000 7f099a26e740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd8d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f099a26e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0990072950 con 0x55d7ebd7c820 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f099a26e740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0990073da0 con 0x55d7ebd87a00 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f099a26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0990073000 con 0x55d7ebd7d140 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f099a26e740 10 client.0 _ll_register_callbacks cb 0x7f097c030ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:43012/0 (socket says 172.21.15.52:43012) 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 -- 172.21.15.52:0/3802420686 learned_addr learned my addr 172.21.15.52:0/3802420686 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f0994a00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd8c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f0995400640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd8d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 -- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 msgr2=0x55d7ebd8d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd8d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 -- 172.21.15.52:0/3802420686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 msgr2=0x55d7ebd8c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd8c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f098fe00640 1 -- 172.21.15.52:0/3802420686 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ebdfc520 con 0x55d7ebd7d140 2024-04-26T02:01:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.478+0000 7f0995400640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd8d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f098fe00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f097c030ab0 tx=0x7f097c0303a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6350c42189b7844d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f0976000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c05d070 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8521+0+0 (secure 0 0 0) 0x7f097c004860 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f097c04b030 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f097c004a60 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f099a26e740 10 client.27082 fetch_fsmap learned FSMap version 62 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.479+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d7ebdfd760 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.480+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f097c004c00 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 62) ==== 8346+0+0 (secure 0 0 0) 0x7f097c044040 con 0x55d7ebd7d140 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 10 client.27082 fetch_fsmap finished waiting for FSMap version 62 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 10 client.27082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 20 client.27082 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebdfddd0 0x55d7ebe001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 4 client.27082 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f097c04b030 con 0x55d7ebdfddd0 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f0994a00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebdfddd0 0x55d7ebe001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f0994a00640 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebdfddd0 0x55d7ebe001f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be40f3ca3b89e861 server_cookie=7c111c7a1e86b26b in_seq=0 out_seq=0 2024-04-26T02:01:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.481+0000 7f0976000640 10 client.27082 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.626+0000 7f0976000640 1 -- 172.21.15.52:0/3802420686 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f097c04b030 con 0x55d7ebdfddd0 2024-04-26T02:01:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.626+0000 7f0976000640 10 client.27082 handle_command_reply: tid=0 2024-04-26T02:01:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.626+0000 7f099a26e740 1 client.27082 shutdown 2024-04-26T02:01:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 msgr2=0x55d7ebd8cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f097c030ab0 tx=0x7f097c0303a0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 shutdown_connections 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d7ebdfddd0 0x55d7ebe001f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ebd87a00 0x55d7ebd8d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ebd7d140 0x55d7ebd8cf10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 --2- 172.21.15.52:0/3802420686 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ebd7c820 0x55d7ebd8c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 >> 172.21.15.52:0/3802420686 conn(0x55d7ebd78d30 msgr2=0x55d7ebd84140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f0976000640 0 client.27082 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 shutdown_connections 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.627+0000 7f099a26e740 1 -- 172.21.15.52:0/3802420686 wait complete. 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.628+0000 7f099a26e740 20 client.27082 trim_cache size 0 max 16384 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:01:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:01:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.629+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 msgr2=0x7f09780a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.629+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 0x7f09780a63e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09901a02a0 tx=0x7f097c005c90 comp rx=0 tx=0).stop 2024-04-26T02:01:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.629+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 msgr2=0x7f099019eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.629+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f098800cce0 tx=0x7f098800a620 comp rx=0 tx=0).stop 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 shutdown_connections 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f09780a3f20 0x7f09780a63e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09901062f0 0x7f099019eb00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09901018f0 0x7f09901a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 --2- 172.21.15.52:0/3822610685 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0990100fd0 0x7f09901a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 >> 172.21.15.52:0/3822610685 conn(0x7f0990077fd0 msgr2=0x7f0990103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 shutdown_connections 2024-04-26T02:01:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:34.630+0000 7f0997400640 1 -- 172.21.15.52:0/3822610685 wait complete. 2024-04-26T02:01:34.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:34 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:34.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:34 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:34.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:34 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:34.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 62433 23.17 MB/sec execute 51 sec latency 162.335 ms 2024-04-26T02:01:35.364 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:01:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T02:01:35.364 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:01:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T02:01:35.364 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:01:34 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/1940634468 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e326869a40 con 0x55e32683c800 2024-04-26T02:01:35.364 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:01:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T02:01:35.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:35 smithi191 ceph-mon[32745]: pgmap v524: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:01:35.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:35 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T02:01:35.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:35 smithi183 ceph-mon[32752]: pgmap v524: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:01:35.679 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:35 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T02:01:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:35 smithi052 ceph-mon[27851]: pgmap v524: 97 pgs: 97 active+clean; 627 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:01:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:35 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T02:01:35.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 63108 23.51 MB/sec execute 52 sec latency 203.910 ms 2024-04-26T02:01:36.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:36 smithi191 ceph-mon[32745]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:36.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:36 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/2553596297,v1:172.21.15.183:6837/2553596297] up:boot 2024-04-26T02:01:36.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:36 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:01:36.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:36 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:01:36.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:36 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:36.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:36 smithi183 ceph-mon[32752]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:36.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:36 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/2553596297,v1:172.21.15.183:6837/2553596297] up:boot 2024-04-26T02:01:36.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:36 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:01:36.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:36 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:01:36.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:36 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:36.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:36 smithi052 ceph-mon[27851]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T02:01:36.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:36 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/2553596297,v1:172.21.15.183:6837/2553596297] up:boot 2024-04-26T02:01:36.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:36 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:01:36.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:36 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:01:36.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:36 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:01:36.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 63886 23.77 MB/sec execute 53 sec latency 62.085 ms 2024-04-26T02:01:37.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:37 smithi191 ceph-mon[32745]: pgmap v526: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:37.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:37 smithi183 ceph-mon[32752]: pgmap v526: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:37 smithi052 ceph-mon[27851]: pgmap v526: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:37.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 64622 24.10 MB/sec execute 54 sec latency 123.114 ms 2024-04-26T02:01:38.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 65311 24.38 MB/sec execute 55 sec latency 69.327 ms 2024-04-26T02:01:39.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:39 smithi191 ceph-mon[32745]: pgmap v527: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:39.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:39 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:39.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:39 smithi183 ceph-mon[32752]: pgmap v527: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:39.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:39 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:39 smithi052 ceph-mon[27851]: pgmap v527: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:39 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:01:39.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 65853 24.50 MB/sec execute 56 sec latency 68.606 ms 2024-04-26T02:01:40.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:40 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:40.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:40 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:40.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:40 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:01:40.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:40 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:40.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:40 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:40.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:40 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:01:40.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:40 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:01:40.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:40 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:01:40.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:40 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:01:40.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 66448 24.46 MB/sec execute 57 sec latency 55.780 ms 2024-04-26T02:01:41.652 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:41 smithi191 ceph-mon[32745]: pgmap v528: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:41.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:41 smithi183 ceph-mon[32752]: pgmap v528: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:41.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:41 smithi052 ceph-mon[27851]: pgmap v528: 97 pgs: 97 active+clean; 635 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 843 op/s 2024-04-26T02:01:41.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 67215 24.91 MB/sec execute 58 sec latency 48.014 ms 2024-04-26T02:01:42.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 67918 25.04 MB/sec execute 59 sec latency 52.365 ms 2024-04-26T02:01:43.654 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:43 smithi191 ceph-mon[32745]: pgmap v529: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:43.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:43 smithi183 ceph-mon[32752]: pgmap v529: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:43.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:43 smithi052 ceph-mon[27851]: pgmap v529: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:43.726 INFO:tasks.workunit.client.0.smithi052.stdout: 10 68690 25.28 MB/sec execute 60 sec latency 56.665 ms 2024-04-26T02:01:44.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 69505 25.64 MB/sec execute 61 sec latency 38.941 ms 2024-04-26T02:01:45.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:45 smithi183 ceph-mon[32752]: pgmap v530: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:45.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:45 smithi052 ceph-mon[27851]: pgmap v530: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:45.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 70129 25.86 MB/sec execute 62 sec latency 109.475 ms 2024-04-26T02:01:45.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:45 smithi191 ceph-mon[32745]: pgmap v530: 97 pgs: 97 active+clean; 664 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 589 op/s 2024-04-26T02:01:46.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 70714 25.93 MB/sec execute 63 sec latency 55.531 ms 2024-04-26T02:01:47.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:47 smithi183 ceph-mon[32752]: pgmap v531: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.7 MiB/s wr, 756 op/s 2024-04-26T02:01:47.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:47 smithi052 ceph-mon[27851]: pgmap v531: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.7 MiB/s wr, 756 op/s 2024-04-26T02:01:47.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 71297 26.01 MB/sec execute 64 sec latency 73.926 ms 2024-04-26T02:01:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 -- 172.21.15.52:0/1939993739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 msgr2=0x7f5a7010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 --2- 172.21.15.52:0/1939993739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a7010d440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009be0 tx=0x7f5a6802a6d0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 -- 172.21.15.52:0/1939993739 shutdown_connections 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 --2- 172.21.15.52:0/1939993739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a7010d440 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 --2- 172.21.15.52:0/1939993739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a701005c0 0x7f5a70100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 --2- 172.21.15.52:0/1939993739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a70106680 0x7f5a70106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 -- 172.21.15.52:0/1939993739 >> 172.21.15.52:0/1939993739 conn(0x7f5a700773e0 msgr2=0x7f5a700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 -- 172.21.15.52:0/1939993739 shutdown_connections 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.827+0000 7f5a77800640 1 -- 172.21.15.52:0/1939993739 wait complete. 2024-04-26T02:01:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.829+0000 7f5a77800640 1 Processor -- start 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.829+0000 7f5a77800640 1 -- start start 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.829+0000 7f5a77800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 0x7f5a70073640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.829+0000 7f5a77800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a70073b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a77800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 0x7f5a7006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a77800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701143c0 con 0x7f5a70100f80 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a77800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70114240 con 0x7f5a70106680 2024-04-26T02:01:47.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a77800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70114540 con 0x7f5a701005c0 2024-04-26T02:01:47.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a75800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 0x7f5a7006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a75800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 0x7f5a7006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59042/0 (socket says 172.21.15.52:59042) 2024-04-26T02:01:47.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a75800640 1 -- 172.21.15.52:0/1459881436 learned_addr learned my addr 172.21.15.52:0/1459881436 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:47.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 0x7f5a70073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.830+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a70073b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 -- 172.21.15.52:0/1459881436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 msgr2=0x7f5a7006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 0x7f5a7006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 -- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 msgr2=0x7f5a70073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a70073b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 -- 172.21.15.52:0/1459881436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7006d0a0 con 0x7f5a701005c0 2024-04-26T02:01:47.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 0x7f5a70073640 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400ea60 tx=0x7f5a6400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.831+0000 7f5a6d600640 1 -- 172.21.15.52:0/1459881436 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6401b070 con 0x7f5a701005c0 2024-04-26T02:01:47.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.832+0000 7f5a6d600640 1 -- 172.21.15.52:0/1459881436 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6400cbc0 con 0x7f5a701005c0 2024-04-26T02:01:47.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.832+0000 7f5a6d600640 1 -- 172.21.15.52:0/1459881436 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400b5c0 con 0x7f5a701005c0 2024-04-26T02:01:47.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.832+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7006d390 con 0x7f5a701005c0 2024-04-26T02:01:47.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.832+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a70071340 con 0x7f5a701005c0 2024-04-26T02:01:47.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.839+0000 7f5a7a5e1740 1 Processor -- start 2024-04-26T02:01:47.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.839+0000 7f5a7a5e1740 1 -- start start 2024-04-26T02:01:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.839+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a6d600640 1 -- 172.21.15.52:0/1459881436 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5a6401e070 con 0x7f5a701005c0 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a75800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a254c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a27df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bd78980900 con 0x55bd78a1a820 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bd78926ef0 con 0x55bd78a1b140 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bd7897c8c0 con 0x55bd78a25a00 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a75800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33880/0 (socket says 172.21.15.52:33880) 2024-04-26T02:01:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.840+0000 7f5a75800640 1 -- 172.21.15.52:0/1031717218 learned_addr learned my addr 172.21.15.52:0/1031717218 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a27df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a75800640 1 -- 172.21.15.52:0/1031717218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 msgr2=0x55bd78a27df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a75800640 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a27df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a75800640 1 -- 172.21.15.52:0/1031717218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 msgr2=0x55bd78a254c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a75800640 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a254c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a75800640 1 -- 172.21.15.52:0/1031717218 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd78693210 con 0x55bd78a1a820 2024-04-26T02:01:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.841+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a27df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:01:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.842+0000 7f5a75800640 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5a680309e0 tx=0x7f5a68030a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b274cca35a7d9601 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.843+0000 7f5a62000640 1 -- 172.21.15.52:0/1031717218 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68041030 con 0x55bd78a1a820 2024-04-26T02:01:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.843+0000 7f5a62000640 1 -- 172.21.15.52:0/1031717218 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a68030dc0 con 0x55bd78a1a820 2024-04-26T02:01:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.843+0000 7f5a62000640 1 -- 172.21.15.52:0/1031717218 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68036040 con 0x55bd78a1a820 2024-04-26T02:01:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.843+0000 7f5a6d600640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 0x7f5a580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.843+0000 7f5a6d600640 1 -- 172.21.15.52:0/1459881436 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7f5a640d0210 con 0x7f5a701005c0 2024-04-26T02:01:47.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 0x7f5a580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/1031717218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 msgr2=0x55bd78a1ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5a680309e0 tx=0x7f5a68030a10 comp rx=0 tx=0).stop 2024-04-26T02:01:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/1031717218 shutdown_connections 2024-04-26T02:01:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a27df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a254c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/1031717218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a1ac00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.844+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/1031717218 >> 172.21.15.52:0/1031717218 conn(0x55bd78a16d30 msgr2=0x55bd78a17110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.845+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/1031717218 shutdown_connections 2024-04-26T02:01:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.845+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/1031717218 wait complete. 2024-04-26T02:01:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.845+0000 7f5a74e00640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 0x7f5a580a6300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55bd78a1bdc0 tx=0x7f5a6400ef90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.845+0000 7f5a7a5e1740 1 -- start start 2024-04-26T02:01:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.846+0000 7f5a7a5e1740 1 Processor -- start 2024-04-26T02:01:47.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.846+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a22f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a25a00 0x55bd78a23440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bd78a285d0 con 0x55bd78a25a00 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55bd7893a310 con 0x55bd78a1b140 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a75800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55bd78a28750 con 0x55bd78a1a820 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a75800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53644/0 (socket says 172.21.15.52:53644) 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a75800640 1 -- 172.21.15.52:0/2931315423 learned_addr learned my addr 172.21.15.52:0/2931315423 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a7a5e1740 10 client.0 _ll_register_callbacks cb 0x55bd78908cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.847+0000 7f5a75800640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a25a00 0x55bd78a23440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 -- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 msgr2=0x55bd78a22f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a22f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 -- 172.21.15.52:0/2931315423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a25a00 msgr2=0x55bd78a23440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a25a00 0x55bd78a23440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 -- 172.21.15.52:0/2931315423 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd7875f620 con 0x55bd78a1a820 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a75800640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009370 tx=0x7f5a68009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e686a119075e1ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.848+0000 7f5a56a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6804d030 con 0x55bd78a1a820 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7f5a680305b0 con 0x55bd78a1a820 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a68030dc0 con 0x55bd78a1a820 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5a68036070 con 0x55bd78a1a820 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a7a5e1740 10 client.27097 fetch_fsmap learned FSMap version 64 2024-04-26T02:01:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55bd78a285d0 con 0x55bd78a1a820 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.849+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a68030dc0 con 0x55bd78a1a820 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f5a6803d790 con 0x55bd78a1a820 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 10 client.27097 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 10 client.27097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 20 client.27097 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a35e70 0x55bd78a382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 4 client.27097 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:01:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd789948b0 con 0x55bd78a35e70 2024-04-26T02:01:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.850+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a35e70 0x55bd78a382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.852+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a35e70 0x55bd78a382f0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c11097d34c80329 server_cookie=84beff4f1bdd8d8b in_seq=0 out_seq=0 2024-04-26T02:01:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.852+0000 7f5a56a00640 10 client.27097 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.853+0000 7f5a56a00640 1 -- 172.21.15.52:0/2931315423 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd789948b0 con 0x55bd78a35e70 2024-04-26T02:01:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.853+0000 7f5a56a00640 10 client.27097 handle_command_reply: tid=0 2024-04-26T02:01:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.853+0000 7f5a7a5e1740 1 client.27097 shutdown 2024-04-26T02:01:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 msgr2=0x55bd78a229c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009370 tx=0x7f5a68009500 comp rx=0 tx=0).stop 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 shutdown_connections 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a35e70 0x55bd78a382f0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a25a00 0x55bd78a23440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a22f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a56a00640 0 client.27097 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2931315423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a229c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.854+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 >> 172.21.15.52:0/2931315423 conn(0x55bd78a16d30 msgr2=0x55bd78a1cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.855+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 shutdown_connections 2024-04-26T02:01:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.855+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2931315423 wait complete. 2024-04-26T02:01:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.855+0000 7f5a7a5e1740 20 client.27097 trim_cache size 0 max 16384 2024-04-26T02:01:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 Processor -- start 2024-04-26T02:01:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 -- start start 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1b140 0x55bd78a21b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a25a00 0x55bd78a15d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b67f0 con 0x55bd78a1b140 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b4060 con 0x55bd78a25a00 2024-04-26T02:01:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b1eb0 con 0x55bd78a1a820 2024-04-26T02:01:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.882+0000 7f5a75800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.883+0000 7f5a75800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:53646/0 (socket says 172.21.15.52:53646) 2024-04-26T02:01:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.883+0000 7f5a75800640 1 -- 172.21.15.52:0/2972910110 learned_addr learned my addr 172.21.15.52:0/2972910110 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.883+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1b140 0x55bd78a21b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.883+0000 7f5a75800640 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a25a00 0x55bd78a15d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 -- 172.21.15.52:0/2972910110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a25a00 msgr2=0x55bd78a15d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a25a00 0x55bd78a15d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 -- 172.21.15.52:0/2972910110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1b140 msgr2=0x55bd78a21b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1b140 0x55bd78a21b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 -- 172.21.15.52:0/2972910110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd78a1af70 con 0x55bd78a1a820 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.884+0000 7f5a75800640 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55bd78a20b20 tx=0x7f5a68000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d72101b9136f97a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.885+0000 7f5a61600640 1 -- 172.21.15.52:0/2972910110 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6804d070 con 0x55bd78a1a820 2024-04-26T02:01:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.885+0000 7f5a61600640 1 -- 172.21.15.52:0/2972910110 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6804e040 con 0x55bd78a1a820 2024-04-26T02:01:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.885+0000 7f5a61600640 1 -- 172.21.15.52:0/2972910110 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68041050 con 0x55bd78a1a820 2024-04-26T02:01:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.885+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2972910110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 msgr2=0x55bd78a31490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55bd78a20b20 tx=0x7f5a68000f40 comp rx=0 tx=0).stop 2024-04-26T02:01:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2972910110 shutdown_connections 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a25a00 0x55bd78a15d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1b140 0x55bd78a21b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2972910110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a1a820 0x55bd78a31490 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2972910110 >> 172.21.15.52:0/2972910110 conn(0x55bd78a16d30 msgr2=0x55bd78a15600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.886+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2972910110 shutdown_connections 2024-04-26T02:01:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.887+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2972910110 wait complete. 2024-04-26T02:01:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.887+0000 7f5a7a5e1740 1 -- start start 2024-04-26T02:01:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.887+0000 7f5a7a5e1740 1 Processor -- start 2024-04-26T02:01:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.888+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a2af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.888+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a7a5e1740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a94400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a7a5e1740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b2520 con 0x55bd78a1a820 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a6fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a6fe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59066/0 (socket says 172.21.15.52:59066) 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a6fe00640 1 -- 172.21.15.52:0/2650390869 learned_addr learned my addr 172.21.15.52:0/2650390869 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a75800640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a2af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b3990 con 0x55bd78a1b140 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701b2bf0 con 0x55bd78a25a00 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a7a5e1740 10 client.0 _ll_register_callbacks cb 0x7f5a68003b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:01:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.889+0000 7f5a75800640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a94400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.890+0000 7f5a75800640 1 -- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 msgr2=0x55bd78a2b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.890+0000 7f5a75800640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.890+0000 7f5a75800640 1 -- 172.21.15.52:0/2650390869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 msgr2=0x55bd78a2af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.890+0000 7f5a75800640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a2af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.891+0000 7f5a75800640 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd78a94a70 con 0x55bd78a25a00 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.891+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.891+0000 7f5a75800640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a94400 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5a68003a90 tx=0x7f5a68004850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a723bfbb9b8bea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:01:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.891+0000 7f5a56a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:01:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.892+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6804d070 con 0x55bd78a25a00 2024-04-26T02:01:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.892+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7f5a680305b0 con 0x55bd78a25a00 2024-04-26T02:01:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.893+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a6804e040 con 0x55bd78a25a00 2024-04-26T02:01:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.893+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f5a68041040 con 0x55bd78a25a00 2024-04-26T02:01:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.893+0000 7f5a7a5e1740 10 client.27103 fetch_fsmap learned FSMap version 64 2024-04-26T02:01:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55bd78a9bf20 con 0x55bd78a25a00 2024-04-26T02:01:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6804e040 con 0x55bd78a25a00 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f5a6803d460 con 0x55bd78a25a00 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a7a5e1740 10 client.27103 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a7a5e1740 10 client.27103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a7a5e1740 20 client.27103 populate_metadata read hostname 'smithi052' 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.894+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a9c340 0x55bd78a9e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.895+0000 7f5a7a5e1740 4 client.27103 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:01:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.895+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd7898e320 con 0x55bd78a9c340 2024-04-26T02:01:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.895+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a9c340 0x55bd78a9e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:01:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.896+0000 7f5a6fe00640 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a9c340 0x55bd78a9e760 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a9cd716938e9b4 server_cookie=b1b916ba9624c4e6 in_seq=0 out_seq=0 2024-04-26T02:01:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:47.896+0000 7f5a56a00640 10 client.27103 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:47.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:47 smithi191 ceph-mon[32745]: pgmap v531: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.7 MiB/s wr, 756 op/s 2024-04-26T02:01:48.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.086+0000 7f5a56a00640 1 -- 172.21.15.52:0/2650390869 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd7898e320 con 0x55bd78a9c340 2024-04-26T02:01:48.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.086+0000 7f5a56a00640 10 client.27103 handle_command_reply: tid=0 2024-04-26T02:01:48.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.086+0000 7f5a7a5e1740 1 client.27103 shutdown 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 msgr2=0x55bd78a94400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a94400 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5a68003a90 tx=0x7f5a68004850 comp rx=0 tx=0).stop 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 shutdown_connections 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55bd78a9c340 0x55bd78a9e760 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a56a00640 0 client.27103 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55bd78a25a00 0x55bd78a94400 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55bd78a1b140 0x55bd78a2b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 --2- 172.21.15.52:0/2650390869 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bd78a1a820 0x55bd78a2af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.087+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 >> 172.21.15.52:0/2650390869 conn(0x55bd78a16d30 msgr2=0x55bd78a15e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.088+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 shutdown_connections 2024-04-26T02:01:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.088+0000 7f5a7a5e1740 1 -- 172.21.15.52:0/2650390869 wait complete. 2024-04-26T02:01:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.089+0000 7f5a7a5e1740 20 client.27103 trim_cache size 0 max 16384 2024-04-26T02:01:48.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:01:48.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:01:48.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:01:48.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:01:48.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.091+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 msgr2=0x7f5a580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:48.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.091+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 0x7f5a580a6300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55bd78a1bdc0 tx=0x7f5a6400ef90 comp rx=0 tx=0).stop 2024-04-26T02:01:48.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.091+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 msgr2=0x7f5a70073640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:01:48.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.091+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 0x7f5a70073640 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400ea60 tx=0x7f5a6400ef20 comp rx=0 tx=0).stop 2024-04-26T02:01:48.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 shutdown_connections 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5a580a3e40 0x7f5a580a6300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a70106680 0x7f5a7006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5a70100f80 0x7f5a70073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 --2- 172.21.15.52:0/1459881436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a701005c0 0x7f5a70073640 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 >> 172.21.15.52:0/1459881436 conn(0x7f5a700773e0 msgr2=0x7f5a700fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 shutdown_connections 2024-04-26T02:01:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:01:48.092+0000 7f5a77800640 1 -- 172.21.15.52:0/1459881436 wait complete. 2024-04-26T02:01:48.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 71594 25.81 MB/sec execute 65 sec latency 343.837 ms 2024-04-26T02:01:49.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:49 smithi183 ceph-mon[32752]: pgmap v532: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:49.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:49 smithi052 ceph-mon[27851]: pgmap v532: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:49.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 71884 25.59 MB/sec execute 66 sec latency 313.117 ms 2024-04-26T02:01:49.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:49 smithi191 ceph-mon[32745]: pgmap v532: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:50.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 72595 25.72 MB/sec execute 67 sec latency 81.013 ms 2024-04-26T02:01:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:51 smithi183 ceph-mon[32752]: pgmap v533: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:51 smithi052 ceph-mon[27851]: pgmap v533: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:51.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 73512 26.17 MB/sec execute 68 sec latency 43.803 ms 2024-04-26T02:01:51.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:51 smithi191 ceph-mon[32745]: pgmap v533: 97 pgs: 97 active+clean; 691 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 406 op/s 2024-04-26T02:01:52.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 74186 26.40 MB/sec execute 69 sec latency 69.273 ms 2024-04-26T02:01:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:53 smithi052 ceph-mon[27851]: pgmap v534: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-26T02:01:53.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 74775 26.51 MB/sec execute 70 sec latency 52.995 ms 2024-04-26T02:01:53.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:53 smithi191 ceph-mon[32745]: pgmap v534: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-26T02:01:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:53 smithi183 ceph-mon[32752]: pgmap v534: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-26T02:01:54.727 INFO:tasks.workunit.client.0.smithi052.stdout: 10 75070 26.29 MB/sec execute 71 sec latency 542.126 ms 2024-04-26T02:01:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:55 smithi052 ceph-mon[27851]: pgmap v535: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.5 MiB/s wr, 528 op/s 2024-04-26T02:01:55.728 INFO:tasks.workunit.client.0.smithi052.stdout: 10 75270 26.03 MB/sec execute 72 sec latency 1054.128 ms 2024-04-26T02:01:55.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:55 smithi191 ceph-mon[32745]: pgmap v535: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.5 MiB/s wr, 528 op/s 2024-04-26T02:01:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:55 smithi183 ceph-mon[32752]: pgmap v535: 97 pgs: 97 active+clean; 695 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 7.5 MiB/s wr, 528 op/s 2024-04-26T02:01:56.728 INFO:tasks.workunit.client.0.smithi052.stdout: 10 75643 25.91 MB/sec execute 73 sec latency 235.524 ms 2024-04-26T02:01:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:57 smithi052 ceph-mon[27851]: pgmap v536: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 929 op/s 2024-04-26T02:01:57.728 INFO:tasks.workunit.client.0.smithi052.stdout: 10 76437 26.14 MB/sec execute 74 sec latency 62.430 ms 2024-04-26T02:01:57.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:57 smithi191 ceph-mon[32745]: pgmap v536: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 929 op/s 2024-04-26T02:01:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:57 smithi183 ceph-mon[32752]: pgmap v536: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 929 op/s 2024-04-26T02:01:58.728 INFO:tasks.workunit.client.0.smithi052.stdout: 10 77205 26.37 MB/sec execute 75 sec latency 109.201 ms 2024-04-26T02:01:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:01:59 smithi052 ceph-mon[27851]: pgmap v537: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 731 op/s 2024-04-26T02:01:59.729 INFO:tasks.workunit.client.0.smithi052.stdout: 10 77787 26.47 MB/sec execute 76 sec latency 236.272 ms 2024-04-26T02:01:59.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:01:59 smithi191 ceph-mon[32745]: pgmap v537: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 731 op/s 2024-04-26T02:01:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:01:59 smithi183 ceph-mon[32752]: pgmap v537: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 731 op/s 2024-04-26T02:02:00.729 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78177 26.38 MB/sec execute 77 sec latency 474.939 ms 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 -- 172.21.15.52:0/1621651586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb81092d0 msgr2=0x7fceb8110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 --2- 172.21.15.52:0/1621651586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb81092d0 0x7fceb8110ea0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009ca0 tx=0x7fceb002a8c0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 -- 172.21.15.52:0/1621651586 shutdown_connections 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 --2- 172.21.15.52:0/1621651586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb81092d0 0x7fceb8110ea0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 --2- 172.21.15.52:0/1621651586 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb8104bb0 0x7fceb8108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 --2- 172.21.15.52:0/1621651586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb8104200 0x7fceb81045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 -- 172.21.15.52:0/1621651586 >> 172.21.15.52:0/1621651586 conn(0x7fceb80ffa60 msgr2=0x7fceb8101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 -- 172.21.15.52:0/1621651586 shutdown_connections 2024-04-26T02:02:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.254+0000 7fcebee00640 1 -- 172.21.15.52:0/1621651586 wait complete. 2024-04-26T02:02:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 Processor -- start 2024-04-26T02:02:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 -- start start 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 0x7fceb81a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 0x7fceb81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb819a5b0 0x7fceb819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8114290 con 0x7fceb81092d0 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8114110 con 0x7fceb8104200 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.255+0000 7fcebee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8114410 con 0x7fceb819a5b0 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebd000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 0x7fceb81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 0x7fceb81a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebd000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 0x7fceb81a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45118/0 (socket says 172.21.15.52:45118) 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebd000640 1 -- 172.21.15.52:0/2801923815 learned_addr learned my addr 172.21.15.52:0/2801923815 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebe400640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb819a5b0 0x7fceb819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 -- 172.21.15.52:0/2801923815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb819a5b0 msgr2=0x7fceb819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb819a5b0 0x7fceb819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 -- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 msgr2=0x7fceb81a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 0x7fceb81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 -- 172.21.15.52:0/2801923815 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb819b0d0 con 0x7fceb8104200 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.256+0000 7fcebda00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 0x7fceb81a0400 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fceac00e9f0 tx=0x7fceac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.257+0000 7fceb6000640 1 -- 172.21.15.52:0/2801923815 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac00ccc0 con 0x7fceb8104200 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.257+0000 7fceb6000640 1 -- 172.21.15.52:0/2801923815 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceac005020 con 0x7fceb8104200 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.257+0000 7fceb6000640 1 -- 172.21.15.52:0/2801923815 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac004a00 con 0x7fceb8104200 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.257+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb819b3c0 con 0x7fceb8104200 2024-04-26T02:02:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.257+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb81a7610 con 0x7fceb8104200 2024-04-26T02:02:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.260+0000 7fceb6000640 1 -- 172.21.15.52:0/2801923815 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fceac004ba0 con 0x7fceb8104200 2024-04-26T02:02:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.260+0000 7fcec3214740 1 Processor -- start 2024-04-26T02:02:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.260+0000 7fcec3214740 1 -- start start 2024-04-26T02:02:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0dad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0daff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5600f0d08900 con 0x5600f0da32d0 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5600f0caeef0 con 0x5600f0da29b0 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcec3214740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5600f0d048c0 con 0x5600f0dadb90 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebda00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0dad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33080/0 (socket says 172.21.15.52:33080) 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 -- 172.21.15.52:0/1499920640 learned_addr learned my addr 172.21.15.52:0/1499920640 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebd000640 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0daff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 -- 172.21.15.52:0/1499920640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 msgr2=0x5600f0daff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0daff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 -- 172.21.15.52:0/1499920640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 msgr2=0x5600f0dad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0dad650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.261+0000 7fcebe400640 1 -- 172.21.15.52:0/1499920640 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600f0a1b210 con 0x5600f0da29b0 2024-04-26T02:02:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fcebe400640 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009870 tx=0x7fceb00098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80392aee518081dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fceb6000640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 0x7fcea00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fceaaa00640 1 -- 172.21.15.52:0/1499920640 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0031cf0 con 0x5600f0da29b0 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fceaaa00640 1 -- 172.21.15.52:0/1499920640 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb0036040 con 0x5600f0da29b0 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fceaaa00640 1 -- 172.21.15.52:0/1499920640 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0004280 con 0x5600f0da29b0 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fcebda00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 0x7fcea00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.262+0000 7fceb6000640 1 -- 172.21.15.52:0/2801923815 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7fceac0d07d0 con 0x7fceb8104200 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 -- 172.21.15.52:0/1499920640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 msgr2=0x5600f0da2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009870 tx=0x7fceb00098a0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 -- 172.21.15.52:0/1499920640 shutdown_connections 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0daff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0dad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 --2- 172.21.15.52:0/1499920640 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da29b0 0x5600f0da2d90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 -- 172.21.15.52:0/1499920640 >> 172.21.15.52:0/1499920640 conn(0x5600f0d9eec0 msgr2=0x5600f0d9f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 -- 172.21.15.52:0/1499920640 shutdown_connections 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcec3214740 1 -- 172.21.15.52:0/1499920640 wait complete. 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.263+0000 7fcebda00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 0x7fcea00a6250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x5600f0da3f50 tx=0x7fceac0d6530 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.264+0000 7fcec3214740 1 -- start start 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.264+0000 7fcec3214740 1 Processor -- start 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0daab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5600f0db0760 con 0x5600f0da29b0 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5600f0cc2310 con 0x5600f0da32d0 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5600f0db08e0 con 0x5600f0dadb90 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcec3214740 10 client.0 _ll_register_callbacks cb 0x5600f0c90cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33090/0 (socket says 172.21.15.52:33090) 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 -- 172.21.15.52:0/1403879956 learned_addr learned my addr 172.21.15.52:0/1403879956 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebe400640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0daab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 -- 172.21.15.52:0/1403879956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 msgr2=0x5600f0dab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 -- 172.21.15.52:0/1403879956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 msgr2=0x5600f0daab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0daab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebd000640 1 -- 172.21.15.52:0/1403879956 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600f0ae7620 con 0x5600f0da32d0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.265+0000 7fcebe400640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0daab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fcebd000640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcea4012740 tx=0x7fcea4012c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4877b1ba328eebf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fce9f400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea400f8c0 con 0x5600f0da32d0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7fcea400fa60 con 0x5600f0da32d0 2024-04-26T02:02:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb0036040 con 0x5600f0da32d0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fcea400fc60 con 0x5600f0da32d0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fcec3214740 10 client.27110 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.266+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5600f0dbca70 con 0x5600f0da32d0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceb0036040 con 0x5600f0da32d0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7fcea40049f0 con 0x5600f0da32d0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 10 client.27110 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 10 client.27110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 20 client.27110 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0dbdfa0 0x5600f0dc0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 4 client.27110 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600f0d1c8b0 con 0x5600f0dbdfa0 2024-04-26T02:02:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.267+0000 7fcebe400640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0dbdfa0 0x5600f0dc0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.268+0000 7fcebe400640 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0dbdfa0 0x5600f0dc0420 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d441da950deaf2b server_cookie=39a39d2562b65bc3 in_seq=0 out_seq=0 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.268+0000 7fce9f400640 10 client.27110 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.268+0000 7fce9f400640 1 -- 172.21.15.52:0/1403879956 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600f0d1c8b0 con 0x5600f0dbdfa0 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.268+0000 7fce9f400640 10 client.27110 handle_command_reply: tid=0 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.268+0000 7fcec3214740 1 client.27110 shutdown 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 msgr2=0x5600f0dab0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcea4012740 tx=0x7fcea4012c00 comp rx=0 tx=0).stop 2024-04-26T02:02:01.271 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 shutdown_connections 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0dbdfa0 0x5600f0dc0420 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab0b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 --2- 172.21.15.52:0/1403879956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0daab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fce9f400640 0 client.27110 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 >> 172.21.15.52:0/1403879956 conn(0x5600f0d9eec0 msgr2=0x5600f0da4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 shutdown_connections 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 1 -- 172.21.15.52:0/1403879956 wait complete. 2024-04-26T02:02:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.269+0000 7fcec3214740 20 client.27110 trim_cache size 0 max 16384 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 Processor -- start 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 -- start start 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0db9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dc5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81b2580 con 0x5600f0da29b0 2024-04-26T02:02:01.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81afdf0 con 0x5600f0da32d0 2024-04-26T02:02:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.286+0000 7fcec3214740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81adc40 con 0x5600f0dadb90 2024-04-26T02:02:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebe400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0db9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebe400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0db9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45148/0 (socket says 172.21.15.52:45148) 2024-04-26T02:02:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebe400640 1 -- 172.21.15.52:0/4124190483 learned_addr learned my addr 172.21.15.52:0/4124190483 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebe400640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dc5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 -- 172.21.15.52:0/4124190483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 msgr2=0x5600f0dc5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dc5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 -- 172.21.15.52:0/4124190483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 msgr2=0x5600f0db9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0db9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 -- 172.21.15.52:0/4124190483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600f0d9a820 con 0x5600f0da32d0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebe400640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dc5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.287+0000 7fcebd000640 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcea4004e40 tx=0x7fcea400fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc53e6de9f684bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fceaa000640 1 -- 172.21.15.52:0/4124190483 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea401c600 con 0x5600f0da32d0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fceaa000640 1 -- 172.21.15.52:0/4124190483 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea4012e50 con 0x5600f0da32d0 2024-04-26T02:02:01.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fceaa000640 1 -- 172.21.15.52:0/4124190483 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4005210 con 0x5600f0da32d0 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 -- 172.21.15.52:0/4124190483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 msgr2=0x5600f0dab410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcea4004e40 tx=0x7fcea400fe80 comp rx=0 tx=0).stop 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 -- 172.21.15.52:0/4124190483 shutdown_connections 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0dadb90 0x5600f0dc5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0da32d0 0x5600f0dab410 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 --2- 172.21.15.52:0/4124190483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da29b0 0x5600f0db9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 -- 172.21.15.52:0/4124190483 >> 172.21.15.52:0/4124190483 conn(0x5600f0d9eec0 msgr2=0x5600f0da4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 -- 172.21.15.52:0/4124190483 shutdown_connections 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.288+0000 7fcec3214740 1 -- 172.21.15.52:0/4124190483 wait complete. 2024-04-26T02:02:01.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.289+0000 7fcec3214740 1 -- start start 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.289+0000 7fcec3214740 1 Processor -- start 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.289+0000 7fcec3214740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 0x5600f0db5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.289+0000 7fcec3214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.289+0000 7fcec3214740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0dadb90 0x5600f0db3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcec3214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81ae2b0 con 0x5600f0da32d0 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcec3214740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81af720 con 0x5600f0dadb90 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcec3214740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fceb81ae980 con 0x5600f0da29b0 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcec3214740 10 client.0 _ll_register_callbacks cb 0x7fcea4009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45158/0 (socket says 172.21.15.52:45158) 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 -- 172.21.15.52:0/3604515693 learned_addr learned my addr 172.21.15.52:0/3604515693 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebe400640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0dadb90 0x5600f0db3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebe400640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 0x5600f0db5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 -- 172.21.15.52:0/3604515693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 msgr2=0x5600f0db5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 0x5600f0db5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 -- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0dadb90 msgr2=0x5600f0db3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0dadb90 0x5600f0db3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebd000640 1 -- 172.21.15.52:0/3604515693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600f0db3b70 con 0x5600f0da32d0 2024-04-26T02:02:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.290+0000 7fcebe400640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 0x5600f0db5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fcebd000640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009600 tx=0x7fcea400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc0b125345acc1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fce9f400640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea400f850 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8477+0+0 (secure 0 0 0) 0x7fcea400f9f0 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcea4012e50 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fcea401c800 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fcec3214740 10 client.16968 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.291+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5600f0e242b0 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea4012e50 con 0x5600f0da32d0 2024-04-26T02:02:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7fcea402d070 con 0x5600f0da32d0 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 10 client.16968 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 10 client.16968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 20 client.16968 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0e24a80 0x5600f0e26ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 4 client.16968 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600f0d16320 con 0x5600f0e24a80 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.292+0000 7fcebe400640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0e24a80 0x5600f0e26ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.293+0000 7fcebe400640 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0e24a80 0x5600f0e26ea0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2855784765771d2 server_cookie=bb58ff3a325783c6 in_seq=0 out_seq=0 2024-04-26T02:02:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.293+0000 7fce9f400640 10 client.16968 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fce9f400640 1 -- 172.21.15.52:0/3604515693 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600f0d16320 con 0x5600f0e24a80 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fce9f400640 10 client.16968 handle_command_reply: tid=0 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 client.16968 shutdown 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 msgr2=0x5600f0db30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009600 tx=0x7fcea400ff70 comp rx=0 tx=0).stop 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 shutdown_connections 2024-04-26T02:02:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5600f0e24a80 0x5600f0e26ea0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5600f0dadb90 0x5600f0db3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5600f0da32d0 0x5600f0db30c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 --2- 172.21.15.52:0/3604515693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5600f0da29b0 0x5600f0db5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fce9f400640 0 client.16968 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.442+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 >> 172.21.15.52:0/3604515693 conn(0x5600f0d9eec0 msgr2=0x5600f0da4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.444+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 shutdown_connections 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.444+0000 7fcec3214740 1 -- 172.21.15.52:0/3604515693 wait complete. 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.444+0000 7fcec3214740 20 client.16968 trim_cache size 0 max 16384 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:02:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:02:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:02:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 msgr2=0x7fcea00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 0x7fcea00a6250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x5600f0da3f50 tx=0x7fceac0d6530 comp rx=0 tx=0).stop 2024-04-26T02:02:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 msgr2=0x7fceb81a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 0x7fceb81a0400 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fceac00e9f0 tx=0x7fceac00eeb0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 shutdown_connections 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcea00a3d90 0x7fcea00a6250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fceb819a5b0 0x7fceb819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fceb81092d0 0x7fceb81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 --2- 172.21.15.52:0/2801923815 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb8104200 0x7fceb81a0400 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 >> 172.21.15.52:0/2801923815 conn(0x7fceb80ffa60 msgr2=0x7fceb8101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 shutdown_connections 2024-04-26T02:02:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:01.445+0000 7fcebee00640 1 -- 172.21.15.52:0/2801923815 wait complete. 2024-04-26T02:02:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:01 smithi052 ceph-mon[27851]: pgmap v538: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 732 op/s 2024-04-26T02:02:01.729 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78177 26.05 MB/sec execute 78 sec latency 1475.089 ms 2024-04-26T02:02:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:01 smithi191 ceph-mon[32745]: pgmap v538: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 732 op/s 2024-04-26T02:02:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:01 smithi183 ceph-mon[32752]: pgmap v538: 97 pgs: 97 active+clean; 757 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 16 MiB/s wr, 732 op/s 2024-04-26T02:02:02.729 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78177 25.72 MB/sec execute 79 sec latency 2475.240 ms 2024-04-26T02:02:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:03 smithi052 ceph-mon[27851]: pgmap v539: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 969 op/s 2024-04-26T02:02:03.729 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78280 25.43 MB/sec execute 80 sec latency 2876.655 ms 2024-04-26T02:02:03.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:03 smithi191 ceph-mon[32745]: pgmap v539: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 969 op/s 2024-04-26T02:02:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:03 smithi183 ceph-mon[32752]: pgmap v539: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 969 op/s 2024-04-26T02:02:04.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78280 25.12 MB/sec execute 81 sec latency 1424.761 ms 2024-04-26T02:02:05.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78280 24.82 MB/sec execute 82 sec latency 2424.917 ms 2024-04-26T02:02:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:05 smithi191 ceph-mon[32745]: pgmap v540: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 638 op/s 2024-04-26T02:02:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:05 smithi183 ceph-mon[32752]: pgmap v540: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 638 op/s 2024-04-26T02:02:05.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:05 smithi052 ceph-mon[27851]: pgmap v540: 97 pgs: 97 active+clean; 725 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 638 op/s 2024-04-26T02:02:06.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78280 24.52 MB/sec execute 83 sec latency 3425.078 ms 2024-04-26T02:02:07.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78280 24.22 MB/sec execute 84 sec latency 4425.230 ms 2024-04-26T02:02:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:07 smithi191 ceph-mon[32745]: pgmap v541: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 14 MiB/s wr, 772 op/s 2024-04-26T02:02:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:07 smithi183 ceph-mon[32752]: pgmap v541: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 14 MiB/s wr, 772 op/s 2024-04-26T02:02:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:07 smithi052 ceph-mon[27851]: pgmap v541: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 14 MiB/s wr, 772 op/s 2024-04-26T02:02:08.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78532 24.12 MB/sec execute 85 sec latency 4832.784 ms 2024-04-26T02:02:09.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 78802 24.00 MB/sec execute 86 sec latency 143.290 ms 2024-04-26T02:02:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:09 smithi191 ceph-mon[32745]: pgmap v542: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:09 smithi183 ceph-mon[32752]: pgmap v542: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:09 smithi052 ceph-mon[27851]: pgmap v542: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:10.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 79464 24.14 MB/sec execute 87 sec latency 76.121 ms 2024-04-26T02:02:11.730 INFO:tasks.workunit.client.0.smithi052.stdout: 10 80061 24.26 MB/sec execute 88 sec latency 85.918 ms 2024-04-26T02:02:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:11 smithi191 ceph-mon[32745]: pgmap v543: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:11 smithi183 ceph-mon[32752]: pgmap v543: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:11.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:11 smithi052 ceph-mon[27851]: pgmap v543: 97 pgs: 97 active+clean; 714 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 371 op/s 2024-04-26T02:02:12.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 80385 24.13 MB/sec execute 89 sec latency 129.857 ms 2024-04-26T02:02:12.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:12 smithi191 ceph-mon[32745]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:02:12.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:12 smithi183 ceph-mon[32752]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:02:12.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:12 smithi052 ceph-mon[27851]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T02:02:13.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 80996 24.30 MB/sec execute 90 sec latency 112.503 ms 2024-04-26T02:02:13.825 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:13 smithi183 ceph-mon[32752]: pgmap v545: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:13 smithi191 ceph-mon[32745]: pgmap v545: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:13 smithi052 ceph-mon[27851]: pgmap v545: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.642+0000 7f1b8e800640 1 -- 172.21.15.52:0/1360884458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88102f10 msgr2=0x7f1b881032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.642+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1360884458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88102f10 0x7f1b881032f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009900 tx=0x7f1b7c02a480 comp rx=0 tx=0).stop 2024-04-26T02:02:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.643+0000 7f1b8e800640 1 -- 172.21.15.52:0/1360884458 shutdown_connections 2024-04-26T02:02:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.643+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1360884458 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b8810c330 0x7f1b8810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.643+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1360884458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b88103830 0x7f1b8810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.643+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1360884458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88102f10 0x7f1b881032f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.643+0000 7f1b8e800640 1 -- 172.21.15.52:0/1360884458 >> 172.21.15.52:0/1360884458 conn(0x7f1b880fc770 msgr2=0x7f1b880feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.644+0000 7f1b8e800640 1 -- 172.21.15.52:0/1360884458 shutdown_connections 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.644+0000 7f1b8e800640 1 -- 172.21.15.52:0/1360884458 wait complete. 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.645+0000 7f1b8e800640 1 Processor -- start 2024-04-26T02:02:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.645+0000 7f1b8e800640 1 -- start start 2024-04-26T02:02:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88102f10 0x7f1b8810ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b8810c330 0x7f1b88108ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881022d0 con 0x7f1b88102f10 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b88102150 con 0x7f1b8810c330 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8e800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b88102450 con 0x7f1b88103830 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.646+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:38146/0 (socket says 172.21.15.52:38146) 2024-04-26T02:02:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1816969859 learned_addr learned my addr 172.21.15.52:0/1816969859 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1816969859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b8810c330 msgr2=0x7f1b88108ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b8810c330 0x7f1b88108ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88102f10 msgr2=0x7f1b8810ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88102f10 0x7f1b8810ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.647+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1816969859 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b88109770 con 0x7f1b88103830 2024-04-26T02:02:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.648+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400e9f0 tx=0x7f1b7400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.648+0000 7f1b86000640 1 -- 172.21.15.52:0/1816969859 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400ccc0 con 0x7f1b88103830 2024-04-26T02:02:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.649+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b881ab140 con 0x7f1b88103830 2024-04-26T02:02:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.649+0000 7f1b86000640 1 -- 172.21.15.52:0/1816969859 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b740108e0 con 0x7f1b88103830 2024-04-26T02:02:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.649+0000 7f1b86000640 1 -- 172.21.15.52:0/1816969859 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74018880 con 0x7f1b88103830 2024-04-26T02:02:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.649+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b881ab650 con 0x7f1b88103830 2024-04-26T02:02:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.653+0000 7f1b86000640 1 -- 172.21.15.52:0/1816969859 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1b74010430 con 0x7f1b88103830 2024-04-26T02:02:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.655+0000 7f1b86000640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 0x7f1b700a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.655+0000 7f1b92bf9740 1 Processor -- start 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.655+0000 7f1b86000640 1 -- 172.21.15.52:0/1816969859 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8565+0+0 (secure 0 0 0) 0x7f1b740d0c10 con 0x7f1b88103830 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.655+0000 7f1b92bf9740 1 -- start start 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d64fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d65cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d64d5b5900 con 0x55d64d64f9b0 2024-04-26T02:02:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d64d55bef0 con 0x55d64d65ab90 2024-04-26T02:02:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d64d5b18c0 con 0x55d64d6502d0 2024-04-26T02:02:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.656+0000 7f1b8de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d65cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:38152/0 (socket says 172.21.15.52:38152) 2024-04-26T02:02:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 -- 172.21.15.52:0/3259662875 learned_addr learned my addr 172.21.15.52:0/3259662875 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 0x7f1b700a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 -- 172.21.15.52:0/3259662875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 msgr2=0x55d64d65cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d65cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.657+0000 7f1b8ca00640 1 -- 172.21.15.52:0/3259662875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 msgr2=0x55d64d64fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.658+0000 7f1b8d400640 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d64fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.658+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d64fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.658+0000 7f1b8ca00640 1 -- 172.21.15.52:0/3259662875 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d64d2c8210 con 0x55d64d6502d0 2024-04-26T02:02:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.658+0000 7f1b8d400640 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d64fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.659+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1b740a5260 tx=0x7f1b7409d220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=853b92f47f6dec62 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.659+0000 7f1b7a600640 1 -- 172.21.15.52:0/3259662875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740a0ed0 con 0x55d64d6502d0 2024-04-26T02:02:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.659+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 0x7f1b700a62d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0096c0 tx=0x7f1b7c000bc0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.659+0000 7f1b7a600640 1 -- 172.21.15.52:0/3259662875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b7409de50 con 0x55d64d6502d0 2024-04-26T02:02:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.659+0000 7f1b7a600640 1 -- 172.21.15.52:0/3259662875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740ce550 con 0x55d64d6502d0 2024-04-26T02:02:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.660+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3259662875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 msgr2=0x55d64d65a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.660+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1b740a5260 tx=0x7f1b7409d220 comp rx=0 tx=0).stop 2024-04-26T02:02:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3259662875 shutdown_connections 2024-04-26T02:02:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d65cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d65a650 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3259662875 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d64fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3259662875 >> 172.21.15.52:0/3259662875 conn(0x55d64d64bec0 msgr2=0x55d64d64c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3259662875 shutdown_connections 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.661+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3259662875 wait complete. 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.662+0000 7f1b92bf9740 1 -- start start 2024-04-26T02:02:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.662+0000 7f1b92bf9740 1 Processor -- start 2024-04-26T02:02:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 0x55d64d657ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 0x55d64d6580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d658620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d64d65d760 con 0x55d64d6502d0 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d64d56f310 con 0x55d64d65ab90 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d64d65d8e0 con 0x55d64d64f9b0 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b92bf9740 10 client.0 _ll_register_callbacks cb 0x55d64d53dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.663+0000 7f1b8d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 0x55d64d657ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d658620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 0x55d64d657ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:38162/0 (socket says 172.21.15.52:38162) 2024-04-26T02:02:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8d400640 1 -- 172.21.15.52:0/3007234481 learned_addr learned my addr 172.21.15.52:0/3007234481 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 0x55d64d6580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 -- 172.21.15.52:0/3007234481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 msgr2=0x55d64d657ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 0x55d64d657ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 -- 172.21.15.52:0/3007234481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 msgr2=0x55d64d6580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 0x55d64d6580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 -- 172.21.15.52:0/3007234481 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d64d394620 con 0x55d64d65ab90 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 0x55d64d6580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b8de00640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d658620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x55d64d64d050 tx=0x7f1b8000fe40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4c1ec20dbe69fad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.664+0000 7f1b6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000dd00 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8565+0+0 (secure 0 0 0) 0x7f1b80018430 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b7409de50 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1b8000dea0 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 10 client.27125 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d64d669a70 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b7409de50 con 0x55d64d65ab90 2024-04-26T02:02:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f1b80010760 con 0x55d64d65ab90 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 10 client.27125 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 10 client.27125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.666+0000 7f1b92bf9740 20 client.27125 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.667+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d66d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.667+0000 7f1b92bf9740 4 client.27125 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.667+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d64d5c98b0 con 0x55d64d66af80 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.667+0000 7f1b8d400640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d66d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.668+0000 7f1b8d400640 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d66d3b0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d598c3efea6fdef6 server_cookie=cd2b6f9e08af809d in_seq=0 out_seq=0 2024-04-26T02:02:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.668+0000 7f1b6ea00640 10 client.27125 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b6ea00640 1 -- 172.21.15.52:0/3007234481 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d64d5c98b0 con 0x55d64d66af80 2024-04-26T02:02:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b6ea00640 10 client.27125 handle_command_reply: tid=0 2024-04-26T02:02:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 client.27125 shutdown 2024-04-26T02:02:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 msgr2=0x55d64d658620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d658620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x55d64d64d050 tx=0x7f1b8000fe40 comp rx=0 tx=0).stop 2024-04-26T02:02:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 shutdown_connections 2024-04-26T02:02:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d66d3b0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d658620 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d6502d0 0x55d64d6580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.671+0000 7f1b6ea00640 0 client.27125 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.670+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/3007234481 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d64f9b0 0x55d64d657ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.671+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 >> 172.21.15.52:0/3007234481 conn(0x55d64d64bec0 msgr2=0x55d64d651c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.671+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 shutdown_connections 2024-04-26T02:02:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.671+0000 7f1b92bf9740 1 -- 172.21.15.52:0/3007234481 wait complete. 2024-04-26T02:02:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.672+0000 7f1b92bf9740 20 client.27125 trim_cache size 0 max 16384 2024-04-26T02:02:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.696+0000 7f1b92bf9740 1 Processor -- start 2024-04-26T02:02:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.696+0000 7f1b92bf9740 1 -- start start 2024-04-26T02:02:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.696+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d666620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.696+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d652370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.696+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d659ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b6a40 con 0x55d64d64f9b0 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b42b0 con 0x55d64d65ab90 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b2120 con 0x55d64d6502d0 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d666620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d659ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d652370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8d400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d666620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33242/0 (socket says 172.21.15.52:33242) 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8de00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d659ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:42976/0 (socket says 172.21.15.52:42976) 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8d400640 1 -- 172.21.15.52:0/807379352 learned_addr learned my addr 172.21.15.52:0/807379352 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 -- 172.21.15.52:0/807379352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 msgr2=0x55d64d659ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d659ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 -- 172.21.15.52:0/807379352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 msgr2=0x55d64d666620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d666620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.697+0000 7f1b8ca00640 1 -- 172.21.15.52:0/807379352 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d64d64b9f0 con 0x55d64d6502d0 2024-04-26T02:02:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.698+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d652370 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1b7409b070 tx=0x7f1b74009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae2c6ad31398a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.698+0000 7f1b79c00640 1 -- 172.21.15.52:0/807379352 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740b5070 con 0x55d64d6502d0 2024-04-26T02:02:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.698+0000 7f1b79c00640 1 -- 172.21.15.52:0/807379352 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b740a0ed0 con 0x55d64d6502d0 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.698+0000 7f1b79c00640 1 -- 172.21.15.52:0/807379352 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740ce550 con 0x55d64d6502d0 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 -- 172.21.15.52:0/807379352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 msgr2=0x55d64d652370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d652370 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1b7409b070 tx=0x7f1b74009f90 comp rx=0 tx=0).stop 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 -- 172.21.15.52:0/807379352 shutdown_connections 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d659ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d652370 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/807379352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d666620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 -- 172.21.15.52:0/807379352 >> 172.21.15.52:0/807379352 conn(0x55d64d64bec0 msgr2=0x55d64d6593d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 -- 172.21.15.52:0/807379352 shutdown_connections 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.699+0000 7f1b92bf9740 1 -- 172.21.15.52:0/807379352 wait complete. 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.700+0000 7f1b92bf9740 1 -- start start 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.700+0000 7f1b92bf9740 1 Processor -- start 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d660460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d660ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b2770 con 0x55d64d64f9b0 2024-04-26T02:02:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b3be0 con 0x55d64d65ab90 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881b2e40 con 0x55d64d6502d0 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b92bf9740 10 client.0 _ll_register_callbacks cb 0x7f1b8000f5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b8ca00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:38192/0 (socket says 172.21.15.52:38192) 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1863898158 learned_addr learned my addr 172.21.15.52:0/1863898158 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d660460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.701+0000 7f1b8de00640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d660ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 msgr2=0x55d64d660ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d660ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1863898158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 msgr2=0x55d64d660460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d660460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 -- 172.21.15.52:0/1863898158 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d64d6d0540 con 0x55d64d6502d0 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8de00640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d660ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d660460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b8ca00640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1b7409cab0 tx=0x7f1b7409cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188df4a23d383950 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740b5070 con 0x55d64d6502d0 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.702+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8565+0+0 (secure 0 0 0) 0x7f1b740ce550 con 0x55d64d6502d0 2024-04-26T02:02:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.703+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b740a0ed0 con 0x55d64d6502d0 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.703+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1b7409de50 con 0x55d64d6502d0 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.703+0000 7f1b92bf9740 10 client.27133 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.703+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55d64d6d1730 con 0x55d64d6502d0 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b740a0ed0 con 0x55d64d6502d0 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f1b740bbce0 con 0x55d64d6502d0 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 10 client.27133 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 10 client.27133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 20 client.27133 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d6d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 4 client.27133 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d64d5c3320 con 0x55d64d66af80 2024-04-26T02:02:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.704+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d6d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.705+0000 7f1b8d400640 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d6d39b0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=759fe1b750b9e671 server_cookie=a15f220953ed2451 in_seq=0 out_seq=0 2024-04-26T02:02:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.705+0000 7f1b6ea00640 10 client.27133 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:14.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 81694 24.39 MB/sec execute 91 sec latency 859.077 ms 2024-04-26T02:02:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.844+0000 7f1b6ea00640 1 -- 172.21.15.52:0/1863898158 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d64d5c3320 con 0x55d64d66af80 2024-04-26T02:02:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.844+0000 7f1b6ea00640 10 client.27133 handle_command_reply: tid=0 2024-04-26T02:02:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.844+0000 7f1b92bf9740 1 client.27133 shutdown 2024-04-26T02:02:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 msgr2=0x55d64d6609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1b7409cab0 tx=0x7f1b7409cb20 comp rx=0 tx=0).stop 2024-04-26T02:02:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 shutdown_connections 2024-04-26T02:02:14.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55d64d66af80 0x55d64d6d39b0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d64d65ab90 0x55d64d660ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d64d6502d0 0x55d64d6609a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 --2- 172.21.15.52:0/1863898158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d64d64f9b0 0x55d64d660460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 >> 172.21.15.52:0/1863898158 conn(0x55d64d64bec0 msgr2=0x55d64d659c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.845+0000 7f1b6ea00640 0 client.27133 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.846+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 shutdown_connections 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.846+0000 7f1b92bf9740 1 -- 172.21.15.52:0/1863898158 wait complete. 2024-04-26T02:02:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.846+0000 7f1b92bf9740 20 client.27133 trim_cache size 0 max 16384 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 msgr2=0x7f1b700a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 0x7f1b700a62d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0096c0 tx=0x7f1b7c000bc0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 msgr2=0x7f1b88108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400e9f0 tx=0x7f1b7400eeb0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 shutdown_connections 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1b700a3e10 0x7f1b700a62d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b8810c330 0x7f1b88108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b88103830 0x7f1b88108ab0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 --2- 172.21.15.52:0/1816969859 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b88102f10 0x7f1b8810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.848+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 >> 172.21.15.52:0/1816969859 conn(0x7f1b880fc770 msgr2=0x7f1b8810dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.850+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 shutdown_connections 2024-04-26T02:02:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:14.850+0000 7f1b8e800640 1 -- 172.21.15.52:0/1816969859 wait complete. 2024-04-26T02:02:15.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 82379 24.58 MB/sec execute 92 sec latency 1859.127 ms 2024-04-26T02:02:15.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:15 smithi191 ceph-mon[32745]: pgmap v546: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:15 smithi183 ceph-mon[32752]: pgmap v546: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:15 smithi052 ceph-mon[27851]: pgmap v546: 97 pgs: 97 active+clean; 730 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 493 op/s 2024-04-26T02:02:16.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 82910 24.65 MB/sec execute 93 sec latency 2859.174 ms 2024-04-26T02:02:17.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 83460 24.65 MB/sec execute 94 sec latency 3859.225 ms 2024-04-26T02:02:17.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:17 smithi052 ceph-mon[27851]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:17.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:17 smithi191 ceph-mon[32745]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:17 smithi183 ceph-mon[32752]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:18.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 84128 24.75 MB/sec execute 95 sec latency 4859.280 ms 2024-04-26T02:02:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:19 smithi183 ceph-mon[32752]: pgmap v548: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:19.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:19 smithi183 ceph-mon[32752]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:19 smithi052 ceph-mon[27851]: pgmap v548: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:19.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:19 smithi052 ceph-mon[27851]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:19.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 84618 24.80 MB/sec execute 96 sec latency 5859.337 ms 2024-04-26T02:02:19.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:19 smithi191 ceph-mon[32745]: pgmap v548: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-04-26T02:02:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:19 smithi191 ceph-mon[32745]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T02:02:20.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 85010 24.80 MB/sec execute 97 sec latency 6859.445 ms 2024-04-26T02:02:21.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:21 smithi183 ceph-mon[32752]: pgmap v550: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.5 MiB/s wr, 316 op/s 2024-04-26T02:02:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:21 smithi052 ceph-mon[27851]: pgmap v550: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.5 MiB/s wr, 316 op/s 2024-04-26T02:02:21.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 85268 24.63 MB/sec execute 98 sec latency 7859.511 ms 2024-04-26T02:02:21.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:21 smithi191 ceph-mon[32745]: pgmap v550: 97 pgs: 97 active+clean; 746 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.5 MiB/s wr, 316 op/s 2024-04-26T02:02:22.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 85652 24.61 MB/sec execute 99 sec latency 8859.575 ms 2024-04-26T02:02:23.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:23 smithi183 ceph-mon[32752]: pgmap v551: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:23.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:23 smithi052 ceph-mon[27851]: pgmap v551: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:23.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 86034 24.53 MB/sec execute 100 sec latency 9454.608 ms 2024-04-26T02:02:23.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:23 smithi191 ceph-mon[32745]: pgmap v551: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:24.731 INFO:tasks.workunit.client.0.smithi052.stdout: 10 86590 24.62 MB/sec execute 101 sec latency 5719.150 ms 2024-04-26T02:02:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:25 smithi052 ceph-mon[27851]: pgmap v552: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:25.732 INFO:tasks.workunit.client.0.smithi052.stdout: 10 87013 24.56 MB/sec execute 102 sec latency 6720.231 ms 2024-04-26T02:02:25.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:25 smithi191 ceph-mon[32745]: pgmap v552: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:25 smithi183 ceph-mon[32752]: pgmap v552: 97 pgs: 97 active+clean; 771 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-04-26T02:02:26.732 INFO:tasks.workunit.client.0.smithi052.stdout: 10 87519 24.55 MB/sec execute 103 sec latency 7093.450 ms 2024-04-26T02:02:27.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:27 smithi052 ceph-mon[27851]: pgmap v553: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:27.733 INFO:tasks.workunit.client.0.smithi052.stdout: 10 88170 24.76 MB/sec execute 104 sec latency 68.076 ms 2024-04-26T02:02:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:27 smithi191 ceph-mon[32745]: pgmap v553: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:27 smithi183 ceph-mon[32752]: pgmap v553: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.029+0000 7f29a9a00640 1 -- 172.21.15.52:0/3711502880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4109480 msgr2=0x7f29a4111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.029+0000 7f29a9a00640 1 --2- 172.21.15.52:0/3711502880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4109480 0x7f29a4111050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f299c009ca0 tx=0x7f299c02a8c0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 -- 172.21.15.52:0/3711502880 shutdown_connections 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 --2- 172.21.15.52:0/3711502880 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a4109480 0x7f29a4111050 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 --2- 172.21.15.52:0/3711502880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a4104d60 0x7f29a4108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 --2- 172.21.15.52:0/3711502880 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a41043b0 0x7f29a4104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 -- 172.21.15.52:0/3711502880 >> 172.21.15.52:0/3711502880 conn(0x7f29a40ffc50 msgr2=0x7f29a4102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 -- 172.21.15.52:0/3711502880 shutdown_connections 2024-04-26T02:02:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.030+0000 7f29a9a00640 1 -- 172.21.15.52:0/3711502880 wait complete. 2024-04-26T02:02:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.031+0000 7f29a9a00640 1 Processor -- start 2024-04-26T02:02:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 -- start start 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 0x7f29a419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 0x7f29a419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 0x7f29a419f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4114420 con 0x7f29a41043b0 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41142a0 con 0x7f29a4109480 2024-04-26T02:02:28.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.032+0000 7f29a9a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41145a0 con 0x7f29a419f010 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 0x7f29a419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 0x7f29a419ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43768/0 (socket says 172.21.15.52:43768) 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a2000640 1 -- 172.21.15.52:0/913434690 learned_addr learned my addr 172.21.15.52:0/913434690 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 0x7f29a419f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a2a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 0x7f29a419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 -- 172.21.15.52:0/913434690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 msgr2=0x7f29a419ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 0x7f29a419ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 -- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 msgr2=0x7f29a419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 0x7f29a419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a3400640 1 -- 172.21.15.52:0/913434690 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a419fb00 con 0x7f29a419f010 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.033+0000 7f29a2a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 0x7f29a419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.034+0000 7f29a3400640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 0x7f29a419f4c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f299c02ade0 tx=0x7f299c0319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.034+0000 7f299b400640 1 -- 172.21.15.52:0/913434690 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c003db0 con 0x7f29a419f010 2024-04-26T02:02:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.034+0000 7f299b400640 1 -- 172.21.15.52:0/913434690 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f299c0044f0 con 0x7f29a419f010 2024-04-26T02:02:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.034+0000 7f299b400640 1 -- 172.21.15.52:0/913434690 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c02fcf0 con 0x7f29a419f010 2024-04-26T02:02:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.035+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a419fd90 con 0x7f29a419f010 2024-04-26T02:02:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.035+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a41a7560 con 0x7f29a419f010 2024-04-26T02:02:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.039+0000 7f299b400640 1 -- 172.21.15.52:0/913434690 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f299c004040 con 0x7f29a419f010 2024-04-26T02:02:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.040+0000 7f29ac870740 1 Processor -- start 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.041+0000 7f29ac870740 1 -- start start 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.041+0000 7f299b400640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 0x7f29880a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f299b400640 1 -- 172.21.15.52:0/913434690 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f299c0f1f40 con 0x7f29a419f010 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29ac870740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29ac870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c86e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29ac870740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c870df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56244/0 (socket says 172.21.15.52:56244) 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29a2a00640 1 -- 172.21.15.52:0/2618459689 learned_addr learned my addr 172.21.15.52:0/2618459689 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29a2000640 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c86e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29a3400640 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c870df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.042+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f26c7c9900 con 0x55f26c86ea00 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f26c76fef0 con 0x55f26c864140 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f26c7c58c0 con 0x55f26c863820 2024-04-26T02:02:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2000640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 0x7f29880a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2a00640 1 -- 172.21.15.52:0/2618459689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 msgr2=0x55f26c86e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c86e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2a00640 1 -- 172.21.15.52:0/2618459689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 msgr2=0x55f26c870df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c870df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.043+0000 7f29a2a00640 1 -- 172.21.15.52:0/2618459689 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26c4dc210 con 0x55f26c863820 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.044+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f299400bc20 tx=0x7f299400d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad52e7858672fc11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.044+0000 7f298fe00640 1 -- 172.21.15.52:0/2618459689 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994017620 con 0x55f26c863820 2024-04-26T02:02:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.044+0000 7f298fe00640 1 -- 172.21.15.52:0/2618459689 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2994017c70 con 0x55f26c863820 2024-04-26T02:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.044+0000 7f298fe00640 1 -- 172.21.15.52:0/2618459689 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299400fc60 con 0x55f26c863820 2024-04-26T02:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 msgr2=0x55f26c863c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f299400bc20 tx=0x7f299400d710 comp rx=0 tx=0).stop 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29a2000640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 0x7f29880a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f26c864dc0 tx=0x7f299000a380 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 shutdown_connections 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c870df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c86e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 --2- 172.21.15.52:0/2618459689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c863c00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 >> 172.21.15.52:0/2618459689 conn(0x55f26c85fd30 msgr2=0x55f26c860110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 shutdown_connections 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.045+0000 7f29ac870740 1 -- 172.21.15.52:0/2618459689 wait complete. 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.046+0000 7f29ac870740 1 -- start start 2024-04-26T02:02:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.046+0000 7f29ac870740 1 Processor -- start 2024-04-26T02:02:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.046+0000 7f29ac870740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c86b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.046+0000 7f29ac870740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c86beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.046+0000 7f29ac870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29ac870740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f26c871630 con 0x55f26c864140 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29ac870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f26c783310 con 0x55f26c86ea00 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c86b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43782/0 (socket says 172.21.15.52:43782) 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c86b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56258/0 (socket says 172.21.15.52:56258) 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a3400640 1 -- 172.21.15.52:0/3543118153 learned_addr learned my addr 172.21.15.52:0/3543118153 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29a2000640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c86beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f26c8717b0 con 0x55f26c863820 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.047+0000 7f29ac870740 10 client.0 _ll_register_callbacks cb 0x55f26c751cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 -- 172.21.15.52:0/3543118153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 msgr2=0x55f26c86b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c86b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 -- 172.21.15.52:0/3543118153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 msgr2=0x55f26c86beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c86beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 -- 172.21.15.52:0/3543118153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26c5a8620 con 0x55f26c86ea00 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a2000640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c86beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29a3400640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f299c0bdcf0 tx=0x7f299c0bd950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255a136563f6f3ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f298cc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c0dc060 con 0x55f26c86ea00 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2994017c70 con 0x55f26c86ea00 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f299c0ba660 con 0x55f26c86ea00 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.048+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f299c0ce050 con 0x55f26c86ea00 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.049+0000 7f29ac870740 10 client.27140 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.049+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55f26c871630 con 0x55f26c86ea00 2024-04-26T02:02:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.049+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2994017c70 con 0x55f26c86ea00 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f299c0c67d0 con 0x55f26c86ea00 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 10 client.27140 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 10 client.27140 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 20 client.27140 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c889990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 4 client.27140 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f26c7dd8b0 con 0x55f26c887530 2024-04-26T02:02:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.050+0000 7f29a2a00640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c889990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.051+0000 7f29a2a00640 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c889990 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=270ee756eab10f83 server_cookie=65be8278e5d710f0 in_seq=0 out_seq=0 2024-04-26T02:02:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.051+0000 7f298cc00640 10 client.27140 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f298cc00640 1 -- 172.21.15.52:0/3543118153 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f26c7dd8b0 con 0x55f26c887530 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f298cc00640 10 client.27140 handle_command_reply: tid=0 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 client.27140 shutdown 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 msgr2=0x55f26c86c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f299c0bdcf0 tx=0x7f299c0bd950 comp rx=0 tx=0).stop 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 shutdown_connections 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c889990 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c86ea00 0x55f26c86c3f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f298cc00640 0 client.27140 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c86beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 --2- 172.21.15.52:0/3543118153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c86b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.053+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 >> 172.21.15.52:0/3543118153 conn(0x55f26c85fd30 msgr2=0x55f26c865a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.054+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 shutdown_connections 2024-04-26T02:02:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.054+0000 7f29ac870740 1 -- 172.21.15.52:0/3543118153 wait complete. 2024-04-26T02:02:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.054+0000 7f29ac870740 20 client.27140 trim_cache size 0 max 16384 2024-04-26T02:02:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.088+0000 7f29ac870740 1 Processor -- start 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 -- start start 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c88f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41ae130 con 0x55f26c864140 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41ab9a0 con 0x55f26c863820 2024-04-26T02:02:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.089+0000 7f29ac870740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41a9810 con 0x55f26c86ea00 2024-04-26T02:02:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.090+0000 7f29a3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.090+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.090+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43794/0 (socket says 172.21.15.52:43794) 2024-04-26T02:02:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.090+0000 7f29a2a00640 1 -- 172.21.15.52:0/3280911473 learned_addr learned my addr 172.21.15.52:0/3280911473 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.090+0000 7f29a3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:56268/0 (socket says 172.21.15.52:56268) 2024-04-26T02:02:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.091+0000 7f29a2000640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c88f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.091+0000 7f29a2a00640 1 -- 172.21.15.52:0/3280911473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 msgr2=0x55f26c88f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.091+0000 7f29a2a00640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.091+0000 7f29a2a00640 1 -- 172.21.15.52:0/3280911473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 msgr2=0x55f26c88f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.091+0000 7f29a2a00640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c88f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.092+0000 7f29a2a00640 1 -- 172.21.15.52:0/3280911473 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26c869ec0 con 0x55f26c863820 2024-04-26T02:02:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.092+0000 7f29a3400640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.093+0000 7f29a2000640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c88f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.094+0000 7f29a2a00640 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f299400b530 tx=0x7f299400f7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63278872e441b68c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.094+0000 7f298f400640 1 -- 172.21.15.52:0/3280911473 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994005c70 con 0x55f26c863820 2024-04-26T02:02:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.094+0000 7f298f400640 1 -- 172.21.15.52:0/3280911473 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2994005e10 con 0x55f26c863820 2024-04-26T02:02:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.094+0000 7f298f400640 1 -- 172.21.15.52:0/3280911473 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29940055a0 con 0x55f26c863820 2024-04-26T02:02:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.095+0000 7f29ac870740 1 -- 172.21.15.52:0/3280911473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 msgr2=0x55f26c8661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.095+0000 7f29ac870740 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f299400b530 tx=0x7f299400f7c0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.096+0000 7f29ac870740 1 -- 172.21.15.52:0/3280911473 shutdown_connections 2024-04-26T02:02:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.096+0000 7f29ac870740 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c86ea00 0x55f26c88f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.096+0000 7f29ac870740 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c864140 0x55f26c88f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.096+0000 7f29ac870740 1 --2- 172.21.15.52:0/3280911473 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c863820 0x55f26c8661e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.096+0000 7f29ac870740 1 -- 172.21.15.52:0/3280911473 >> 172.21.15.52:0/3280911473 conn(0x55f26c85fd30 msgr2=0x55f26c866c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.098+0000 7f29ac870740 1 -- 172.21.15.52:0/3280911473 shutdown_connections 2024-04-26T02:02:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.098+0000 7f29ac870740 1 -- 172.21.15.52:0/3280911473 wait complete. 2024-04-26T02:02:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.099+0000 7f29ac870740 1 -- start start 2024-04-26T02:02:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.099+0000 7f29ac870740 1 Processor -- start 2024-04-26T02:02:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.101+0000 7f29ac870740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c87f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.101+0000 7f29ac870740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c87c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.101+0000 7f29ac870740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29ac870740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41a9e60 con 0x55f26c86ea00 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29ac870740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41ab2d0 con 0x55f26c864140 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29ac870740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41aa530 con 0x55f26c863820 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29a3400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29a3400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54916/0 (socket says 172.21.15.52:54916) 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29a3400640 1 -- 172.21.15.52:0/2348258000 learned_addr learned my addr 172.21.15.52:0/2348258000 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.101+0000 7f29a2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c87f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29ac870740 10 client.0 _ll_register_callbacks cb 0x7f299c0bde90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.102+0000 7f29a2000640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c87c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a3400640 1 -- 172.21.15.52:0/2348258000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 msgr2=0x55f26c87f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a3400640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c87f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a3400640 1 -- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 msgr2=0x55f26c87c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a3400640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c87c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a3400640 1 -- 172.21.15.52:0/2348258000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26c87d2d0 con 0x55f26c86ea00 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2994005e10 con 0x55f26c86ea00 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a2000640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c87c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.103+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c87f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.104+0000 7f29a3400640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f299c0c23f0 tx=0x7f299c003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81727fa70297785 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.104+0000 7f298cc00640 10 client.16986 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:02:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.104+0000 7f298cc00640 1 -- 172.21.15.52:0/2348258000 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c0efa50 con 0x55f26c86ea00 2024-04-26T02:02:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.105+0000 7f298cc00640 1 -- 172.21.15.52:0/2348258000 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f299c004b40 con 0x55f26c86ea00 2024-04-26T02:02:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.105+0000 7f298cc00640 1 -- 172.21.15.52:0/2348258000 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f299c004d60 con 0x55f26c86ea00 2024-04-26T02:02:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.105+0000 7f29ac870740 10 client.16986 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.105+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=64}) -- 0x55f26c8ed7d0 con 0x55f26c86ea00 2024-04-26T02:02:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.106+0000 7f298cc00640 1 -- 172.21.15.52:0/2348258000 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f299c0d1070 con 0x55f26c86ea00 2024-04-26T02:02:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.106+0000 7f29ac870740 10 client.16986 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.106+0000 7f29ac870740 10 client.16986 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.106+0000 7f29ac870740 20 client.16986 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.106+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c8efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.107+0000 7f29ac870740 4 client.16986 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.107+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f26c7d7320 con 0x55f26c887530 2024-04-26T02:02:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.107+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c8efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.108+0000 7f29a2a00640 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c8efcd0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26fe7d3b7fa650ff server_cookie=30f51fe1962c5133 in_seq=0 out_seq=0 2024-04-26T02:02:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.108+0000 7f298cc00640 10 client.16986 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.304+0000 7f298cc00640 1 -- 172.21.15.52:0/2348258000 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f26c7d7320 con 0x55f26c887530 2024-04-26T02:02:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.304+0000 7f298cc00640 10 client.16986 handle_command_reply: tid=0 2024-04-26T02:02:28.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.305+0000 7f29ac870740 1 client.16986 shutdown 2024-04-26T02:02:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.306+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 msgr2=0x55f26c87cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.306+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f299c0c23f0 tx=0x7f299c003f50 comp rx=0 tx=0).stop 2024-04-26T02:02:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.306+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 shutdown_connections 2024-04-26T02:02:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.306+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f26c887530 0x55f26c8efcd0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.306+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f26c86ea00 0x55f26c87cd60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f26c864140 0x55f26c87c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f29ac870740 1 --2- 172.21.15.52:0/2348258000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f26c863820 0x55f26c87f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 >> 172.21.15.52:0/2348258000 conn(0x55f26c85fd30 msgr2=0x55f26c86d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f298cc00640 0 client.16986 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 shutdown_connections 2024-04-26T02:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.307+0000 7f29ac870740 1 -- 172.21.15.52:0/2348258000 wait complete. 2024-04-26T02:02:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.308+0000 7f29ac870740 20 client.16986 trim_cache size 0 max 16384 2024-04-26T02:02:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:02:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:02:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:02:28.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 msgr2=0x7f29880a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 0x7f29880a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f26c864dc0 tx=0x7f299000a380 comp rx=0 tx=0).stop 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 msgr2=0x7f29a419f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 0x7f29a419f4c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f299c02ade0 tx=0x7f299c0319f0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 shutdown_connections 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29880a3f20 0x7f29880a63e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29a419f010 0x7f29a419f4c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29a4109480 0x7f29a419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 --2- 172.21.15.52:0/913434690 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29a41043b0 0x7f29a419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 >> 172.21.15.52:0/913434690 conn(0x7f29a40ffc50 msgr2=0x7f29a4100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.309+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 shutdown_connections 2024-04-26T02:02:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:28.310+0000 7f29a9a00640 1 -- 172.21.15.52:0/913434690 wait complete. 2024-04-26T02:02:28.733 INFO:tasks.workunit.client.0.smithi052.stdout: 10 88563 24.72 MB/sec execute 105 sec latency 156.716 ms 2024-04-26T02:02:29.635 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-26T02:02:29.636 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-04-26T02:02:29.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:02:29.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:29 smithi052 ceph-mon[27851]: pgmap v554: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:29.733 INFO:tasks.workunit.client.0.smithi052.stdout: 10 88935 24.58 MB/sec execute 106 sec latency 439.626 ms 2024-04-26T02:02:29.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:29 smithi191 ceph-mon[32745]: pgmap v554: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:29 smithi183 ceph-mon[32752]: pgmap v554: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 490 op/s 2024-04-26T02:02:30.187 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:02:30.733 INFO:tasks.workunit.client.0.smithi052.stdout: 10 89376 24.63 MB/sec execute 107 sec latency 269.244 ms 2024-04-26T02:02:31.462 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:31 smithi052 ceph-mon[27851]: pgmap v555: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.2 MiB/s wr, 413 op/s 2024-04-26T02:02:31.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 89867 24.67 MB/sec execute 108 sec latency 284.316 ms 2024-04-26T02:02:31.902 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:31 smithi191 ceph-mon[32745]: pgmap v555: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.2 MiB/s wr, 413 op/s 2024-04-26T02:02:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:31 smithi183 ceph-mon[32752]: pgmap v555: 97 pgs: 97 active+clean; 821 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.2 MiB/s wr, 413 op/s 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 -- 172.21.15.52:0/4160751290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97006a750 msgr2=0x7fa97006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 --2- 172.21.15.52:0/4160751290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97006a750 0x7fa97006abb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009990 tx=0x7fa95c009ca0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 -- 172.21.15.52:0/4160751290 shutdown_connections 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 --2- 172.21.15.52:0/4160751290 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 0x7fa97010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 --2- 172.21.15.52:0/4160751290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970108a30 0x7fa970108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 --2- 172.21.15.52:0/4160751290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa97006a750 0x7fa97006abb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.590+0000 7fa977400700 1 -- 172.21.15.52:0/4160751290 >> 172.21.15.52:0/4160751290 conn(0x7fa9700788f0 msgr2=0x7fa970078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.591+0000 7fa977400700 1 -- 172.21.15.52:0/4160751290 shutdown_connections 2024-04-26T02:02:32.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.591+0000 7fa977400700 1 -- 172.21.15.52:0/4160751290 wait complete. 2024-04-26T02:02:32.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.592+0000 7fa977400700 1 Processor -- start 2024-04-26T02:02:32.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.593+0000 7fa977400700 1 -- start start 2024-04-26T02:02:32.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 0x7fa97019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:32.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 0x7fa97019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:32.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 0x7fa97019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:32.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa970112c60 con 0x7fa97006a750 2024-04-26T02:02:32.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa970112ae0 con 0x7fa97006b0f0 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa977400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa970112de0 con 0x7fa970108a30 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa974a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 0x7fa97019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa974a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 0x7fa97019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54948/0 (socket says 172.21.15.52:54948) 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa974a00700 1 -- 172.21.15.52:0/1545694957 learned_addr learned my addr 172.21.15.52:0/1545694957 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.594+0000 7fa975400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 0x7fa97019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa96fe00700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 0x7fa97019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:32.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa975400700 1 -- 172.21.15.52:0/1545694957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 msgr2=0x7fa97019b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:32.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa975400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 0x7fa97019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa975400700 1 -- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 msgr2=0x7fa97019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:32.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa975400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 0x7fa97019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:32.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.595+0000 7fa975400700 1 -- 172.21.15.52:0/1545694957 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9701a0300 con 0x7fa970108a30 2024-04-26T02:02:32.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.596+0000 7fa975400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 0x7fa97019fbc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa96800efc0 tx=0x7fa96800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.596+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680092e0 con 0x7fa970108a30 2024-04-26T02:02:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.596+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa968004bb0 con 0x7fa970108a30 2024-04-26T02:02:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.596+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968004080 con 0x7fa970108a30 2024-04-26T02:02:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.597+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9701a0590 con 0x7fa970108a30 2024-04-26T02:02:32.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.597+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9701a7e70 con 0x7fa970108a30 2024-04-26T02:02:32.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.599+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa970069730 con 0x7fa970108a30 2024-04-26T02:02:32.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.602+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa96801b030 con 0x7fa970108a30 2024-04-26T02:02:32.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.604+0000 7fa96d600700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 0x7fa9580a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:32.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.604+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7fa9680d02f0 con 0x7fa970108a30 2024-04-26T02:02:32.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.604+0000 7fa974a00700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 0x7fa9580a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:32.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.605+0000 7fa974a00700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 0x7fa9580a6440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa964005950 tx=0x7fa96400b500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:32.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:32.608+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa968022020 con 0x7fa970108a30 2024-04-26T02:02:32.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 90360 24.65 MB/sec execute 109 sec latency 105.613 ms 2024-04-26T02:02:33.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.063+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa97019c650 con 0x7fa970108a30 2024-04-26T02:02:33.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.064+0000 7fa96d600700 1 -- 172.21.15.52:0/1545694957 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11885 (secure 0 0 0) 0x7fa968010040 con 0x7fa970108a30 2024-04-26T02:02:33.063 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:02:33.063 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":64,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:01:35.377405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27086":{"gid":27086,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2553596297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2553596297},{"type":"v1","addr":"172.21.15.183:6837","nonce":2553596297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 msgr2=0x7fa9580a6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 0x7fa9580a6440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa964005950 tx=0x7fa96400b500 comp rx=0 tx=0).stop 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 msgr2=0x7fa97019fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 0x7fa97019fbc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa96800efc0 tx=0x7fa96800c3a0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 shutdown_connections 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fa9580a3f80 0x7fa9580a6440 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa970108a30 0x7fa97019fbc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97006a750 0x7fa97019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 --2- 172.21.15.52:0/1545694957 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa97006b0f0 0x7fa97019b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:33.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.067+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 >> 172.21.15.52:0/1545694957 conn(0x7fa9700788f0 msgr2=0x7fa97010df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:33.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.068+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 shutdown_connections 2024-04-26T02:02:33.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:33.068+0000 7fa977400700 1 -- 172.21.15.52:0/1545694957 wait complete. 2024-04-26T02:02:33.069 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 64 2024-04-26T02:02:33.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:33 smithi052 ceph-mon[27851]: pgmap v556: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 728 op/s 2024-04-26T02:02:33.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:33 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1545694957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:33.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:33 smithi191 ceph-mon[32745]: pgmap v556: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 728 op/s 2024-04-26T02:02:33.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:33 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1545694957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:33.734 INFO:tasks.workunit.client.0.smithi052.stdout: 10 90943 24.73 MB/sec execute 110 sec latency 139.272 ms 2024-04-26T02:02:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:33 smithi183 ceph-mon[32752]: pgmap v556: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 728 op/s 2024-04-26T02:02:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:33 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1545694957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:02:34.008 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0' 2024-04-26T02:02:34.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db / --timeout 60 --expiration 82.55 --await-for 42.0 2024-04-26T02:02:34.589 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:02:34.734 INFO:tasks.workunit.client.0.smithi052.stdout: 10 91523 24.81 MB/sec execute 111 sec latency 73.720 ms 2024-04-26T02:02:35.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:35 smithi052 ceph-mon[27851]: pgmap v557: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.0 MiB/s wr, 537 op/s 2024-04-26T02:02:37.055 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:35 smithi183 ceph-mon[32752]: pgmap v557: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.0 MiB/s wr, 537 op/s 2024-04-26T02:02:37.056 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:35 smithi191 ceph-mon[32745]: pgmap v557: 97 pgs: 97 active+clean; 824 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.0 MiB/s wr, 537 op/s 2024-04-26T02:02:37.056 INFO:tasks.workunit.client.0.smithi052.stdout: 10 92097 24.77 MB/sec execute 112 sec latency 93.712 ms 2024-04-26T02:02:37.056 INFO:tasks.workunit.client.0.smithi052.stdout: 10 92728 24.87 MB/sec execute 113 sec latency 93.619 ms 2024-04-26T02:02:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:37 smithi052 ceph-mon[27851]: pgmap v558: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-26T02:02:37.734 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93318 25.03 MB/sec execute 114 sec latency 195.405 ms 2024-04-26T02:02:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:37 smithi191 ceph-mon[32745]: pgmap v558: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-26T02:02:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:37 smithi183 ceph-mon[32752]: pgmap v558: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 739 op/s 2024-04-26T02:02:38.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93732 24.96 MB/sec execute 115 sec latency 115.168 ms 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 -- 172.21.15.52:0/4201123535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 msgr2=0x7f34441054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 --2- 172.21.15.52:0/4201123535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 0x7f34441054a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3434009990 tx=0x7f3434009ca0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 -- 172.21.15.52:0/4201123535 shutdown_connections 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 --2- 172.21.15.52:0/4201123535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f344406a740 0x7f344406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 --2- 172.21.15.52:0/4201123535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 0x7f34441054a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 --2- 172.21.15.52:0/4201123535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f34441059e0 0x7f3444105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.963+0000 7f3448a00700 1 -- 172.21.15.52:0/4201123535 >> 172.21.15.52:0/4201123535 conn(0x7f34440788e0 msgr2=0x7f3444078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.964+0000 7f3448a00700 1 -- 172.21.15.52:0/4201123535 shutdown_connections 2024-04-26T02:02:38.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.964+0000 7f3448a00700 1 -- 172.21.15.52:0/4201123535 wait complete. 2024-04-26T02:02:38.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.965+0000 7f3448a00700 1 Processor -- start 2024-04-26T02:02:38.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.965+0000 7f3448a00700 1 -- start start 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344406a740 0x7f344419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34441059e0 0x7f344419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 0x7f34441a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3444119210 con 0x7f34441059e0 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3444119090 con 0x7f344406a740 2024-04-26T02:02:38.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3448a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3444119390 con 0x7f34441079f0 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3441400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34441059e0 0x7f344419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.966+0000 7f3441400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34441059e0 0x7f344419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56284/0 (socket says 172.21.15.52:56284) 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 -- 172.21.15.52:0/4103265989 learned_addr learned my addr 172.21.15.52:0/4103265989 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441e00700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344406a740 0x7f344419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3442800700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 0x7f34441a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 -- 172.21.15.52:0/4103265989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 msgr2=0x7f34441a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34441079f0 0x7f34441a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 -- 172.21.15.52:0/4103265989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344406a740 msgr2=0x7f344419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:38.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f344406a740 0x7f344419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.967+0000 7f3441400700 1 -- 172.21.15.52:0/4103265989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34441a46b0 con 0x7f34441059e0 2024-04-26T02:02:38.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.968+0000 7f3441400700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34441059e0 0x7f344419fbe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f343000e9a0 tx=0x7f343000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:38.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.968+0000 7f343aa00700 1 -- 172.21.15.52:0/4103265989 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343000cbd0 con 0x7f34441059e0 2024-04-26T02:02:38.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.968+0000 7f343aa00700 1 -- 172.21.15.52:0/4103265989 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f343000cd70 con 0x7f34441059e0 2024-04-26T02:02:38.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.969+0000 7f343aa00700 1 -- 172.21.15.52:0/4103265989 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430018680 con 0x7f34441059e0 2024-04-26T02:02:38.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.969+0000 7f3448a00700 1 -- 172.21.15.52:0/4103265989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34441a49a0 con 0x7f34441059e0 2024-04-26T02:02:38.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.969+0000 7f3448a00700 1 -- 172.21.15.52:0/4103265989 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34441ac220 con 0x7f34441059e0 2024-04-26T02:02:38.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.973+0000 7f343aa00700 1 -- 172.21.15.52:0/4103265989 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3430018820 con 0x7f34441059e0 2024-04-26T02:02:38.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.975+0000 7f343aa00700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34280a3ed0 0x7f34280a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.975+0000 7f343aa00700 1 -- 172.21.15.52:0/4103265989 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f34300d0b30 con 0x7f34441059e0 2024-04-26T02:02:38.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.976+0000 7f345337eb80 1 Processor -- start 2024-04-26T02:02:38.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.976+0000 7f3441e00700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34280a3ed0 0x7f34280a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.976+0000 7f345337eb80 1 -- start start 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f345337eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928da1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f345337eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f345337eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f345337eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55979282d020 con 0x5597928da6f0 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f3442800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f3442800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39538/0 (socket says 172.21.15.52:39538) 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f3442800700 1 -- 172.21.15.52:0/390500383 learned_addr learned my addr 172.21.15.52:0/390500383 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.977+0000 7f3441400700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5597927d38c0 con 0x5597928d9dd0 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559792828fe0 con 0x5597928e5560 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f3441e00700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928da1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f3442800700 1 -- 172.21.15.52:0/390500383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 msgr2=0x5597928da1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f3442800700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928da1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f3442800700 1 -- 172.21.15.52:0/390500383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 msgr2=0x5597928e5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.978+0000 7f3442800700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.979+0000 7f3442800700 1 -- 172.21.15.52:0/390500383 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597927c9810 con 0x5597928e5560 2024-04-26T02:02:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.979+0000 7f3441e00700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928da1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:02:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.979+0000 7f3441400700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.980+0000 7f3442800700 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f343c011630 tx=0x7f343c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d61683df681112 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.980+0000 7f3441e00700 1 --2- 172.21.15.52:0/4103265989 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f34280a3ed0 0x7f34280a6390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f343400b2e0 tx=0x7f3434009550 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.980+0000 7f342f400700 1 -- 172.21.15.52:0/390500383 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c003a00 con 0x5597928e5560 2024-04-26T02:02:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.981+0000 7f342f400700 1 -- 172.21.15.52:0/390500383 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f343c01a070 con 0x5597928e5560 2024-04-26T02:02:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.981+0000 7f342f400700 1 -- 172.21.15.52:0/390500383 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c007500 con 0x5597928e5560 2024-04-26T02:02:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.982+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 msgr2=0x5597928e7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.982+0000 7f345337eb80 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f343c011630 tx=0x7f343c009920 comp rx=0 tx=0).stop 2024-04-26T02:02:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.983+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 shutdown_connections 2024-04-26T02:02:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.983+0000 7f345337eb80 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928e7950 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.983+0000 7f345337eb80 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928da1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.983+0000 7f345337eb80 1 --2- 172.21.15.52:0/390500383 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.983+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 >> 172.21.15.52:0/390500383 conn(0x5597928090e0 msgr2=0x5597928d62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.984+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 shutdown_connections 2024-04-26T02:02:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.985+0000 7f345337eb80 1 -- 172.21.15.52:0/390500383 wait complete. 2024-04-26T02:02:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.985+0000 7f345337eb80 1 -- start start 2024-04-26T02:02:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.986+0000 7f345337eb80 1 Processor -- start 2024-04-26T02:02:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928d9dd0 0x5597928e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 0x5597928e2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5597928e7ff0 con 0x5597928da6f0 2024-04-26T02:02:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5597927e69f0 con 0x5597928e5560 2024-04-26T02:02:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5597928e8170 con 0x5597928d9dd0 2024-04-26T02:02:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f345337eb80 10 client.0 _ll_register_callbacks cb 0x5597927c58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f3441400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f3441400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56304/0 (socket says 172.21.15.52:56304) 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f3441400700 1 -- 172.21.15.52:0/2486669371 learned_addr learned my addr 172.21.15.52:0/2486669371 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.987+0000 7f3442800700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 0x5597928e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441e00700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928d9dd0 0x5597928e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441400700 1 -- 172.21.15.52:0/2486669371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928d9dd0 msgr2=0x5597928e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441400700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928d9dd0 0x5597928e4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441400700 1 -- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 msgr2=0x5597928e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441400700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 0x5597928e2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3441400700 1 -- 172.21.15.52:0/2486669371 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55979277aa60 con 0x5597928da6f0 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.988+0000 7f3442800700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 0x5597928e2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.989+0000 7f3441400700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f34300ce280 tx=0x7f343009d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980c93b063386a07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.989+0000 7f3423e00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.989+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34300a0e20 con 0x5597928da6f0 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.989+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f34300a5250 con 0x5597928da6f0 2024-04-26T02:02:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.989+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f343c01a070 con 0x5597928da6f0 2024-04-26T02:02:38.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.990+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f34300a56e0 con 0x5597928da6f0 2024-04-26T02:02:38.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.990+0000 7f345337eb80 10 client.17004 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:38.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.990+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x5597927c3c80 con 0x5597928da6f0 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.990+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f343c01a070 con 0x5597928da6f0 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f34300b9020 con 0x5597928da6f0 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 10 client.17004 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 10 client.17004 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 20 client.17004 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5597928f4bd0 0x5597928f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 4 client.17004 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.991+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55979283a810 con 0x5597928f4bd0 2024-04-26T02:02:38.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.992+0000 7f3441e00700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5597928f4bd0 0x5597928f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:38.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.993+0000 7f3441e00700 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5597928f4bd0 0x5597928f6fc0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24a0f02f716e651a server_cookie=373ca7cf01682acb in_seq=0 out_seq=0 2024-04-26T02:02:38.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.993+0000 7f3423e00700 10 client.17004 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:38.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.994+0000 7f3423e00700 1 -- 172.21.15.52:0/2486669371 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55979283a810 con 0x5597928f4bd0 2024-04-26T02:02:38.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.994+0000 7f3423e00700 10 client.17004 handle_command_reply: tid=0 2024-04-26T02:02:38.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.994+0000 7f345337eb80 1 client.17004 shutdown 2024-04-26T02:02:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.995+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 msgr2=0x5597928e1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.995+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f34300ce280 tx=0x7f343009d280 comp rx=0 tx=0).stop 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 shutdown_connections 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5597928f4bd0 0x5597928f6fc0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928e5560 0x5597928e2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928d9dd0 0x5597928e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 --2- 172.21.15.52:0/2486669371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928e1fd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 >> 172.21.15.52:0/2486669371 conn(0x5597928090e0 msgr2=0x5597928d5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f3423e00700 0 client.17004 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:38.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.996+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 shutdown_connections 2024-04-26T02:02:38.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.997+0000 7f345337eb80 1 -- 172.21.15.52:0/2486669371 wait complete. 2024-04-26T02:02:38.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:38.997+0000 7f345337eb80 20 client.17004 trim_cache size 0 max 16384 2024-04-26T02:02:39.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 Processor -- start 2024-04-26T02:02:39.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 -- start start 2024-04-26T02:02:39.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 0x5597928e3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928df290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b6210 con 0x5597928d9dd0 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b3a40 con 0x5597928da6f0 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f345337eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b18b0 con 0x5597928e5560 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.027+0000 7f3441e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 0x5597928e3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37466/0 (socket says 172.21.15.52:37466) 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 -- 172.21.15.52:0/4150862092 learned_addr learned my addr 172.21.15.52:0/4150862092 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:39.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 0x5597928e3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56318/0 (socket says 172.21.15.52:56318) 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3442800700 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928df290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 -- 172.21.15.52:0/4150862092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 msgr2=0x5597928df290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928df290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 -- 172.21.15.52:0/4150862092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 msgr2=0x5597928e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 0x5597928e3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 -- 172.21.15.52:0/4150862092 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597928d76f0 con 0x5597928da6f0 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.028+0000 7f3441400700 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f343009ca00 tx=0x7f3430010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871d91652ff50c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:39.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.029+0000 7f342ea00700 1 -- 172.21.15.52:0/4150862092 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34300a0e20 con 0x5597928da6f0 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.029+0000 7f342ea00700 1 -- 172.21.15.52:0/4150862092 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34300bd0b0 con 0x5597928da6f0 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.029+0000 7f342ea00700 1 -- 172.21.15.52:0/4150862092 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430087440 con 0x5597928da6f0 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 -- 172.21.15.52:0/4150862092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 msgr2=0x5597928f0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f343009ca00 tx=0x7f3430010c80 comp rx=0 tx=0).stop 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 -- 172.21.15.52:0/4150862092 shutdown_connections 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928df290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928d9dd0 0x5597928e3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 --2- 172.21.15.52:0/4150862092 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928da6f0 0x5597928f0ef0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.030+0000 7f345337eb80 1 -- 172.21.15.52:0/4150862092 >> 172.21.15.52:0/4150862092 conn(0x5597928090e0 msgr2=0x5597928d5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:39.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.031+0000 7f345337eb80 1 -- 172.21.15.52:0/4150862092 shutdown_connections 2024-04-26T02:02:39.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.031+0000 7f345337eb80 1 -- 172.21.15.52:0/4150862092 wait complete. 2024-04-26T02:02:39.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.032+0000 7f345337eb80 1 -- start start 2024-04-26T02:02:39.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.032+0000 7f345337eb80 1 Processor -- start 2024-04-26T02:02:39.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928eae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b1f00 con 0x5597928da6f0 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b3370 con 0x5597928d9dd0 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f34441b25d0 con 0x5597928e5560 2024-04-26T02:02:39.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.033+0000 7f345337eb80 10 client.0 _ll_register_callbacks cb 0x5597927c58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.034+0000 7f3441e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.034+0000 7f3441e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37472/0 (socket says 172.21.15.52:37472) 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.034+0000 7f3441e00700 1 -- 172.21.15.52:0/3987354076 learned_addr learned my addr 172.21.15.52:0/3987354076 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.034+0000 7f3441400700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.034+0000 7f3442800700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 -- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 msgr2=0x5597928ea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928ea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 -- 172.21.15.52:0/3987354076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 msgr2=0x5597928ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 -- 172.21.15.52:0/3987354076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55979295b0f0 con 0x5597928e5560 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3442800700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928eae50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f343c003fd0 tx=0x7f343c011f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f8f2c7a0340258 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3423e00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:02:39.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c003e10 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f343c004050 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.035+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34300bd0b0 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f343c004290 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f345337eb80 10 client.27175 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55979295c390 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34300bd0b0 con 0x5597928e5560 2024-04-26T02:02:39.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f343c015450 con 0x5597928e5560 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f345337eb80 10 client.27175 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f345337eb80 10 client.27175 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.036+0000 7f345337eb80 20 client.27175 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.037+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55979295c9a0 0x55979295edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.037+0000 7f345337eb80 4 client.27175 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}] 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.037+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.55, "await_for": 42.0}) -- 0x5597928348b0 con 0x55979295c9a0 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.037+0000 7f3441e00700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55979295c9a0 0x55979295edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.038+0000 7f3441e00700 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55979295c9a0 0x55979295edc0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e27c0ff6f55dfc server_cookie=7296c077c412c879 in_seq=0 out_seq=0 2024-04-26T02:02:39.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:02:39.038+0000 7f3423e00700 10 client.27175 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:39.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:39 smithi052 ceph-mon[27851]: pgmap v559: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:39.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 24.85 MB/sec execute 116 sec latency 663.846 ms 2024-04-26T02:02:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:39 smithi191 ceph-mon[32745]: pgmap v559: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:39 smithi183 ceph-mon[32752]: pgmap v559: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:40.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:40 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:40.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 24.63 MB/sec execute 117 sec latency 1663.971 ms 2024-04-26T02:02:40.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:40 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:40.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:40 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:02:41.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 -- 172.21.15.52:0/3599603439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1021d0 msgr2=0x7f523c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 --2- 172.21.15.52:0/3599603439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1021d0 0x7f523c1025b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f522c009900 tx=0x7f522c02a3f0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 -- 172.21.15.52:0/3599603439 shutdown_connections 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 --2- 172.21.15.52:0/3599603439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c1072a0 0x7f523c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 --2- 172.21.15.52:0/3599603439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c102b80 0x7f523c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 --2- 172.21.15.52:0/3599603439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1021d0 0x7f523c1025b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.456+0000 7f5241c00640 1 -- 172.21.15.52:0/3599603439 >> 172.21.15.52:0/3599603439 conn(0x7f523c0793e0 msgr2=0x7f523c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.457+0000 7f5241c00640 1 -- 172.21.15.52:0/3599603439 shutdown_connections 2024-04-26T02:02:41.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.457+0000 7f5241c00640 1 -- 172.21.15.52:0/3599603439 wait complete. 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 Processor -- start 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 -- start start 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 0x7f523c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 0x7f523c19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f523c112230 con 0x7f523c19a6e0 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1120b0 con 0x7f523c1072a0 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f5241c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1123b0 con 0x7f523c1021d0 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.458+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 0x7f523c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39584/0 (socket says 172.21.15.52:39584) 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 -- 172.21.15.52:0/22993929 learned_addr learned my addr 172.21.15.52:0/22993929 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:41.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523b400640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 0x7f523c19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 -- 172.21.15.52:0/22993929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 msgr2=0x7f523c1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 0x7f523c1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 -- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 msgr2=0x7f523c19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 0x7f523c19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 -- 172.21.15.52:0/22993929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523c19b2e0 con 0x7f523c1021d0 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523b400640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 0x7f523c19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523a000640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 0x7f523c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.459+0000 7f523aa00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f522c005ab0 tx=0x7f522c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.460+0000 7f5233400640 1 -- 172.21.15.52:0/22993929 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c03f070 con 0x7f523c1021d0 2024-04-26T02:02:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.460+0000 7f5233400640 1 -- 172.21.15.52:0/22993929 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c031b60 con 0x7f523c1021d0 2024-04-26T02:02:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.460+0000 7f5233400640 1 -- 172.21.15.52:0/22993929 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c035030 con 0x7f523c1021d0 2024-04-26T02:02:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.460+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523c19b570 con 0x7f523c1021d0 2024-04-26T02:02:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.460+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f523c1a7660 con 0x7f523c1021d0 2024-04-26T02:02:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.463+0000 7f5233400640 1 -- 172.21.15.52:0/22993929 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f522c042070 con 0x7f523c1021d0 2024-04-26T02:02:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.463+0000 7f5244ba3740 1 Processor -- start 2024-04-26T02:02:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 -- start start 2024-04-26T02:02:41.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 0x556dec0034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec005df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556debf5e900 con 0x556debff8820 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556debf04ef0 con 0x556dec003a00 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f5244ba3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556debf5a8c0 con 0x556debff9140 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f523b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f523b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56356/0 (socket says 172.21.15.52:56356) 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f523b400640 1 -- 172.21.15.52:0/2226992509 learned_addr learned my addr 172.21.15.52:0/2226992509 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.464+0000 7f523aa00640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 0x556dec0034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f5233400640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 0x7f52200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523a000640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec005df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f5233400640 1 -- 172.21.15.52:0/22993929 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f522c0f0b90 con 0x7f523c1021d0 2024-04-26T02:02:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523aa00640 1 -- 172.21.15.52:0/2226992509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 msgr2=0x556dec005df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523aa00640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec005df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523aa00640 1 -- 172.21.15.52:0/2226992509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 msgr2=0x556debff8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523aa00640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523aa00640 1 -- 172.21.15.52:0/2226992509 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556debc71210 con 0x556debff9140 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523b400640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.465+0000 7f523b400640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 0x7f52200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.466+0000 7f523aa00640 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 0x556dec0034c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f522c0ee280 tx=0x7f522c03b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9dc5137775b92e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.466+0000 7f5227e00640 1 -- 172.21.15.52:0/2226992509 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c049070 con 0x556debff9140 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.466+0000 7f5227e00640 1 -- 172.21.15.52:0/2226992509 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c0bd1d0 con 0x556debff9140 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.466+0000 7f5227e00640 1 -- 172.21.15.52:0/2226992509 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c0f5040 con 0x556debff9140 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- 172.21.15.52:0/2226992509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 msgr2=0x556dec0034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f523b400640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 0x7f52200a6300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f52340099a0 tx=0x7f5234004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 0x556dec0034c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f522c0ee280 tx=0x7f522c03b800 comp rx=0 tx=0).stop 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- 172.21.15.52:0/2226992509 shutdown_connections 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec005df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff9140 0x556dec0034c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 --2- 172.21.15.52:0/2226992509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff8820 0x556debff8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- 172.21.15.52:0/2226992509 >> 172.21.15.52:0/2226992509 conn(0x556debff4d30 msgr2=0x556debff5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- 172.21.15.52:0/2226992509 shutdown_connections 2024-04-26T02:02:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- 172.21.15.52:0/2226992509 wait complete. 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.467+0000 7f5244ba3740 1 -- start start 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 Processor -- start 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec000f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec001440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556debf5a8c0 con 0x556debff9140 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556debf18310 con 0x556dec003a00 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556dec0063f0 con 0x556debff8820 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f5244ba3740 10 client.0 _ll_register_callbacks cb 0x556debee6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39594/0 (socket says 172.21.15.52:39594) 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f523aa00640 1 -- 172.21.15.52:0/47361613 learned_addr learned my addr 172.21.15.52:0/47361613 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.468+0000 7f523b400640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec001440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523a000640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec000f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 -- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 msgr2=0x556dec001440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec001440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 -- 172.21.15.52:0/47361613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 msgr2=0x556dec000f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec000f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 -- 172.21.15.52:0/47361613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556debd3d620 con 0x556debff8820 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523a000640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec000f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f523aa00640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f522c031440 tx=0x7f522c031d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eabd6f7acaf27ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f5224c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c049070 con 0x556debff8820 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f522c0badd0 con 0x556debff8820 2024-04-26T02:02:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.469+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f522c0bd1d0 con 0x556debff8820 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f522c0bc430 con 0x556debff8820 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 10 client.27187 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x556dec012500 con 0x556debff8820 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c0bd1d0 con 0x556debff8820 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f522c03b460 con 0x556debff8820 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 10 client.27187 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 10 client.27187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 20 client.27187 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec0138a0 0x556dec015cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 4 client.27187 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556debf728b0 con 0x556dec0138a0 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.470+0000 7f523a000640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec0138a0 0x556dec015cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.471+0000 7f523a000640 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec0138a0 0x556dec015cd0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39aff03734c89504 server_cookie=81f5d1715b3dae83 in_seq=0 out_seq=0 2024-04-26T02:02:41.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.471+0000 7f5224c00640 10 client.27187 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:41.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5224c00640 1 -- 172.21.15.52:0/47361613 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556debf728b0 con 0x556dec0138a0 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5224c00640 10 client.27187 handle_command_reply: tid=0 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 client.27187 shutdown 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 msgr2=0x556dec0009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f522c031440 tx=0x7f522c031d10 comp rx=0 tx=0).stop 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 shutdown_connections 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec0138a0 0x556dec015cd0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec001440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec000f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5224c00640 0 client.27187 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 --2- 172.21.15.52:0/47361613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec0009c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 >> 172.21.15.52:0/47361613 conn(0x556debff4d30 msgr2=0x556debffaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 shutdown_connections 2024-04-26T02:02:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 1 -- 172.21.15.52:0/47361613 wait complete. 2024-04-26T02:02:41.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.472+0000 7f5244ba3740 20 client.27187 trim_cache size 0 max 16384 2024-04-26T02:02:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 Processor -- start 2024-04-26T02:02:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 -- start start 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec01b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556dec003a00 0x556dec01bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1b2690 con 0x556debff9140 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1aff00 con 0x556debff8820 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f5244ba3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1add70 con 0x556dec003a00 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f523aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f523aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37516/0 (socket says 172.21.15.52:37516) 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.489+0000 7f523aa00640 1 -- 172.21.15.52:0/155724497 learned_addr learned my addr 172.21.15.52:0/155724497 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523a000640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec01b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523b400640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556dec003a00 0x556dec01bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 -- 172.21.15.52:0/155724497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556dec003a00 msgr2=0x556dec01bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556dec003a00 0x556dec01bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 -- 172.21.15.52:0/155724497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 msgr2=0x556dec01b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec01b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 -- 172.21.15.52:0/155724497 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556debff9f60 con 0x556debff8820 2024-04-26T02:02:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523a000640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec01b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:02:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.490+0000 7f523aa00640 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f522c0f0e80 tx=0x7f522c0dd050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6fdf0a637a3be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5227400640 1 -- 172.21.15.52:0/155724497 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c049070 con 0x556debff8820 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5227400640 1 -- 172.21.15.52:0/155724497 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c0bd1d0 con 0x556debff8820 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5227400640 1 -- 172.21.15.52:0/155724497 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c0bae80 con 0x556debff8820 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 -- 172.21.15.52:0/155724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 msgr2=0x556dec00f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f522c0f0e80 tx=0x7f522c0dd050 comp rx=0 tx=0).stop 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 -- 172.21.15.52:0/155724497 shutdown_connections 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556dec003a00 0x556dec01bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec01b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 --2- 172.21.15.52:0/155724497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556debff8820 0x556dec00f130 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.491+0000 7f5244ba3740 1 -- 172.21.15.52:0/155724497 >> 172.21.15.52:0/155724497 conn(0x556debff4d30 msgr2=0x556debffbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.492+0000 7f5244ba3740 1 -- 172.21.15.52:0/155724497 shutdown_connections 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.492+0000 7f5244ba3740 1 -- 172.21.15.52:0/155724497 wait complete. 2024-04-26T02:02:41.493 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.492+0000 7f5244ba3740 1 -- start start 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.492+0000 7f5244ba3740 1 Processor -- start 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec009570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec009ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1ae3c0 con 0x556debff9140 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1af830 con 0x556dec003a00 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f523c1aea90 con 0x556debff8820 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f5244ba3740 10 client.0 _ll_register_callbacks cb 0x7f522c0f2fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39610/0 (socket says 172.21.15.52:39610) 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec009ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 -- 172.21.15.52:0/323990817 learned_addr learned my addr 172.21.15.52:0/323990817 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523a000640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec009570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 -- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 msgr2=0x556dec009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec009ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 -- 172.21.15.52:0/323990817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 msgr2=0x556dec009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec009570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.493+0000 7f523aa00640 1 -- 172.21.15.52:0/323990817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dec078e60 con 0x556debff8820 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f523aa00640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f522c0bad80 tx=0x7f522c03b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576f375e2f7cbd1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c049070 con 0x556debff8820 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8521+0+0 (secure 0 0 0) 0x7f522c03b460 con 0x556debff8820 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f522c0bd1d0 con 0x556debff8820 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f522c0bcad0 con 0x556debff8820 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5244ba3740 10 client.27193 fetch_fsmap learned FSMap version 64 2024-04-26T02:02:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x556dec078e60 con 0x556debff8820 2024-04-26T02:02:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f522c0bd1d0 con 0x556debff8820 2024-04-26T02:02:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.494+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 64) ==== 8346+0+0 (secure 0 0 0) 0x7f522c0bbf20 con 0x556debff8820 2024-04-26T02:02:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 10 client.27193 fetch_fsmap finished waiting for FSMap version 64 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 10 client.27193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 20 client.27193 populate_metadata read hostname 'smithi052' 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec07a360 0x556dec07c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 4 client.27193 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556debf6c320 con 0x556dec07a360 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f523a000640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec07a360 0x556dec07c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f523a000640 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec07a360 0x556dec07c750 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fff85c8195251c47 server_cookie=46183658a6d04d94 in_seq=0 out_seq=0 2024-04-26T02:02:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:02:41.495+0000 7f5224c00640 10 client.27193 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:02:41.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:41 smithi052 ceph-mon[27851]: pgmap v560: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:41.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 24.43 MB/sec execute 118 sec latency 2664.402 ms 2024-04-26T02:02:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:41 smithi191 ceph-mon[32745]: pgmap v560: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:41 smithi183 ceph-mon[32752]: pgmap v560: 97 pgs: 97 active+clean; 838 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 521 op/s 2024-04-26T02:02:42.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:42 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:42.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:42 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:42.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:42 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:02:42.735 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 24.22 MB/sec execute 119 sec latency 3664.523 ms 2024-04-26T02:02:42.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:42 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:42.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:42 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:42.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:42 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:02:42.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:42 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:02:42.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:42 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:02:42.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:42 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:02:43.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:43 smithi052 ceph-mon[27851]: pgmap v561: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 677 op/s 2024-04-26T02:02:43.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 24.02 MB/sec execute 120 sec latency 4664.686 ms 2024-04-26T02:02:43.827 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:43 smithi183 ceph-mon[32752]: pgmap v561: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 677 op/s 2024-04-26T02:02:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:43 smithi191 ceph-mon[32745]: pgmap v561: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 9.0 MiB/s wr, 677 op/s 2024-04-26T02:02:44.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 23.82 MB/sec execute 121 sec latency 5664.846 ms 2024-04-26T02:02:45.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 23.62 MB/sec execute 122 sec latency 6665.014 ms 2024-04-26T02:02:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:45 smithi191 ceph-mon[32745]: pgmap v562: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 358 op/s 2024-04-26T02:02:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:45 smithi183 ceph-mon[32752]: pgmap v562: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 358 op/s 2024-04-26T02:02:45.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:45 smithi052 ceph-mon[27851]: pgmap v562: 97 pgs: 97 active+clean; 862 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 358 op/s 2024-04-26T02:02:46.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 23.43 MB/sec execute 123 sec latency 7665.137 ms 2024-04-26T02:02:47.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 23.24 MB/sec execute 124 sec latency 8665.326 ms 2024-04-26T02:02:47.766 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:47 smithi052 ceph-mon[27851]: pgmap v563: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.3 MiB/s wr, 487 op/s 2024-04-26T02:02:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:47 smithi191 ceph-mon[32745]: pgmap v563: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.3 MiB/s wr, 487 op/s 2024-04-26T02:02:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:47 smithi183 ceph-mon[32752]: pgmap v563: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.3 MiB/s wr, 487 op/s 2024-04-26T02:02:48.736 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 23.06 MB/sec execute 125 sec latency 9665.442 ms 2024-04-26T02:02:49.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.87 MB/sec execute 126 sec latency 10665.562 ms 2024-04-26T02:02:49.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:49 smithi191 ceph-mon[32745]: pgmap v564: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:49 smithi183 ceph-mon[32752]: pgmap v564: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:49 smithi052 ceph-mon[27851]: pgmap v564: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:50.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.69 MB/sec execute 127 sec latency 11665.721 ms 2024-04-26T02:02:51.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.52 MB/sec execute 128 sec latency 12665.868 ms 2024-04-26T02:02:51.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:51 smithi191 ceph-mon[32745]: pgmap v565: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:51.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:51 smithi183 ceph-mon[32752]: pgmap v565: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:51 smithi052 ceph-mon[27851]: pgmap v565: 97 pgs: 97 active+clean; 873 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 285 op/s 2024-04-26T02:02:52.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.34 MB/sec execute 129 sec latency 13666.035 ms 2024-04-26T02:02:53.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.17 MB/sec execute 130 sec latency 14666.181 ms 2024-04-26T02:02:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:53 smithi191 ceph-mon[32745]: pgmap v566: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-04-26T02:02:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:53 smithi183 ceph-mon[32752]: pgmap v566: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-04-26T02:02:53.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:53 smithi052 ceph-mon[27851]: pgmap v566: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 332 op/s 2024-04-26T02:02:54.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 22.00 MB/sec execute 131 sec latency 15666.389 ms 2024-04-26T02:02:55.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.83 MB/sec execute 132 sec latency 16666.564 ms 2024-04-26T02:02:55.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:55 smithi191 ceph-mon[32745]: pgmap v567: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-26T02:02:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:55 smithi183 ceph-mon[32752]: pgmap v567: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-26T02:02:55.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:55 smithi052 ceph-mon[27851]: pgmap v567: 97 pgs: 97 active+clean; 850 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-26T02:02:56.737 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.67 MB/sec execute 133 sec latency 17666.645 ms 2024-04-26T02:02:57.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.51 MB/sec execute 134 sec latency 18666.759 ms 2024-04-26T02:02:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:57 smithi191 ceph-mon[32745]: pgmap v568: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 4.2 MiB/s wr, 246 op/s 2024-04-26T02:02:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:57 smithi183 ceph-mon[32752]: pgmap v568: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 4.2 MiB/s wr, 246 op/s 2024-04-26T02:02:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:57 smithi052 ceph-mon[27851]: pgmap v568: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 4.2 MiB/s wr, 246 op/s 2024-04-26T02:02:58.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.35 MB/sec execute 135 sec latency 19666.900 ms 2024-04-26T02:02:59.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.19 MB/sec execute 136 sec latency 20667.058 ms 2024-04-26T02:02:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:02:59 smithi191 ceph-mon[32745]: pgmap v569: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:02:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:02:59 smithi183 ceph-mon[32752]: pgmap v569: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:02:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:02:59 smithi052 ceph-mon[27851]: pgmap v569: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:03:00.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 21.04 MB/sec execute 137 sec latency 21667.153 ms 2024-04-26T02:03:01.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.89 MB/sec execute 138 sec latency 22667.243 ms 2024-04-26T02:03:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:01 smithi191 ceph-mon[32745]: pgmap v570: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:03:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:01 smithi183 ceph-mon[32752]: pgmap v570: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:03:01.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:01 smithi052 ceph-mon[27851]: pgmap v570: 97 pgs: 97 active+clean; 848 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 2.3 MiB/s wr, 118 op/s 2024-04-26T02:03:02.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.73 MB/sec execute 139 sec latency 23667.414 ms 2024-04-26T02:03:03.738 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.59 MB/sec execute 140 sec latency 24667.585 ms 2024-04-26T02:03:03.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:03 smithi191 ceph-mon[32745]: pgmap v571: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-26T02:03:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:03 smithi183 ceph-mon[32752]: pgmap v571: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-26T02:03:03.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:03 smithi052 ceph-mon[27851]: pgmap v571: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-26T02:03:04.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.44 MB/sec execute 141 sec latency 25667.752 ms 2024-04-26T02:03:05.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.30 MB/sec execute 142 sec latency 26667.868 ms 2024-04-26T02:03:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:05 smithi191 ceph-mon[32745]: pgmap v572: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 3.8 MiB/s wr, 153 op/s 2024-04-26T02:03:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:05 smithi183 ceph-mon[32752]: pgmap v572: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 3.8 MiB/s wr, 153 op/s 2024-04-26T02:03:05.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:05 smithi052 ceph-mon[27851]: pgmap v572: 97 pgs: 97 active+clean; 846 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 3.8 MiB/s wr, 153 op/s 2024-04-26T02:03:06.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.15 MB/sec execute 143 sec latency 27667.955 ms 2024-04-26T02:03:07.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 20.01 MB/sec execute 144 sec latency 28668.021 ms 2024-04-26T02:03:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:07 smithi191 ceph-mon[32745]: pgmap v573: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 5.9 MiB/s wr, 222 op/s 2024-04-26T02:03:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:07 smithi191 ceph-mon[32745]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:03:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:07 smithi183 ceph-mon[32752]: pgmap v573: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 5.9 MiB/s wr, 222 op/s 2024-04-26T02:03:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:07 smithi183 ceph-mon[32752]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:03:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:07 smithi052 ceph-mon[27851]: pgmap v573: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 5.9 MiB/s wr, 222 op/s 2024-04-26T02:03:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:07 smithi052 ceph-mon[27851]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T02:03:08.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.88 MB/sec execute 145 sec latency 29668.144 ms 2024-04-26T02:03:09.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.74 MB/sec execute 146 sec latency 30668.322 ms 2024-04-26T02:03:09.905 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:09 smithi191 ceph-mon[32745]: pgmap v575: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:09 smithi183 ceph-mon[32752]: pgmap v575: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:09 smithi052 ceph-mon[27851]: pgmap v575: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:10.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.61 MB/sec execute 147 sec latency 31668.432 ms 2024-04-26T02:03:11.739 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.47 MB/sec execute 148 sec latency 32668.578 ms 2024-04-26T02:03:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:11 smithi191 ceph-mon[32745]: pgmap v576: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:11 smithi183 ceph-mon[32752]: pgmap v576: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:11.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:11 smithi052 ceph-mon[27851]: pgmap v576: 97 pgs: 97 active+clean; 845 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 4.9 MiB/s wr, 182 op/s 2024-04-26T02:03:12.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.34 MB/sec execute 149 sec latency 33668.730 ms 2024-04-26T02:03:13.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.21 MB/sec execute 150 sec latency 34668.838 ms 2024-04-26T02:03:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:13 smithi191 ceph-mon[32745]: pgmap v577: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 4.9 MiB/s wr, 167 op/s 2024-04-26T02:03:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:13 smithi191 ceph-mon[32745]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:13 smithi183 ceph-mon[32752]: pgmap v577: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 4.9 MiB/s wr, 167 op/s 2024-04-26T02:03:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:13 smithi183 ceph-mon[32752]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:13 smithi052 ceph-mon[27851]: pgmap v577: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 4.9 MiB/s wr, 167 op/s 2024-04-26T02:03:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:13 smithi052 ceph-mon[27851]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T02:03:14.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 19.09 MB/sec execute 151 sec latency 35669.003 ms 2024-04-26T02:03:15.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.96 MB/sec execute 152 sec latency 36669.104 ms 2024-04-26T02:03:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:15 smithi191 ceph-mon[32745]: pgmap v579: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 3.0 MiB/s wr, 106 op/s 2024-04-26T02:03:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:15 smithi183 ceph-mon[32752]: pgmap v579: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 3.0 MiB/s wr, 106 op/s 2024-04-26T02:03:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:15 smithi052 ceph-mon[27851]: pgmap v579: 97 pgs: 97 active+clean; 845 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 3.0 MiB/s wr, 106 op/s 2024-04-26T02:03:16.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.84 MB/sec execute 153 sec latency 37669.258 ms 2024-04-26T02:03:17.740 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.72 MB/sec execute 154 sec latency 38669.477 ms 2024-04-26T02:03:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:17 smithi191 ceph-mon[32745]: pgmap v580: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-04-26T02:03:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:17 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:17 smithi183 ceph-mon[32752]: pgmap v580: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-04-26T02:03:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:17 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:17 smithi052 ceph-mon[27851]: pgmap v580: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-04-26T02:03:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:17 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:18.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.59 MB/sec execute 155 sec latency 39669.654 ms 2024-04-26T02:03:19.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.48 MB/sec execute 156 sec latency 40669.814 ms 2024-04-26T02:03:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:19 smithi191 ceph-mon[32745]: pgmap v581: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:19 smithi183 ceph-mon[32752]: pgmap v581: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:19 smithi052 ceph-mon[27851]: pgmap v581: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:20.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.36 MB/sec execute 157 sec latency 41669.958 ms 2024-04-26T02:03:21.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.039+0000 7f3423e00700 1 -- 172.21.15.52:0/3987354076 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: -115 ) ==== 8+0+593 (crc 0 0 0) 0x5597928348b0 con 0x55979295c9a0 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.039+0000 7f3423e00700 10 client.27175 handle_command_reply: tid=0 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.039+0000 7f345337eb80 1 client.27175 shutdown 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 msgr2=0x5597928eae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928eae50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f343c003fd0 tx=0x7f343c011f20 comp rx=0 tx=0).stop 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 shutdown_connections 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55979295c9a0 0x55979295edc0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5597928e5560 0x5597928eae50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5597928d9dd0 0x5597928ea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 --2- 172.21.15.52:0/3987354076 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5597928da6f0 0x5597928ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f3423e00700 0 client.27175 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 >> 172.21.15.52:0/3987354076 conn(0x5597928090e0 msgr2=0x5597928efee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.040+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 shutdown_connections 2024-04-26T02:03:21.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.041+0000 7f345337eb80 1 -- 172.21.15.52:0/3987354076 wait complete. 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:21.041+0000 7f345337eb80 20 client.27175 trim_cache size 0 max 16384 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 65, 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 49, 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "f112abc": { 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 49, 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 0.0, 2024-04-26T02:03:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCING", 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 42.0 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.5, 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCING", 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 42.0 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:21.045 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:03:21.045 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINPROGRESS: 2024-04-26T02:03:21.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:21 smithi052 ceph-mon[27851]: pgmap v582: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:21.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.24 MB/sec execute 158 sec latency 42670.101 ms 2024-04-26T02:03:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:21 smithi191 ceph-mon[32745]: pgmap v582: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:21 smithi183 ceph-mon[32752]: pgmap v582: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-04-26T02:03:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-04-26T02:03:22.076 WARNING:tasks.quiescer.fs.[cephfs]:Set 'f112abc' hasn't quiesced after 52.4 seconds (timeout: 60). Dumping ops with locks from all ranks. 2024-04-26T02:03:22.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:03:22.568 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:22.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.13 MB/sec execute 159 sec latency 43670.228 ms 2024-04-26T02:03:22.890 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:22 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:22.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:22 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:22.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:22 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:23.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:23 smithi052 ceph-mon[27851]: pgmap v583: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.3 MiB/s wr, 36 op/s 2024-04-26T02:03:23.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 18.01 MB/sec execute 160 sec latency 44670.419 ms 2024-04-26T02:03:23.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:23 smithi191 ceph-mon[32745]: pgmap v583: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.3 MiB/s wr, 36 op/s 2024-04-26T02:03:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:23 smithi183 ceph-mon[32752]: pgmap v583: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.3 MiB/s wr, 36 op/s 2024-04-26T02:03:24.741 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.90 MB/sec execute 161 sec latency 45670.511 ms 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 -- 172.21.15.52:0/1474058748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c113c10 msgr2=0x7fb35c116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 --2- 172.21.15.52:0/1474058748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c113c10 0x7fb35c116000 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb35400b0a0 tx=0x7fb35400b3b0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 -- 172.21.15.52:0/1474058748 shutdown_connections 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 --2- 172.21.15.52:0/1474058748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c113c10 0x7fb35c116000 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 --2- 172.21.15.52:0/1474058748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c074c70 0x7fb35c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 --2- 172.21.15.52:0/1474058748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c1136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 -- 172.21.15.52:0/1474058748 >> 172.21.15.52:0/1474058748 conn(0x7fb35c079ba0 msgr2=0x7fb35c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 -- 172.21.15.52:0/1474058748 shutdown_connections 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.286+0000 7fb360e00700 1 -- 172.21.15.52:0/1474058748 wait complete. 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.287+0000 7fb360e00700 1 Processor -- start 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.287+0000 7fb360e00700 1 -- start start 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c10f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c113c10 0x7fb35c107f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c119200 con 0x7fb35c074c70 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c119080 con 0x7fb35c113c10 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb360e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c119380 con 0x7fb35c075590 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb359e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb35a800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c113c10 0x7fb35c107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb351400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c10f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb359e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34592/0 (socket says 172.21.15.52:34592) 2024-04-26T02:03:25.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.288+0000 7fb359e00700 1 -- 172.21.15.52:0/3614892214 learned_addr learned my addr 172.21.15.52:0/3614892214 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 -- 172.21.15.52:0/3614892214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c113c10 msgr2=0x7fb35c107f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c113c10 0x7fb35c107f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 -- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 msgr2=0x7fb35c10ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 -- 172.21.15.52:0/3614892214 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c108810 con 0x7fb35c075590 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb359e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb351400700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c10f170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb34400e9f0 tx=0x7fb34400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344004220 con 0x7fb35c075590 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3440043c0 con 0x7fb35c075590 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c108b00 con 0x7fb35c075590 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.289+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344005020 con 0x7fb35c075590 2024-04-26T02:03:25.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.290+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c10ca80 con 0x7fb35c075590 2024-04-26T02:03:25.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.291+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35c076360 con 0x7fb35c075590 2024-04-26T02:03:25.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.293+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb344004750 con 0x7fb35c075590 2024-04-26T02:03:25.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.294+0000 7fb353400700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 0x7fb3400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:25.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.294+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7fb3440d0830 con 0x7fb35c075590 2024-04-26T02:03:25.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.295+0000 7fb359e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 0x7fb3400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:25.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.295+0000 7fb359e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 0x7fb3400a6270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011550 tx=0x7fb34c009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:25.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.297+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb34409ce70 con 0x7fb35c075590 2024-04-26T02:03:25.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.584+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb35c069730 con 0x7fb35c075590 2024-04-26T02:03:25.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.585+0000 7fb353400700 1 -- 172.21.15.52:0/3614892214 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+11882 (secure 0 0 0) 0x7fb34409c770 con 0x7fb35c075590 2024-04-26T02:03:25.585 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:03:25.585 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":66,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:03:20.973605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27086":{"gid":27086,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2553596297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2553596297},{"type":"v1","addr":"172.21.15.183:6837","nonce":2553596297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24496,24524]},"id":1}]} 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 msgr2=0x7fb3400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 0x7fb3400a6270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011550 tx=0x7fb34c009260 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 msgr2=0x7fb35c10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c10f170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb34400e9f0 tx=0x7fb34400ed00 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 shutdown_connections 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3400a3db0 0x7fb3400a6270 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb35c113c10 0x7fb35c107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb35c074c70 0x7fb35c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 --2- 172.21.15.52:0/3614892214 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb35c075590 0x7fb35c10f170 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:25.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.589+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 >> 172.21.15.52:0/3614892214 conn(0x7fb35c079ba0 msgr2=0x7fb35c1129d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:25.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.590+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 shutdown_connections 2024-04-26T02:03:25.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:25.590+0000 7fb360e00700 1 -- 172.21.15.52:0/3614892214 wait complete. 2024-04-26T02:03:25.591 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 66 2024-04-26T02:03:25.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.79 MB/sec execute 162 sec latency 46670.610 ms 2024-04-26T02:03:25.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:25 smithi191 ceph-mon[32745]: pgmap v584: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-04-26T02:03:25.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:25 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3614892214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:25 smithi183 ceph-mon[32752]: pgmap v584: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-04-26T02:03:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:25 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3614892214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:25 smithi052 ceph-mon[27851]: pgmap v584: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-04-26T02:03:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:25 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3614892214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:26.647 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 2 (i) to a remote file /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.i.json 2024-04-26T02:03:26.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:2 ops --flags=locks --path=/var/run/ceph/ops-f112abc-1-mds.i.json 2024-04-26T02:03:26.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.68 MB/sec execute 163 sec latency 47670.723 ms 2024-04-26T02:03:27.142 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:27.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.57 MB/sec execute 164 sec latency 48670.861 ms 2024-04-26T02:03:27.868 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:27 smithi052 ceph-mon[27851]: pgmap v585: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.1 MiB/s wr, 33 op/s 2024-04-26T02:03:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:27 smithi191 ceph-mon[32745]: pgmap v585: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.1 MiB/s wr, 33 op/s 2024-04-26T02:03:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:27 smithi183 ceph-mon[32752]: pgmap v585: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 1.1 MiB/s wr, 33 op/s 2024-04-26T02:03:28.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.47 MB/sec execute 165 sec latency 49670.963 ms 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 -- 172.21.15.52:0/132185955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c076f70 msgr2=0x7f892c077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 --2- 172.21.15.52:0/132185955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c076f70 0x7f892c077350 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f891c0098f0 tx=0x7f891c009c00 comp rx=0 tx=0).stop 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 -- 172.21.15.52:0/132185955 shutdown_connections 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 --2- 172.21.15.52:0/132185955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c113900 0x7f892c115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 --2- 172.21.15.52:0/132185955 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c076f70 0x7f892c077350 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 --2- 172.21.15.52:0/132185955 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c077890 0x7f892c1133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 -- 172.21.15.52:0/132185955 >> 172.21.15.52:0/132185955 conn(0x7f892c100750 msgr2=0x7f892c102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.589+0000 7f8931e00700 1 -- 172.21.15.52:0/132185955 shutdown_connections 2024-04-26T02:03:29.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.590+0000 7f8931e00700 1 -- 172.21.15.52:0/132185955 wait complete. 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.590+0000 7f8931e00700 1 Processor -- start 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 -- start start 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 0x7f892c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c113900 0x7f892c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1192e0 con 0x7f892c113900 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f892c119160 con 0x7f892c077890 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.591+0000 7f8931e00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f892c119460 con 0x7f892c076f70 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 0x7f892c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44574/0 (socket says 172.21.15.52:44574) 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 0x7f892c1a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33184/0 (socket says 172.21.15.52:33184) 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 -- 172.21.15.52:0/2924503021 learned_addr learned my addr 172.21.15.52:0/2924503021 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892b400700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c113900 0x7f892c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 -- 172.21.15.52:0/2924503021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 msgr2=0x7f892c1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 0x7f892c1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 -- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c113900 msgr2=0x7f892c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c113900 0x7f892c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 -- 172.21.15.52:0/2924503021 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c19fd40 con 0x7f892c076f70 2024-04-26T02:03:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.592+0000 7f892aa00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f891c009230 tx=0x7f891c00fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.593+0000 7f8930a00700 1 -- 172.21.15.52:0/2924503021 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c005220 con 0x7f892c076f70 2024-04-26T02:03:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.593+0000 7f8930a00700 1 -- 172.21.15.52:0/2924503021 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f891c00b560 con 0x7f892c076f70 2024-04-26T02:03:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.593+0000 7f8930a00700 1 -- 172.21.15.52:0/2924503021 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c003a80 con 0x7f892c076f70 2024-04-26T02:03:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.593+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892c19ffd0 con 0x7f892c076f70 2024-04-26T02:03:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.593+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f892c1ac2f0 con 0x7f892c076f70 2024-04-26T02:03:29.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.596+0000 7f8930a00700 1 -- 172.21.15.52:0/2924503021 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f891c00ba10 con 0x7f892c076f70 2024-04-26T02:03:29.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f8930a00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 0x7f89100a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f8930a00700 1 -- 172.21.15.52:0/2924503021 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f891c0d17e0 con 0x7f892c076f70 2024-04-26T02:03:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f893c726b80 1 Processor -- start 2024-04-26T02:03:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f892a000700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 0x7f89100a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f893c726b80 1 -- start start 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.598+0000 7f893c726b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f893c726b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f893c726b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5dc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f893c726b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce521f60 con 0x55d7ce5ced10 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f893c726b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce4c8800 con 0x55d7ce5cf630 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f893c726b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce51df20 con 0x55d7ce5da4a0 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 0x7f89100a6270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f892c079a70 tx=0x7f8918009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34620/0 (socket says 172.21.15.52:34620) 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892aa00700 1 -- 172.21.15.52:0/737958073 learned_addr learned my addr 172.21.15.52:0/737958073 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33186/0 (socket says 172.21.15.52:33186) 2024-04-26T02:03:29.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892b400700 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5dc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 -- 172.21.15.52:0/737958073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 msgr2=0x55d7ce5dc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5dc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 -- 172.21.15.52:0/737958073 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 msgr2=0x55d7ce5cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892a000700 1 -- 172.21.15.52:0/737958073 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ce4be750 con 0x55d7ce5cf630 2024-04-26T02:03:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.599+0000 7f892aa00700 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.600+0000 7f892a000700 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8918005fd0 tx=0x7f8918014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efbd440f268d2162 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.600+0000 7f8920c00700 1 -- 172.21.15.52:0/737958073 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918003a00 con 0x55d7ce5cf630 2024-04-26T02:03:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.600+0000 7f8920c00700 1 -- 172.21.15.52:0/737958073 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8918003ba0 con 0x55d7ce5cf630 2024-04-26T02:03:29.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.600+0000 7f8920c00700 1 -- 172.21.15.52:0/737958073 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918026660 con 0x55d7ce5cf630 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 -- 172.21.15.52:0/737958073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 msgr2=0x55d7ce5d9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8918005fd0 tx=0x7f8918014000 comp rx=0 tx=0).stop 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 -- 172.21.15.52:0/737958073 shutdown_connections 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5dc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 --2- 172.21.15.52:0/737958073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d9f60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 -- 172.21.15.52:0/737958073 >> 172.21.15.52:0/737958073 conn(0x55d7ce4fe020 msgr2=0x55d7ce5cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 -- 172.21.15.52:0/737958073 shutdown_connections 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.601+0000 7f893c726b80 1 -- 172.21.15.52:0/737958073 wait complete. 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.602+0000 7f893c726b80 1 -- start start 2024-04-26T02:03:29.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.602+0000 7f893c726b80 1 Processor -- start 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5d7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce5dcf30 con 0x55d7ce5ced10 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce4db930 con 0x55d7ce5cf630 2024-04-26T02:03:29.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d7ce5dd0b0 con 0x55d7ce5da4a0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f893c726b80 10 client.0 _ll_register_callbacks cb 0x55d7ce4ba7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892a000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33192/0 (socket says 172.21.15.52:33192) 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892a000700 1 -- 172.21.15.52:0/2748599095 learned_addr learned my addr 172.21.15.52:0/2748599095 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34626/0 (socket says 172.21.15.52:34626) 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.603+0000 7f892b400700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5d7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 -- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 msgr2=0x55d7ce5d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5d7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 -- 172.21.15.52:0/2748599095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 msgr2=0x55d7ce5d9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 -- 172.21.15.52:0/2748599095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ce46f9a0 con 0x55d7ce5cf630 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892b400700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5d7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f892a000700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f891801edb0 tx=0x7f8918003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cfe28cb105a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f8915600700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918034060 con 0x55d7ce5cf630 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f891802f440 con 0x55d7ce5cf630 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.604+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8918003ba0 con 0x55d7ce5cf630 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f891802f680 con 0x55d7ce5cf630 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 10 client.27197 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55d7ce4b8bc0 con 0x55d7ce5cf630 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8918003ba0 con 0x55d7ce5cf630 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7f89180267e0 con 0x55d7ce5cf630 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 10 client.27197 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 10 client.27197 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 20 client.27197 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce5e9ba0 0x55d7ce5ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 4 client.27197 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.605+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7ce52f750 con 0x55d7ce5e9ba0 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.606+0000 7f892aa00700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce5e9ba0 0x55d7ce5ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.606+0000 7f892aa00700 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce5e9ba0 0x55d7ce5ebf90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b964d3a8af777536 server_cookie=a5556164fb878cfa in_seq=0 out_seq=0 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.606+0000 7f8915600700 10 client.27197 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.607+0000 7f8915600700 1 -- 172.21.15.52:0/2748599095 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7ce52f750 con 0x55d7ce5e9ba0 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.607+0000 7f8915600700 10 client.27197 handle_command_reply: tid=0 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.607+0000 7f893c726b80 1 client.27197 shutdown 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.607+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 msgr2=0x55d7ce5d6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.607+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f891801edb0 tx=0x7f8918003e80 comp rx=0 tx=0).stop 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 shutdown_connections 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce5e9ba0 0x55d7ce5ebf90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5d7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f8915600700 0 client.27197 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T02:03:29.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 --2- 172.21.15.52:0/2748599095 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5d6ed0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 >> 172.21.15.52:0/2748599095 conn(0x55d7ce4fe020 msgr2=0x55d7ce5cae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 shutdown_connections 2024-04-26T02:03:29.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 1 -- 172.21.15.52:0/2748599095 wait complete. 2024-04-26T02:03:29.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.608+0000 7f893c726b80 20 client.27197 trim_cache size 0 max 16384 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 Processor -- start 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 -- start start 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b61a0 con 0x55d7ce5ced10 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b3a30 con 0x55d7ce5cf630 2024-04-26T02:03:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f893c726b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b18a0 con 0x55d7ce5da4a0 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f892aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34636/0 (socket says 172.21.15.52:34636) 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.629+0000 7f892aa00700 1 -- 172.21.15.52:0/539536157 learned_addr learned my addr 172.21.15.52:0/539536157 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892a000700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 -- 172.21.15.52:0/539536157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 msgr2=0x55d7ce5eefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5eefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 -- 172.21.15.52:0/539536157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 msgr2=0x55d7ce5d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892b400700 1 -- 172.21.15.52:0/539536157 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ce5dcf80 con 0x55d7ce5da4a0 2024-04-26T02:03:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892aa00700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.630+0000 7f892a000700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5eefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.631+0000 7f892b400700 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5ef4f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55d7ce5d06e0 tx=0x7f892400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac290780c2cf0da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.631+0000 7f8917e00700 1 -- 172.21.15.52:0/539536157 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924003c00 con 0x55d7ce5da4a0 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.631+0000 7f8917e00700 1 -- 172.21.15.52:0/539536157 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8924003da0 con 0x55d7ce5da4a0 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.631+0000 7f8917e00700 1 -- 172.21.15.52:0/539536157 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924016990 con 0x55d7ce5da4a0 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 -- 172.21.15.52:0/539536157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 msgr2=0x55d7ce5ef4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5ef4f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55d7ce5d06e0 tx=0x7f892400b040 comp rx=0 tx=0).stop 2024-04-26T02:03:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 -- 172.21.15.52:0/539536157 shutdown_connections 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5da4a0 0x55d7ce5ef4f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5ced10 0x55d7ce5d25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 --2- 172.21.15.52:0/539536157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5cf630 0x55d7ce5eefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 -- 172.21.15.52:0/539536157 >> 172.21.15.52:0/539536157 conn(0x55d7ce4fe020 msgr2=0x55d7ce5d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 -- 172.21.15.52:0/539536157 shutdown_connections 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.632+0000 7f893c726b80 1 -- 172.21.15.52:0/539536157 wait complete. 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.633+0000 7f893c726b80 1 -- start start 2024-04-26T02:03:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.633+0000 7f893c726b80 1 Processor -- start 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 0x55d7ce5ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5da4a0 0x55d7ce64fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b1ef0 con 0x55d7ce5da4a0 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b3360 con 0x55d7ce5ced10 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f892c1b25c0 con 0x55d7ce5cf630 2024-04-26T02:03:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f893c726b80 10 client.0 _ll_register_callbacks cb 0x55d7ce4ba7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33206/0 (socket says 172.21.15.52:33206) 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892aa00700 1 -- 172.21.15.52:0/2030246366 learned_addr learned my addr 172.21.15.52:0/2030246366 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892a000700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 0x55d7ce5ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892b400700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5da4a0 0x55d7ce64fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892aa00700 1 -- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 msgr2=0x55d7ce5ec4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.634+0000 7f892aa00700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 0x55d7ce5ec4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f892aa00700 1 -- 172.21.15.52:0/2030246366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5da4a0 msgr2=0x55d7ce64fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f892aa00700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5da4a0 0x55d7ce64fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f892aa00700 1 -- 172.21.15.52:0/2030246366 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ce5cfcb0 con 0x55d7ce5ced10 2024-04-26T02:03:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f892a000700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 0x55d7ce5ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f892aa00700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f891c088010 tx=0x7f891c0a4040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=683ba18ca4e9cea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f8915600700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c0bb060 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8924003da0 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f891c00b710 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f891c0a1b60 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f893c726b80 10 client.27203 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.635+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55d7ce6513c0 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8924003da0 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7f891c099e20 con 0x55d7ce5ced10 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 10 client.27203 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:29.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 10 client.27203 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 20 client.27203 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce651a00 0x55d7ce653e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 4 client.27203 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.i.json"}] 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 --> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.i.json"}) -- 0x55d7ce5297f0 con 0x55d7ce651a00 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.636+0000 7f892a000700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce651a00 0x55d7ce653e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.637+0000 7f892a000700 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce651a00 0x55d7ce653e20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=92c031cac50ba7c1 server_cookie=41ce6662a40f4a12 in_seq=0 out_seq=0 2024-04-26T02:03:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.637+0000 7f8915600700 10 client.27203 ms_handle_connect on v2:172.21.15.191:6838/1544004799 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f8915600700 1 -- 172.21.15.52:0/2030246366 <== mds.2 v2:172.21.15.191:6838/1544004799 1 ==== command_reply(tid 0: 0 ) ==== 8+0+57 (crc 0 0 0) 0x55d7ce5297f0 con 0x55d7ce651a00 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f8915600700 10 client.27203 handle_command_reply: tid=0 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 client.27203 shutdown 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 msgr2=0x55d7ce5ebfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f891c088010 tx=0x7f891c0a4040 comp rx=0 tx=0).stop 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 shutdown_connections 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:6838/1544004799,v1:172.21.15.191:6839/1544004799] conn(0x55d7ce651a00 0x55d7ce653e20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d7ce5da4a0 0x55d7ce64fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55d7ce5ced10 0x55d7ce5ebfb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 --2- 172.21.15.52:0/2030246366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d7ce5cf630 0x55d7ce5ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f8915600700 0 client.27203 ms_handle_reset on v2:172.21.15.191:6838/1544004799 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 >> 172.21.15.52:0/2030246366 conn(0x55d7ce4fe020 msgr2=0x55d7ce5d8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 shutdown_connections 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.661+0000 7f893c726b80 1 -- 172.21.15.52:0/2030246366 wait complete. 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.662+0000 7f893c726b80 20 client.27203 trim_cache size 0 max 16384 2024-04-26T02:03:29.661 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:03:29.662 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/var/run/ceph/ops-f112abc-1-mds.i.json" 2024-04-26T02:03:29.662 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:03:29.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:29 smithi052 ceph-mon[27851]: pgmap v586: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 msgr2=0x7f89100a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 0x7f89100a6270 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f892c079a70 tx=0x7f8918009310 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 msgr2=0x7f892c1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f891c009230 tx=0x7f891c00fb20 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 shutdown_connections 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f89100a3db0 0x7f89100a6270 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f892c113900 0x7f892c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f892c076f70 0x7f892c1a5420 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 --2- 172.21.15.52:0/2924503021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f892c077890 0x7f892c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.663+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 >> 172.21.15.52:0/2924503021 conn(0x7f892c100750 msgr2=0x7f892c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.664+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 shutdown_connections 2024-04-26T02:03:29.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:29.664+0000 7f8931e00700 1 -- 172.21.15.52:0/2924503021 wait complete. 2024-04-26T02:03:29.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.36 MB/sec execute 166 sec latency 50671.100 ms 2024-04-26T02:03:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:29 smithi191 ceph-mon[32745]: pgmap v586: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:29 smithi183 ceph-mon[32752]: pgmap v586: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:30.537 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 1 (a) to a remote file /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.a.json 2024-04-26T02:03:30.538 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:1 ops --flags=locks --path=/var/run/ceph/ops-f112abc-1-mds.a.json 2024-04-26T02:03:30.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.26 MB/sec execute 167 sec latency 51671.229 ms 2024-04-26T02:03:31.032 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:31.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.16 MB/sec execute 168 sec latency 52671.343 ms 2024-04-26T02:03:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:31 smithi191 ceph-mon[32745]: pgmap v587: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:31 smithi183 ceph-mon[32752]: pgmap v587: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:31 smithi052 ceph-mon[27851]: pgmap v587: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-04-26T02:03:32.742 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 17.05 MB/sec execute 169 sec latency 53671.493 ms 2024-04-26T02:03:33.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 -- 172.21.15.52:0/2782896511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e41092d0 msgr2=0x7ff5e41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 --2- 172.21.15.52:0/2782896511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e41092d0 0x7ff5e41096b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009990 tx=0x7ff5d4009ca0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 -- 172.21.15.52:0/2782896511 shutdown_connections 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 --2- 172.21.15.52:0/2782896511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e410e3b0 0x7ff5e4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 --2- 172.21.15.52:0/2782896511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e41092d0 0x7ff5e41096b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 --2- 172.21.15.52:0/2782896511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e4109c80 0x7ff5e410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 -- 172.21.15.52:0/2782896511 >> 172.21.15.52:0/2782896511 conn(0x7ff5e4076210 msgr2=0x7ff5e4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 -- 172.21.15.52:0/2782896511 shutdown_connections 2024-04-26T02:03:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 -- 172.21.15.52:0/2782896511 wait complete. 2024-04-26T02:03:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.203+0000 7ff5e9400700 1 Processor -- start 2024-04-26T02:03:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 -- start start 2024-04-26T02:03:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 0x7ff5e41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e4109c80 0x7ff5e41a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 0x7ff5e419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4119290 con 0x7ff5e41092d0 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4119110 con 0x7ff5e410e3b0 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.204+0000 7ff5e9400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4119410 con 0x7ff5e4109c80 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 0x7ff5e41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 0x7ff5e41a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34646/0 (socket says 172.21.15.52:34646) 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e2800700 1 -- 172.21.15.52:0/3356029170 learned_addr learned my addr 172.21.15.52:0/3356029170 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 0x7ff5e419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e4109c80 0x7ff5e41a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 -- 172.21.15.52:0/3356029170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e4109c80 msgr2=0x7ff5e41a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e4109c80 0x7ff5e41a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 -- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 msgr2=0x7ff5e41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 0x7ff5e41a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 -- 172.21.15.52:0/3356029170 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e419fd10 con 0x7ff5e410e3b0 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.205+0000 7ff5e3200700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 0x7ff5e419f5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00efc0 tx=0x7ff5dc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.206+0000 7ff5db400700 1 -- 172.21.15.52:0/3356029170 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0092e0 con 0x7ff5e410e3b0 2024-04-26T02:03:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.206+0000 7ff5db400700 1 -- 172.21.15.52:0/3356029170 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc004bb0 con 0x7ff5e410e3b0 2024-04-26T02:03:33.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.206+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e419ffa0 con 0x7ff5e410e3b0 2024-04-26T02:03:33.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.206+0000 7ff5db400700 1 -- 172.21.15.52:0/3356029170 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc004020 con 0x7ff5e410e3b0 2024-04-26T02:03:33.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.206+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e41ac2c0 con 0x7ff5e410e3b0 2024-04-26T02:03:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.209+0000 7ff5db400700 1 -- 172.21.15.52:0/3356029170 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff5dc01b030 con 0x7ff5e410e3b0 2024-04-26T02:03:33.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 Processor -- start 2024-04-26T02:03:33.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 -- start start 2024-04-26T02:03:33.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5db400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 0x7ff5c80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5db400700 1 -- 172.21.15.52:0/3356029170 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7ff5dc0d01b0 con 0x7ff5e410e3b0 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d1ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d27b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 0x560a55d2a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 0x7ff5c80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560a55c6ff60 con 0x560a55d1d210 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560a55c16800 con 0x560a55d1c8f0 2024-04-26T02:03:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.212+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560a55c6bf20 con 0x560a55d280c0 2024-04-26T02:03:33.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d1ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d27b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d1ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33228/0 (socket says 172.21.15.52:33228) 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e2800700 1 -- 172.21.15.52:0/1684852215 learned_addr learned my addr 172.21.15.52:0/1684852215 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 0x560a55d2a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 -- 172.21.15.52:0/1684852215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 msgr2=0x560a55d1ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d1ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 -- 172.21.15.52:0/1684852215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 msgr2=0x560a55d27b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d27b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e3200700 1 -- 172.21.15.52:0/1684852215 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a55c0c750 con 0x560a55d280c0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.213+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 0x7ff5c80a62f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4078290 tx=0x7ff5d0013900 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.214+0000 7ff5e3200700 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 0x560a55d2a4b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0cd900 tx=0x7ff5dc09bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f22975b1374a913 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.214+0000 7ff5cfe00700 1 -- 172.21.15.52:0/1684852215 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc09c8e0 con 0x560a55d280c0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.214+0000 7ff5cfe00700 1 -- 172.21.15.52:0/1684852215 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc09c1e0 con 0x560a55d280c0 2024-04-26T02:03:33.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.214+0000 7ff5cfe00700 1 -- 172.21.15.52:0/1684852215 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc09c640 con 0x560a55d280c0 2024-04-26T02:03:33.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/1684852215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 msgr2=0x560a55d2a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 0x560a55d2a4b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0cd900 tx=0x7ff5dc09bb40 comp rx=0 tx=0).stop 2024-04-26T02:03:33.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/1684852215 shutdown_connections 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d280c0 0x560a55d2a4b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d1ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/1684852215 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d27b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/1684852215 >> 172.21.15.52:0/1684852215 conn(0x560a55c4b670 msgr2=0x560a55d18da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/1684852215 shutdown_connections 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/1684852215 wait complete. 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.215+0000 7ff5f3d6ab80 1 -- start start 2024-04-26T02:03:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 Processor -- start 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d24b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d25080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560a55d2abe0 con 0x560a55d280c0 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x560a55c29930 con 0x560a55d1c8f0 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560a55d2ad60 con 0x560a55d1d210 2024-04-26T02:03:33.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.216+0000 7ff5f3d6ab80 10 client.0 _ll_register_callbacks cb 0x560a55c087f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d25080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e2800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33232/0 (socket says 172.21.15.52:33232) 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e2800700 1 -- 172.21.15.52:0/2646684394 learned_addr learned my addr 172.21.15.52:0/2646684394 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 -- 172.21.15.52:0/2646684394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 msgr2=0x560a55d24b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d24b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 -- 172.21.15.52:0/2646684394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 msgr2=0x560a55d27840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e3200700 1 -- 172.21.15.52:0/2646684394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a55bbd9a0 con 0x560a55d280c0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d24b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.217+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5e3200700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d25080 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0090a0 tx=0x7ff5dc09c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958e4b048b8761c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5ccc00700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0b9020 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7ff5dc0a4b40 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5dc09c1e0 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7ff5dc0a0590 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5f3d6ab80 10 client.17064 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.218+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x560a55c06bc0 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc09c1e0 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7ff5dc09b1c0 con 0x560a55d280c0 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 10 client.17064 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 10 client.17064 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 20 client.17064 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d37790 0x560a55d39b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 4 client.17064 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a55c7d750 con 0x560a55d37790 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.219+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d37790 0x560a55d39b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.220+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d37790 0x560a55d39b80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98dfc629b3192e7f server_cookie=2d8bc85fb1f17c1f in_seq=0 out_seq=0 2024-04-26T02:03:33.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.220+0000 7ff5ccc00700 10 client.17064 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2646684394 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a55c7d750 con 0x560a55d37790 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5ccc00700 10 client.17064 handle_command_reply: tid=0 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 client.17064 shutdown 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 msgr2=0x560a55d25080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d25080 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0090a0 tx=0x7ff5dc09c670 comp rx=0 tx=0).stop 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 shutdown_connections 2024-04-26T02:03:33.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d37790 0x560a55d39b80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d25080 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2646684394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d24b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 >> 172.21.15.52:0/2646684394 conn(0x560a55c4b670 msgr2=0x560a55d18990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5ccc00700 0 client.17064 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 shutdown_connections 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.221+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2646684394 wait complete. 2024-04-26T02:03:33.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.222+0000 7ff5f3d6ab80 20 client.17064 trim_cache size 0 max 16384 2024-04-26T02:03:33.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.242+0000 7ff5f3d6ab80 1 Processor -- start 2024-04-26T02:03:33.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 -- start start 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1c8f0 0x560a55d21e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d280c0 0x560a55d294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b61e0 con 0x560a55d1d210 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b3a70 con 0x560a55d280c0 2024-04-26T02:03:33.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b18e0 con 0x560a55d1c8f0 2024-04-26T02:03:33.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34668/0 (socket says 172.21.15.52:34668) 2024-04-26T02:03:33.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.243+0000 7ff5e1e00700 1 -- 172.21.15.52:0/3925056253 learned_addr learned my addr 172.21.15.52:0/3925056253 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:33.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e3200700 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d280c0 0x560a55d294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e2800700 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1c8f0 0x560a55d21e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 -- 172.21.15.52:0/3925056253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1c8f0 msgr2=0x560a55d21e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1c8f0 0x560a55d21e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 -- 172.21.15.52:0/3925056253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d280c0 msgr2=0x560a55d294b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d280c0 0x560a55d294b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 -- 172.21.15.52:0/3925056253 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a55d1d5f0 con 0x560a55d1d210 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00209f0 tx=0x7ff5d001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41043b1e757ab9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.244+0000 7ff5cf400700 1 -- 172.21.15.52:0/3925056253 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0004a30 con 0x560a55d1d210 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5cf400700 1 -- 172.21.15.52:0/3925056253 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d00054d0 con 0x560a55d1d210 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5cf400700 1 -- 172.21.15.52:0/3925056253 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0011970 con 0x560a55d1d210 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/3925056253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 msgr2=0x560a55d22360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00209f0 tx=0x7ff5d001f000 comp rx=0 tx=0).stop 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/3925056253 shutdown_connections 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d280c0 0x560a55d294b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1c8f0 0x560a55d21e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/3925056253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d1d210 0x560a55d22360 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/3925056253 >> 172.21.15.52:0/3925056253 conn(0x560a55c4b670 msgr2=0x560a55d33880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.245+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/3925056253 shutdown_connections 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.246+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/3925056253 wait complete. 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.246+0000 7ff5f3d6ab80 1 -- start start 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.246+0000 7ff5f3d6ab80 1 Processor -- start 2024-04-26T02:03:33.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d3ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d96940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b1f30 con 0x560a55d280c0 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b33a0 con 0x560a55d1c8f0 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41b2600 con 0x560a55d1d210 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5f3d6ab80 10 client.0 _ll_register_callbacks cb 0x560a55c087f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5e1e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44686/0 (socket says 172.21.15.52:44686) 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5e1e00700 1 -- 172.21.15.52:0/2147201532 learned_addr learned my addr 172.21.15.52:0/2147201532 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d3ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.247+0000 7ff5e3200700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d96940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 -- 172.21.15.52:0/2147201532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 msgr2=0x560a55d3ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d3ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 -- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 msgr2=0x560a55d96940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d96940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 -- 172.21.15.52:0/2147201532 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a55d96f30 con 0x560a55d1d210 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d3ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5e1e00700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0005e90 tx=0x7ff5d0011840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b809f8194cd98dd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5ccc00700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0039060 con 0x560a55d1d210 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7ff5d0011cb0 con 0x560a55d1d210 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.248+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d00054d0 con 0x560a55d1d210 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7ff5d002a480 con 0x560a55d1d210 2024-04-26T02:03:33.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 10 client.27235 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x560a55d9e2f0 con 0x560a55d1d210 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d00054d0 con 0x560a55d1d210 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7ff5d0010630 con 0x560a55d1d210 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 10 client.27235 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 10 client.27235 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 20 client.27235 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d9e9b0 0x560a55da0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.249+0000 7ff5f3d6ab80 4 client.27235 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.a.json"}] 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.250+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 --> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.a.json"}) -- 0x560a55c777f0 con 0x560a55d9e9b0 2024-04-26T02:03:33.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.250+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d9e9b0 0x560a55da0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:33.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.250+0000 7ff5e2800700 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d9e9b0 0x560a55da0dd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a92559290191df1 server_cookie=95836169f2d5cb8c in_seq=0 out_seq=0 2024-04-26T02:03:33.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.250+0000 7ff5ccc00700 10 client.27235 ms_handle_connect on v2:172.21.15.52:6840/1834450967 2024-04-26T02:03:33.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.278+0000 7ff5ccc00700 1 -- 172.21.15.52:0/2147201532 <== mds.1 v2:172.21.15.52:6840/1834450967 1 ==== command_reply(tid 0: 0 ) ==== 8+0+57 (crc 0 0 0) 0x560a55c777f0 con 0x560a55d9e9b0 2024-04-26T02:03:33.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.278+0000 7ff5ccc00700 10 client.27235 handle_command_reply: tid=0 2024-04-26T02:03:33.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.278+0000 7ff5f3d6ab80 1 client.27235 shutdown 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 msgr2=0x560a55d3b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0005e90 tx=0x7ff5d0011840 comp rx=0 tx=0).stop 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 shutdown_connections 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:6840/1834450967,v1:172.21.15.52:6841/1834450967] conn(0x560a55d9e9b0 0x560a55da0dd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560a55d280c0 0x560a55d96940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5ccc00700 0 client.27235 ms_handle_reset on v2:172.21.15.52:6840/1834450967 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x560a55d1c8f0 0x560a55d3ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 --2- 172.21.15.52:0/2147201532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a55d1d210 0x560a55d3b340 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 >> 172.21.15.52:0/2147201532 conn(0x560a55c4b670 msgr2=0x560a55d32ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 shutdown_connections 2024-04-26T02:03:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 1 -- 172.21.15.52:0/2147201532 wait complete. 2024-04-26T02:03:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.279+0000 7ff5f3d6ab80 20 client.27235 trim_cache size 0 max 16384 2024-04-26T02:03:33.279 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:03:33.279 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/var/run/ceph/ops-f112abc-1-mds.a.json" 2024-04-26T02:03:33.279 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 msgr2=0x7ff5c80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 0x7ff5c80a62f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4078290 tx=0x7ff5d0013900 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 msgr2=0x7ff5e419f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 0x7ff5e419f5b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00efc0 tx=0x7ff5dc00c3a0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 shutdown_connections 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff5c80a3e30 0x7ff5c80a62f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff5e410e3b0 0x7ff5e419f5b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5e41092d0 0x7ff5e41a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 --2- 172.21.15.52:0/3356029170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e4109c80 0x7ff5e41a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:33.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.280+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 >> 172.21.15.52:0/3356029170 conn(0x7ff5e4076210 msgr2=0x7ff5e410cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.281+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 shutdown_connections 2024-04-26T02:03:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:33.281+0000 7ff5e9400700 1 -- 172.21.15.52:0/3356029170 wait complete. 2024-04-26T02:03:33.743 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.95 MB/sec execute 170 sec latency 54671.586 ms 2024-04-26T02:03:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:33 smithi191 ceph-mon[32745]: pgmap v588: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 12 op/s 2024-04-26T02:03:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:33 smithi183 ceph-mon[32752]: pgmap v588: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 12 op/s 2024-04-26T02:03:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:33 smithi052 ceph-mon[27851]: pgmap v588: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 12 op/s 2024-04-26T02:03:34.196 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 0 (b) to a remote file /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.b.json 2024-04-26T02:03:34.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph tell mds.1:0 ops --flags=locks --path=/var/run/ceph/ops-f112abc-1-mds.b.json 2024-04-26T02:03:34.668 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:34.743 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.85 MB/sec execute 171 sec latency 55671.690 ms 2024-04-26T02:03:35.743 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.76 MB/sec execute 172 sec latency 56671.827 ms 2024-04-26T02:03:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:35 smithi191 ceph-mon[32745]: pgmap v589: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-04-26T02:03:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:35 smithi183 ceph-mon[32752]: pgmap v589: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-04-26T02:03:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:35 smithi052 ceph-mon[27851]: pgmap v589: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2024-04-26T02:03:36.743 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.66 MB/sec execute 173 sec latency 57671.959 ms 2024-04-26T02:03:36.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 -- 172.21.15.52:0/2196369771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b41034b0 msgr2=0x7f70b410f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:36.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 --2- 172.21.15.52:0/2196369771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b41034b0 0x7f70b410f920 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f70ac009f90 tx=0x7f70ac00b330 comp rx=0 tx=0).stop 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 -- 172.21.15.52:0/2196369771 shutdown_connections 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 --2- 172.21.15.52:0/2196369771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b41034b0 0x7f70b410f920 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 --2- 172.21.15.52:0/2196369771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 0x7f70b4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 --2- 172.21.15.52:0/2196369771 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b4102af0 0x7f70b4102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 -- 172.21.15.52:0/2196369771 >> 172.21.15.52:0/2196369771 conn(0x7f70b4079ba0 msgr2=0x7f70b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 -- 172.21.15.52:0/2196369771 shutdown_connections 2024-04-26T02:03:36.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.876+0000 7f70bc200700 1 -- 172.21.15.52:0/2196369771 wait complete. 2024-04-26T02:03:36.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.877+0000 7f70bc200700 1 Processor -- start 2024-04-26T02:03:36.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.877+0000 7f70bc200700 1 -- start start 2024-04-26T02:03:36.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 0x7f70b419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 0x7f70b419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 0x7f70b419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4112c90 con 0x7f70b4107940 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4112b10 con 0x7f70b41034b0 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70bc200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4112e10 con 0x7f70b4102af0 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 0x7f70b419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b9800700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 0x7f70b419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 0x7f70b419b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53650/0 (socket says 172.21.15.52:53650) 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b8e00700 1 -- 172.21.15.52:0/4170959606 learned_addr learned my addr 172.21.15.52:0/4170959606 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70ba200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 0x7f70b419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b9800700 1 -- 172.21.15.52:0/4170959606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 msgr2=0x7f70b419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b9800700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 0x7f70b419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.878+0000 7f70b9800700 1 -- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 msgr2=0x7f70b419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b9800700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 0x7f70b419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b9800700 1 -- 172.21.15.52:0/4170959606 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b41a0370 con 0x7f70b4102af0 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b9800700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 0x7f70b419b360 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f70a800e9f0 tx=0x7f70a800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b2000700 1 -- 172.21.15.52:0/4170959606 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a800c9c0 con 0x7f70b4102af0 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b2000700 1 -- 172.21.15.52:0/4170959606 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a800cb60 con 0x7f70b4102af0 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b41a0660 con 0x7f70b4102af0 2024-04-26T02:03:36.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70b2000700 1 -- 172.21.15.52:0/4170959606 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a8018680 con 0x7f70b4102af0 2024-04-26T02:03:36.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.879+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b41a7ee0 con 0x7f70b4102af0 2024-04-26T02:03:36.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.883+0000 7f70b2000700 1 -- 172.21.15.52:0/4170959606 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f70a8010430 con 0x7f70b4102af0 2024-04-26T02:03:36.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.885+0000 7f70c6c7eb80 1 Processor -- start 2024-04-26T02:03:36.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.885+0000 7f70b2000700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 0x7f709c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.885+0000 7f70c6c7eb80 1 -- start start 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.885+0000 7f70b2000700 1 -- 172.21.15.52:0/4170959606 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f70a80d0890 con 0x7f70b4102af0 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb085f60 con 0x55b6cb13e4a0 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb02c800 con 0x55b6cb132d10 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb081f20 con 0x55b6cb133630 2024-04-26T02:03:36.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70ba200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 0x7f709c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70b9800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70b9800700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53654/0 (socket says 172.21.15.52:53654) 2024-04-26T02:03:36.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70b9800700 1 -- 172.21.15.52:0/3779391719 learned_addr learned my addr 172.21.15.52:0/3779391719 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:36.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.886+0000 7f70b8e00700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70ba200700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 -- 172.21.15.52:0/3779391719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 msgr2=0x55b6cb13df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 -- 172.21.15.52:0/3779391719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 msgr2=0x55b6cb140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 -- 172.21.15.52:0/3779391719 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6cb022750 con 0x55b6cb132d10 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70ba200700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb140890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70b9800700 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a4f10 tx=0x7f70a809c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ef918370b65d861 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70ba200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 0x7f709c0a62f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55b6cb1346e0 tx=0x7f70ac009ce0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70a6a00700 1 -- 172.21.15.52:0/3779391719 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a80a5440 con 0x55b6cb132d10 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70a6a00700 1 -- 172.21.15.52:0/3779391719 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a809aad0 con 0x55b6cb132d10 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.887+0000 7f70a6a00700 1 -- 172.21.15.52:0/3779391719 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a80a5210 con 0x55b6cb132d10 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/3779391719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 msgr2=0x55b6cb1330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a4f10 tx=0x7f70a809c420 comp rx=0 tx=0).stop 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/3779391719 shutdown_connections 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb140890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb132d10 0x55b6cb1330f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/3779391719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/3779391719 >> 172.21.15.52:0/3779391719 conn(0x55b6cb062020 msgr2=0x55b6cb12f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.888+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/3779391719 shutdown_connections 2024-04-26T02:03:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.889+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/3779391719 wait complete. 2024-04-26T02:03:36.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.889+0000 7f70c6c7eb80 1 -- start start 2024-04-26T02:03:36.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.889+0000 7f70c6c7eb80 1 Processor -- start 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb13e4a0 0x55b6cb13b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb140e70 con 0x55b6cb132d10 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb03f930 con 0x55b6cb13e4a0 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b6cb140ff0 con 0x55b6cb133630 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70c6c7eb80 10 client.0 _ll_register_callbacks cb 0x55b6cb01e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70b9800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70b9800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60670/0 (socket says 172.21.15.52:60670) 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70b9800700 1 -- 172.21.15.52:0/482275314 learned_addr learned my addr 172.21.15.52:0/482275314 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.890+0000 7f70ba200700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb13e4a0 0x55b6cb13b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 -- 172.21.15.52:0/482275314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 msgr2=0x55b6cb13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 -- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb13e4a0 msgr2=0x55b6cb13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb13e4a0 0x55b6cb13b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 -- 172.21.15.52:0/482275314 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6cafd39a0 con 0x55b6cb132d10 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b8e00700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.891+0000 7f70b9800700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f70a80ce500 tx=0x7f70a80cdf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd39d027e83f908 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f709b400700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:03:36.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a80a51b0 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f70a809bc20 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a809aad0 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f70a8082050 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f70c6c7eb80 10 client.17079 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55b6cb01cbc0 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.892+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a809aad0 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7f70a8098520 con 0x55b6cb132d10 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 10 client.17079 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 20 client.17079 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb14d9d0 0x55b6cb14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 4 client.17079 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6cb093750 con 0x55b6cb14d9d0 2024-04-26T02:03:36.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.893+0000 7f70b8e00700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb14d9d0 0x55b6cb14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:36.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.894+0000 7f70b8e00700 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb14d9d0 0x55b6cb14fdc0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf5c681639fcbd8c server_cookie=89427472aece98ef in_seq=0 out_seq=0 2024-04-26T02:03:36.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:36.894+0000 7f709b400700 10 client.17079 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:37.743 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.56 MB/sec execute 174 sec latency 58672.109 ms 2024-04-26T02:03:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:37 smithi191 ceph-mon[32745]: pgmap v590: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-26T02:03:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:37 smithi183 ceph-mon[32752]: pgmap v590: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-26T02:03:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:37 smithi052 ceph-mon[27851]: pgmap v590: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-26T02:03:38.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 93944 16.47 MB/sec execute 175 sec latency 59672.261 ms 2024-04-26T02:03:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.224+0000 7f5224c00640 1 -- 172.21.15.52:0/323990817 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556debf6c320 con 0x556dec07a360 2024-04-26T02:03:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.224+0000 7f5224c00640 10 client.27193 handle_command_reply: tid=0 2024-04-26T02:03:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.224+0000 7f5244ba3740 1 client.27193 shutdown 2024-04-26T02:03:39.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.225+0000 7f709b400700 1 -- 172.21.15.52:0/482275314 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6cb093750 con 0x55b6cb14d9d0 2024-04-26T02:03:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 msgr2=0x556dec009030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f522c0bad80 tx=0x7f522c03b800 comp rx=0 tx=0).stop 2024-04-26T02:03:39.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.225+0000 7f709b400700 10 client.17079 handle_command_reply: tid=0 2024-04-26T02:03:39.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.225+0000 7f70c6c7eb80 1 client.17079 shutdown 2024-04-26T02:03:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 shutdown_connections 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x556dec07a360 0x556dec07c750 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5224c00640 0 client.27193 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x556dec003a00 0x556dec009ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556debff9140 0x556dec009570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 --2- 172.21.15.52:0/323990817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556debff8820 0x556dec009030 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 >> 172.21.15.52:0/323990817 conn(0x556debff4d30 msgr2=0x556dec000960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.227+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 shutdown_connections 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.228+0000 7f5244ba3740 1 -- 172.21.15.52:0/323990817 wait complete. 2024-04-26T02:03:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.228+0000 7f5244ba3740 20 client.27193 trim_cache size 0 max 16384 2024-04-26T02:03:39.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.229+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 msgr2=0x55b6cb13dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.229+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f70a80ce500 tx=0x7f70a80cdf20 comp rx=0 tx=0).stop 2024-04-26T02:03:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:03:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:03:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:03:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:03:39.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 shutdown_connections 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb14d9d0 0x55b6cb14fdc0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb13e4a0 0x55b6cb13b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb13dc90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/482275314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb133630 0x55b6cb13af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f709b400700 0 client.17079 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:39.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 >> 172.21.15.52:0/482275314 conn(0x55b6cb062020 msgr2=0x55b6cb12ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 msgr2=0x7f52200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 0x7f52200a6300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f52340099a0 tx=0x7f5234004a40 comp rx=0 tx=0).stop 2024-04-26T02:03:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 msgr2=0x7f523c1a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f522c005ab0 tx=0x7f522c003c10 comp rx=0 tx=0).stop 2024-04-26T02:03:39.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.230+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 shutdown_connections 2024-04-26T02:03:39.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/482275314 wait complete. 2024-04-26T02:03:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 shutdown_connections 2024-04-26T02:03:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f52200a3e40 0x7f52200a6300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f523c19a6e0 0x7f523c19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f523c1072a0 0x7f523c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 --2- 172.21.15.52:0/22993929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f523c1021d0 0x7f523c1a0620 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 >> 172.21.15.52:0/22993929 conn(0x7f523c0793e0 msgr2=0x7f523c10d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.231+0000 7f70c6c7eb80 20 client.17079 trim_cache size 0 max 16384 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.232+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 shutdown_connections 2024-04-26T02:03:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:39.233+0000 7f5241c00640 1 -- 172.21.15.52:0/22993929 wait complete. 2024-04-26T02:03:39.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.271+0000 7f70c6c7eb80 1 Processor -- start 2024-04-26T02:03:39.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 -- start start 2024-04-26T02:03:39.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 0x55b6cb152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb138230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41b1e50 con 0x55b6cb13e4a0 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41af6e0 con 0x55b6cb133630 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.272+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41ad550 con 0x55b6cb132d10 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53674/0 (socket says 172.21.15.52:53674) 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 -- 172.21.15.52:0/2574428382 learned_addr learned my addr 172.21.15.52:0/2574428382 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b9800700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 0x55b6cb152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70ba200700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 -- 172.21.15.52:0/2574428382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 msgr2=0x55b6cb152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 0x55b6cb152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 -- 172.21.15.52:0/2574428382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 msgr2=0x55b6cb138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb138230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b8e00700 1 -- 172.21.15.52:0/2574428382 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6cb133340 con 0x55b6cb133630 2024-04-26T02:03:39.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.273+0000 7f70b9800700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 0x55b6cb152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:39.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.274+0000 7f70b8e00700 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f70a00098f0 tx=0x7f70a0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc445f37e8810316 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:39.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.274+0000 7f70a6000700 1 -- 172.21.15.52:0/2574428382 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0004830 con 0x55b6cb133630 2024-04-26T02:03:39.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.274+0000 7f70a6000700 1 -- 172.21.15.52:0/2574428382 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a0005020 con 0x55b6cb133630 2024-04-26T02:03:39.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.274+0000 7f70a6000700 1 -- 172.21.15.52:0/2574428382 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a00223d0 con 0x55b6cb133630 2024-04-26T02:03:39.274 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.275+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/2574428382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 msgr2=0x55b6cb153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.275+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f70a00098f0 tx=0x7f70a0009310 comp rx=0 tx=0).stop 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/2574428382 shutdown_connections 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb13e4a0 0x55b6cb138230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb132d10 0x55b6cb152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/2574428382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb153330 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/2574428382 >> 172.21.15.52:0/2574428382 conn(0x55b6cb062020 msgr2=0x55b6cb13d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/2574428382 shutdown_connections 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.276+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/2574428382 wait complete. 2024-04-26T02:03:39.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.277+0000 7f70c6c7eb80 1 -- start start 2024-04-26T02:03:39.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.277+0000 7f70c6c7eb80 1 Processor -- start 2024-04-26T02:03:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb1abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb13e4a0 0x55b6cb1a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41adba0 con 0x55b6cb132d10 2024-04-26T02:03:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41af010 con 0x55b6cb133630 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41ae270 con 0x55b6cb13e4a0 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70c6c7eb80 10 client.0 _ll_register_callbacks cb 0x55b6cb01e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70b8e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:53690/0 (socket says 172.21.15.52:53690) 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.278+0000 7f70b8e00700 1 -- 172.21.15.52:0/1058913156 learned_addr learned my addr 172.21.15.52:0/1058913156 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.279+0000 7f70b9800700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb1abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.279+0000 7f70ba200700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb13e4a0 0x55b6cb1a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.279+0000 7f70b8e00700 1 -- 172.21.15.52:0/1058913156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb13e4a0 msgr2=0x55b6cb1a7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.279+0000 7f70b8e00700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb13e4a0 0x55b6cb1a7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.279+0000 7f70b8e00700 1 -- 172.21.15.52:0/1058913156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 msgr2=0x55b6cb1abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.280+0000 7f70b8e00700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb1abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.280+0000 7f70b8e00700 1 -- 172.21.15.52:0/1058913156 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6cb1a8380 con 0x55b6cb133630 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.280+0000 7f70b9800700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb1abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.280+0000 7f70b8e00700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f70a0000c00 tx=0x7f70a0005260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b323e4152634a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:39.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.280+0000 7f709b400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:03:39.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.281+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0005740 con 0x55b6cb133630 2024-04-26T02:03:39.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.281+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f70a0004950 con 0x55b6cb133630 2024-04-26T02:03:39.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.281+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a0005020 con 0x55b6cb133630 2024-04-26T02:03:39.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.282+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f70a0004b50 con 0x55b6cb133630 2024-04-26T02:03:39.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.282+0000 7f70c6c7eb80 10 client.27239 fetch_fsmap learned FSMap version 66 2024-04-26T02:03:39.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.282+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55b6cb1bd160 con 0x55b6cb133630 2024-04-26T02:03:39.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.283+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a0005020 con 0x55b6cb133630 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 66) ==== 8338+0+0 (secure 0 0 0) 0x7f70a0022640 con 0x55b6cb133630 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 10 client.27239 fetch_fsmap finished waiting for FSMap version 66 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 10 client.27239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 20 client.27239 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb1bd7e0 0x55b6cb1bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 4 client.27239 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.b.json"}] 2024-04-26T02:03:39.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-f112abc-1-mds.b.json"}) -- 0x55b6cb08d7f0 con 0x55b6cb1bd7e0 2024-04-26T02:03:39.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.284+0000 7f70b9800700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb1bd7e0 0x55b6cb1bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:39.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.285+0000 7f70b9800700 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb1bd7e0 0x55b6cb1bfc00 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c530b93af21fee server_cookie=67a4cdb8caea9363 in_seq=0 out_seq=0 2024-04-26T02:03:39.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.285+0000 7f709b400700 10 client.27239 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f709b400700 1 -- 172.21.15.52:0/1058913156 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+57 (crc 0 0 0) 0x55b6cb08d7f0 con 0x55b6cb1bd7e0 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f709b400700 10 client.27239 handle_command_reply: tid=0 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f70c6c7eb80 1 client.27239 shutdown 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 msgr2=0x55b6cb1ac530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f70a0000c00 tx=0x7f70a0005260 comp rx=0 tx=0).stop 2024-04-26T02:03:39.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.289+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 shutdown_connections 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55b6cb1bd7e0 0x55b6cb1bfc00 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b6cb13e4a0 0x55b6cb1a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6cb132d10 0x55b6cb1abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 --2- 172.21.15.52:0/1058913156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55b6cb133630 0x55b6cb1ac530 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 >> 172.21.15.52:0/1058913156 conn(0x55b6cb062020 msgr2=0x55b6cb149240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f709b400700 0 client.27239 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:39.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 shutdown_connections 2024-04-26T02:03:39.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 1 -- 172.21.15.52:0/1058913156 wait complete. 2024-04-26T02:03:39.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.290+0000 7f70c6c7eb80 20 client.27239 trim_cache size 0 max 16384 2024-04-26T02:03:39.290 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:03:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/var/run/ceph/ops-f112abc-1-mds.b.json" 2024-04-26T02:03:39.290 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.291+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 msgr2=0x7f709c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.291+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 0x7f709c0a62f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55b6cb1346e0 tx=0x7f70ac009ce0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.291+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 msgr2=0x7f70b419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.291+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 0x7f70b419b360 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f70a800e9f0 tx=0x7f70a800ed00 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 shutdown_connections 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f709c0a3e30 0x7f709c0a62f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70b4107940 0x7f70b419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f70b4102af0 0x7f70b419b360 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 --2- 172.21.15.52:0/4170959606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f70b41034b0 0x7f70b419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:39.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 >> 172.21.15.52:0/4170959606 conn(0x7f70b4079ba0 msgr2=0x7f70b4100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:39.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 shutdown_connections 2024-04-26T02:03:39.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:39.292+0000 7f70bc200700 1 -- 172.21.15.52:0/4170959606 wait complete. 2024-04-26T02:03:39.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 94192 16.46 MB/sec execute 176 sec latency 60050.784 ms 2024-04-26T02:03:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:39 smithi191 ceph-mon[32745]: pgmap v591: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:39.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:39 smithi183 ceph-mon[32752]: pgmap v591: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:39 smithi052 ceph-mon[27851]: pgmap v591: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:40.091 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-04-26T02:03:40.107 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.zbBvABISHq 2024-04-26T02:03:40.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.i.json /tmp/tmp.zbBvABISHq 2024-04-26T02:03:40.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.zbBvABISHq 2024-04-26T02:03:40.313 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.zbBvABISHq is 708KB 2024-04-26T02:03:40.421 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.zbBvABISHq 2024-04-26T02:03:40.439 DEBUG:tasks.quiescer.fs.[cephfs]:read 725248B of ops from '/var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.i.json' on mds.2 (i) 2024-04-26T02:03:40.482 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1199 fp=#0x10000000006' 2024-04-26T02:03:40.484 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1503 fp=#0x10000000fc8' 2024-04-26T02:03:40.484 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1298' 2024-04-26T02:03:40.484 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1313' 2024-04-26T02:03:40.486 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1504 fp=#0x10000000fb5' 2024-04-26T02:03:40.486 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1299' 2024-04-26T02:03:40.486 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1314' 2024-04-26T02:03:40.487 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1200 fp=#0x10000000007' 2024-04-26T02:03:40.487 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1352 fp=#0x1000000020f' 2024-04-26T02:03:40.488 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1246' 2024-04-26T02:03:40.489 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1300' 2024-04-26T02:03:40.489 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1315' 2024-04-26T02:03:40.491 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1247' 2024-04-26T02:03:40.491 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1301' 2024-04-26T02:03:40.491 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1316' 2024-04-26T02:03:40.493 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1248' 2024-04-26T02:03:40.493 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1302' 2024-04-26T02:03:40.494 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1317' 2024-04-26T02:03:40.495 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1249' 2024-04-26T02:03:40.495 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1303' 2024-04-26T02:03:40.496 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1318' 2024-04-26T02:03:40.497 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1250' 2024-04-26T02:03:40.498 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1304' 2024-04-26T02:03:40.498 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1328' 2024-04-26T02:03:40.500 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1251' 2024-04-26T02:03:40.500 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1305' 2024-04-26T02:03:40.500 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1329' 2024-04-26T02:03:40.502 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1252' 2024-04-26T02:03:40.502 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1306' 2024-04-26T02:03:40.502 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1330' 2024-04-26T02:03:40.503 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'client_request(client.14949:299190 unlink #0x10000000f08/HANDOUT.DOC 2024-04-26T02:02:39.648711+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:40.504 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1167' 2024-04-26T02:03:40.504 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1253' 2024-04-26T02:03:40.505 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1331' 2024-04-26T02:03:40.505 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1233 fp=#0x10000000009' 2024-04-26T02:03:40.506 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1168' 2024-04-26T02:03:40.507 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1254' 2024-04-26T02:03:40.507 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1332' 2024-04-26T02:03:40.507 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1208 fp=#0x10000000008' 2024-04-26T02:03:40.510 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1255' 2024-04-26T02:03:40.510 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1333' 2024-04-26T02:03:40.512 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1256' 2024-04-26T02:03:40.512 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1334' 2024-04-26T02:03:40.514 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1257' 2024-04-26T02:03:40.514 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1335' 2024-04-26T02:03:40.516 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1258' 2024-04-26T02:03:40.516 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1336' 2024-04-26T02:03:40.518 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1259' 2024-04-26T02:03:40.519 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1368' 2024-04-26T02:03:40.520 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1260' 2024-04-26T02:03:40.521 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_path:mds.2:1192 fp=#0x1' 2024-04-26T02:03:40.523 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1261' 2024-04-26T02:03:40.523 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1193 fp=#0x1' 2024-04-26T02:03:40.525 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1262' 2024-04-26T02:03:40.526 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'client_request(client.14949:299136 create owner_uid=1000, owner_gid=1316 #0x10000000fe9/NEWS1_1B.PRN 2024-04-26T02:02:39.402714+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:40.527 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1169' 2024-04-26T02:03:40.528 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1263' 2024-04-26T02:03:40.530 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1275' 2024-04-26T02:03:40.530 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1194 fp=#0x10000000000' 2024-04-26T02:03:40.532 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1276' 2024-04-26T02:03:40.533 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'client_request(client.14949:299086 unlink #0x10000000f3f/PPTC321.TMP 2024-04-26T02:02:39.072718+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:40.534 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1277' 2024-04-26T02:03:40.536 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1278' 2024-04-26T02:03:40.537 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1195 fp=#0x10000000001' 2024-04-26T02:03:40.539 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1279' 2024-04-26T02:03:40.541 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1280' 2024-04-26T02:03:40.541 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1196 fp=#0x100000001fa' 2024-04-26T02:03:40.544 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1281' 2024-04-26T02:03:40.546 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1282' 2024-04-26T02:03:40.548 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.1:1283' 2024-04-26T02:03:40.548 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1197 fp=#0x100000001fb' 2024-04-26T02:03:40.550 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1500 fp=#0x10000000f9c' 2024-04-26T02:03:40.550 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1275' 2024-04-26T02:03:40.551 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1310' 2024-04-26T02:03:40.552 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1501 fp=#0x10000000fab' 2024-04-26T02:03:40.553 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1276' 2024-04-26T02:03:40.553 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1311' 2024-04-26T02:03:40.553 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1198 fp=#0x10000000005' 2024-04-26T02:03:40.554 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'internal op quiesce_inode:mds.2:1502 fp=#0x10000000fd4' 2024-04-26T02:03:40.555 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1277' 2024-04-26T02:03:40.555 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 2 (i) for f112abc-1: 'peer_request:mds.0:1312' 2024-04-26T02:03:40.556 INFO:tasks.quiescer.fs.[cephfs]:Pulled 426 ops from rank 2 (i) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/ops-f112abc-1-mds.i.json 2024-04-26T02:03:40.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -- /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.i.json 2024-04-26T02:03:40.582 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-04-26T02:03:40.635 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.esxwOFoMNr 2024-04-26T02:03:40.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.a.json /tmp/tmp.esxwOFoMNr 2024-04-26T02:03:40.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.esxwOFoMNr 2024-04-26T02:03:40.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 94730 16.52 MB/sec execute 177 sec latency 122.536 ms 2024-04-26T02:03:40.932 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.esxwOFoMNr is 776KB 2024-04-26T02:03:41.042 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.esxwOFoMNr 2024-04-26T02:03:41.097 DEBUG:tasks.quiescer.fs.[cephfs]:read 794712B of ops from '/var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.a.json' on mds.1 (a) 2024-04-26T02:03:41.113 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1343' 2024-04-26T02:03:41.113 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1492' 2024-04-26T02:03:41.116 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1344' 2024-04-26T02:03:41.116 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1493' 2024-04-26T02:03:41.118 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1345' 2024-04-26T02:03:41.118 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1494' 2024-04-26T02:03:41.121 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'client_request(client.14949:299120 unlink #0x1000000103e/__52E2B.DB 2024-04-26T02:02:39.263716+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.121 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1495' 2024-04-26T02:03:41.121 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1299' 2024-04-26T02:03:41.123 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1496' 2024-04-26T02:03:41.124 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'client_request(client.14949:299197 create owner_uid=1000, owner_gid=1316 #0x10000001042/__40D6B.DB 2024-04-26T02:02:39.692710+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.126 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1497' 2024-04-26T02:03:41.131 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1229' 2024-04-26T02:03:41.134 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1547' 2024-04-26T02:03:41.134 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1230' 2024-04-26T02:03:41.136 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1259' 2024-04-26T02:03:41.137 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1337' 2024-04-26T02:03:41.141 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1294' 2024-04-26T02:03:41.141 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1338' 2024-04-26T02:03:41.143 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'client_request(client.14949:299112 unlink #0x100000010f2/SALES.PRN 2024-04-26T02:02:39.202716+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.143 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1235' 2024-04-26T02:03:41.143 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1260' 2024-04-26T02:03:41.143 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1295' 2024-04-26T02:03:41.144 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1339' 2024-04-26T02:03:41.146 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'client_request(client.14949:299154 create owner_uid=1000, owner_gid=1316 #0x10000000fd5/PMC185.TMP 2024-04-26T02:02:39.551712+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.146 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1296' 2024-04-26T02:03:41.146 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1300' 2024-04-26T02:03:41.146 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1340' 2024-04-26T02:03:41.150 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1278' 2024-04-26T02:03:41.151 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1297' 2024-04-26T02:03:41.151 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1341' 2024-04-26T02:03:41.153 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1289' 2024-04-26T02:03:41.153 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1261' 2024-04-26T02:03:41.153 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1342' 2024-04-26T02:03:41.155 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1319' 2024-04-26T02:03:41.157 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1320' 2024-04-26T02:03:41.160 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1321' 2024-04-26T02:03:41.162 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1322' 2024-04-26T02:03:41.163 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'client_request(client.14949:299169 create owner_uid=1000, owner_gid=1316 #0x10000000ff1/PMD392.TMP 2024-04-26T02:02:39.625711+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.163 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1343' 2024-04-26T02:03:41.165 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1323' 2024-04-26T02:03:41.166 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1344' 2024-04-26T02:03:41.167 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1324' 2024-04-26T02:03:41.167 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1253' 2024-04-26T02:03:41.170 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1325' 2024-04-26T02:03:41.170 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1254' 2024-04-26T02:03:41.172 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1326' 2024-04-26T02:03:41.173 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1301' 2024-04-26T02:03:41.175 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1255' 2024-04-26T02:03:41.175 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1302' 2024-04-26T02:03:41.177 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1327' 2024-04-26T02:03:41.177 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1256' 2024-04-26T02:03:41.178 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1298' 2024-04-26T02:03:41.178 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1336' 2024-04-26T02:03:41.180 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1337' 2024-04-26T02:03:41.182 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1338' 2024-04-26T02:03:41.182 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1459' 2024-04-26T02:03:41.182 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1257' 2024-04-26T02:03:41.184 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1339' 2024-04-26T02:03:41.187 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1340' 2024-04-26T02:03:41.187 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1489' 2024-04-26T02:03:41.187 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.2:1258' 2024-04-26T02:03:41.189 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1341' 2024-04-26T02:03:41.189 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1490' 2024-04-26T02:03:41.191 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1342' 2024-04-26T02:03:41.192 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 1 (a) for f112abc-1: 'peer_request:mds.0:1491' 2024-04-26T02:03:41.193 INFO:tasks.quiescer.fs.[cephfs]:Pulled 458 ops from rank 1 (a) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/ops-f112abc-1-mds.a.json 2024-04-26T02:03:41.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -- /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.a.json 2024-04-26T02:03:41.280 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-04-26T02:03:41.297 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.uqRMZbnjW8 2024-04-26T02:03:41.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.b.json /tmp/tmp.uqRMZbnjW8 2024-04-26T02:03:41.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.uqRMZbnjW8 2024-04-26T02:03:41.492 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.uqRMZbnjW8 is 81KB 2024-04-26T02:03:41.516 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.uqRMZbnjW8 2024-04-26T02:03:41.533 DEBUG:tasks.quiescer.fs.[cephfs]:read 82947B of ops from '/var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.b.json' on mds.0 (b) 2024-04-26T02:03:41.537 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for f112abc-1: 'client_request(client.14949:299344 create owner_uid=1000, owner_gid=1316 #0x10000000225/PMD383.TMP 2024-04-26T02:03:39.266929+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.539 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for f112abc-1: 'peer_request:client.14949:299345' 2024-04-26T02:03:41.539 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for f112abc-1: 'client_request(client.14949:299356 unlink #0x10000000225/PMD383.TMP 2024-04-26T02:03:39.278929+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.541 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for f112abc-1: 'client_request(client.14949:299359 create owner_uid=1000, owner_gid=1316 #0x10000000ff5/NEWS1_1B.PRN 2024-04-26T02:03:39.280929+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})' 2024-04-26T02:03:41.543 DEBUG:tasks.quiescer.fs.[cephfs]:Outstanding op at rank 0 (b) for f112abc-1: 'peer_request:client.14949:299363' 2024-04-26T02:03:41.544 INFO:tasks.quiescer.fs.[cephfs]:Pulled 5 ops from rank 0 (b) into home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/ops-f112abc-1-mds.b.json 2024-04-26T02:03:41.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -- /var/run/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ops-f112abc-1-mds.b.json 2024-04-26T02:03:41.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:03:41.640 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:41 smithi052 ceph-mon[27851]: pgmap v592: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:41.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 95204 16.55 MB/sec execute 178 sec latency 101.019 ms 2024-04-26T02:03:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:41 smithi191 ceph-mon[32745]: pgmap v592: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:41 smithi183 ceph-mon[32752]: pgmap v592: 97 pgs: 97 active+clean; 857 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:03:42.278 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:42.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 95837 16.67 MB/sec execute 179 sec latency 103.308 ms 2024-04-26T02:03:42.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:42 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:42.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:42 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:42.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:42 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:03:43.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 96341 16.72 MB/sec execute 180 sec latency 124.300 ms 2024-04-26T02:03:43.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:43 smithi183 ceph-mon[32752]: pgmap v593: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-04-26T02:03:43.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:43 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:43.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:43 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:43 smithi052 ceph-mon[27851]: pgmap v593: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-04-26T02:03:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:43 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:43 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:44.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:43 smithi191 ceph-mon[32745]: pgmap v593: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 89 op/s 2024-04-26T02:03:44.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:43 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:44.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:43 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:44.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 -- 172.21.15.52:0/1379405017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 msgr2=0x7fef84115650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:44.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 --2- 172.21.15.52:0/1379405017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef84115650 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fef700098f0 tx=0x7fef70009c00 comp rx=0 tx=0).stop 2024-04-26T02:03:44.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 -- 172.21.15.52:0/1379405017 shutdown_connections 2024-04-26T02:03:44.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 --2- 172.21.15.52:0/1379405017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84115b90 0x7fef84117f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 --2- 172.21.15.52:0/1379405017 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84109a40 0x7fef84109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 --2- 172.21.15.52:0/1379405017 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef84115650 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 -- 172.21.15.52:0/1379405017 >> 172.21.15.52:0/1379405017 conn(0x7fef8406df90 msgr2=0x7fef8406e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 -- 172.21.15.52:0/1379405017 shutdown_connections 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.628+0000 7fef8aa00700 1 -- 172.21.15.52:0/1379405017 wait complete. 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.629+0000 7fef8aa00700 1 Processor -- start 2024-04-26T02:03:44.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 -- start start 2024-04-26T02:03:44.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:44.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef841ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:44.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84115b90 0x7fef841148b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fef8411b410 con 0x7fef8410a360 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fef8411b290 con 0x7fef84109a40 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef8aa00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef8411b590 con 0x7fef84115b90 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef88a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84115b90 0x7fef841148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef83e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35482/0 (socket says 172.21.15.52:35482) 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.630+0000 7fef83e00700 1 -- 172.21.15.52:0/2090898440 learned_addr learned my addr 172.21.15.52:0/2090898440 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:44.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.631+0000 7fef83400700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef841ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:44.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.631+0000 7fef83e00700 1 -- 172.21.15.52:0/2090898440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84115b90 msgr2=0x7fef841148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:44.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.631+0000 7fef83e00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84115b90 0x7fef841148b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:44.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.631+0000 7fef83e00700 1 -- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 msgr2=0x7fef841ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:44.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.632+0000 7fef83e00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef841ad0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:44.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.632+0000 7fef83e00700 1 -- 172.21.15.52:0/2090898440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef84115180 con 0x7fef84109a40 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.632+0000 7fef83e00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fef7400e9a0 tx=0x7fef7400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.633+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7400ca10 con 0x7fef84109a40 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.633+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef84115470 con 0x7fef84109a40 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.633+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7400cbb0 con 0x7fef84109a40 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.633+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74018680 con 0x7fef84109a40 2024-04-26T02:03:44.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.633+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef841b4c70 con 0x7fef84109a40 2024-04-26T02:03:44.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.635+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8406f110 con 0x7fef84109a40 2024-04-26T02:03:44.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.637+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fef74010430 con 0x7fef84109a40 2024-04-26T02:03:44.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.638+0000 7fef80c00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 0x7fef680a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:44.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.638+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7fef740d09a0 con 0x7fef84109a40 2024-04-26T02:03:44.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.639+0000 7fef83400700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 0x7fef680a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:44.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.641+0000 7fef83400700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 0x7fef680a62f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef70009fd0 tx=0x7fef70019040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:44.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:44.645+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef7409cfe0 con 0x7fef84109a40 2024-04-26T02:03:44.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:44 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:44.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 96907 16.77 MB/sec execute 181 sec latency 140.462 ms 2024-04-26T02:03:44.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:44 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:45.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.141+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef841a92f0 con 0x7fef84109a40 2024-04-26T02:03:45.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.142+0000 7fef80c00700 1 -- 172.21.15.52:0/2090898440 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+11883 (secure 0 0 0) 0x7fef7409c8e0 con 0x7fef84109a40 2024-04-26T02:03:45.141 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:03:45.142 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":67,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:03:42.844960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27086":{"gid":27086,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/2553596297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":2553596297},{"type":"v1","addr":"172.21.15.183:6837","nonce":2553596297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T02:03:45.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 msgr2=0x7fef680a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 0x7fef680a62f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef70009fd0 tx=0x7fef70019040 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 msgr2=0x7fef841acb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fef7400e9a0 tx=0x7fef7400ecb0 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 shutdown_connections 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fef680a3e30 0x7fef680a62f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef84115b90 0x7fef841148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fef84109a40 0x7fef841acb70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 --2- 172.21.15.52:0/2090898440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef8410a360 0x7fef841ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:45.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 >> 172.21.15.52:0/2090898440 conn(0x7fef8406df90 msgr2=0x7fef8410b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:45.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.146+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 shutdown_connections 2024-04-26T02:03:45.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:45.147+0000 7fef8aa00700 1 -- 172.21.15.52:0/2090898440 wait complete. 2024-04-26T02:03:45.148 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 67 2024-04-26T02:03:45.152 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:44 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:45.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 97552 16.91 MB/sec execute 182 sec latency 74.163 ms 2024-04-26T02:03:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:45 smithi183 ceph-mon[32752]: pgmap v594: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-26T02:03:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:45 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:45 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:45.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:45 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2090898440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:45.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:45 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:45 smithi052 ceph-mon[27851]: pgmap v594: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-26T02:03:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:45 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:45 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:45 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2090898440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:45 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:46.055 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --set-id f112abc --await-for 42.0' 2024-04-26T02:03:46.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --set-id f112abc --await-for 42.0 2024-04-26T02:03:46.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:45 smithi191 ceph-mon[32745]: pgmap v594: 97 pgs: 97 active+clean; 856 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-26T02:03:46.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:45 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:03:46.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:45 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:03:46.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:45 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2090898440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:03:46.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:45 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:03:46.697 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:03:46.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 98082 16.98 MB/sec execute 183 sec latency 133.510 ms 2024-04-26T02:03:46.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:46 smithi183 ceph-mon[32752]: pgmap v595: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-04-26T02:03:46.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:46 smithi052 ceph-mon[27851]: pgmap v595: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-04-26T02:03:47.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:46 smithi191 ceph-mon[32745]: pgmap v595: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-04-26T02:03:47.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 98619 17.03 MB/sec execute 184 sec latency 82.146 ms 2024-04-26T02:03:48.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 99190 17.09 MB/sec execute 185 sec latency 73.593 ms 2024-04-26T02:03:49.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:48 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:49.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:48 smithi052 ceph-mon[27851]: pgmap v596: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 -- 172.21.15.52:0/2351779449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30107940 msgr2=0x7fbe30107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 --2- 172.21.15.52:0/2351779449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30107940 0x7fbe30107d20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbe200098f0 tx=0x7fbe20009c00 comp rx=0 tx=0).stop 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 -- 172.21.15.52:0/2351779449 shutdown_connections 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 --2- 172.21.15.52:0/2351779449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe301034b0 0x7fbe3010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 --2- 172.21.15.52:0/2351779449 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30107940 0x7fbe30107d20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 --2- 172.21.15.52:0/2351779449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe30102af0 0x7fbe30102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.014+0000 7fbe36a00700 1 -- 172.21.15.52:0/2351779449 >> 172.21.15.52:0/2351779449 conn(0x7fbe30079ba0 msgr2=0x7fbe30079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.015+0000 7fbe36a00700 1 -- 172.21.15.52:0/2351779449 shutdown_connections 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.015+0000 7fbe36a00700 1 -- 172.21.15.52:0/2351779449 wait complete. 2024-04-26T02:03:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.016+0000 7fbe36a00700 1 Processor -- start 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.016+0000 7fbe36a00700 1 -- start start 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30102af0 0x7fbe3019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe301034b0 0x7fbe3019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe30107940 0x7fbe3019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30112c90 con 0x7fbe301034b0 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30112b10 con 0x7fbe30107940 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe36a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30112e10 con 0x7fbe30102af0 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe34a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe30107940 0x7fbe3019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30102af0 0x7fbe3019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe34a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe30107940 0x7fbe3019fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35508/0 (socket says 172.21.15.52:35508) 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe34a00700 1 -- 172.21.15.52:0/1583231241 learned_addr learned my addr 172.21.15.52:0/1583231241 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.017+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30102af0 0x7fbe3019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33298/0 (socket says 172.21.15.52:33298) 2024-04-26T02:03:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe2f400700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe301034b0 0x7fbe3019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 -- 172.21.15.52:0/1583231241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30102af0 msgr2=0x7fbe3019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbe30102af0 0x7fbe3019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 -- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe301034b0 msgr2=0x7fbe3019b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbe301034b0 0x7fbe3019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 -- 172.21.15.52:0/1583231241 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe301a0410 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe34a00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe30107940 0x7fbe3019fcd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800efc0 tx=0x7fbe2800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe2cc00700 1 -- 172.21.15.52:0/1583231241 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280092e0 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe2cc00700 1 -- 172.21.15.52:0/1583231241 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28004bb0 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.018+0000 7fbe2cc00700 1 -- 172.21.15.52:0/1583231241 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28004040 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.019+0000 7fbe36a00700 1 -- 172.21.15.52:0/1583231241 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301a06a0 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.019+0000 7fbe36a00700 1 -- 172.21.15.52:0/1583231241 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe30109840 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.024+0000 7fbe2cc00700 1 -- 172.21.15.52:0/1583231241 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbe2801b030 con 0x7fbe30107940 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.026+0000 7fbe41365b80 1 Processor -- start 2024-04-26T02:03:49.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.026+0000 7fbe41365b80 1 -- start start 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.027+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 0x555f141c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c10f0 0x555f141c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe41365b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555f14108f90 con 0x555f141b5920 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe41365b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555f140af830 con 0x555f141c10f0 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe41365b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555f14104f50 con 0x555f141b6240 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe2cc00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbe140a3e30 0x7fbe140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe2f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe2f400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141b5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56326/0 (socket says 172.21.15.52:56326) 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe2f400700 1 -- 172.21.15.52:0/3095277296 learned_addr learned my addr 172.21.15.52:0/3095277296 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.029+0000 7fbe2cc00700 1 -- 172.21.15.52:0/1583231241 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7fbe280d01f0 con 0x7fbe30107940 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.028+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 0x555f141c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.029+0000 7fbe34a00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbe140a3e30 0x7fbe140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.030+0000 7fbe2fe00700 1 -- 172.21.15.52:0/3095277296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c10f0 msgr2=0x555f141c34e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.030+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c10f0 0x555f141c34e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.030+0000 7fbe2fe00700 1 -- 172.21.15.52:0/3095277296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 msgr2=0x555f141b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.030+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.030+0000 7fbe2fe00700 1 -- 172.21.15.52:0/3095277296 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f140a5780 con 0x555f141b6240 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.031+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 0x555f141c0bb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbe200098c0 tx=0x7fbe20009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=983c13f43f9649e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.031+0000 7fbe34a00700 1 --2- 172.21.15.52:0/1583231241 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbe140a3e30 0x7fbe140a62f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe2809c110 tx=0x7fbe28004ad0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.031+0000 7fbe25600700 1 -- 172.21.15.52:0/3095277296 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20027070 con 0x555f141b6240 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.031+0000 7fbe25600700 1 -- 172.21.15.52:0/3095277296 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe200054f0 con 0x555f141b6240 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.031+0000 7fbe25600700 1 -- 172.21.15.52:0/3095277296 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20004c00 con 0x555f141b6240 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.032+0000 7fbe41365b80 1 -- 172.21.15.52:0/3095277296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 msgr2=0x555f141c0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.032+0000 7fbe41365b80 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 0x555f141c0bb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbe200098c0 tx=0x7fbe20009610 comp rx=0 tx=0).stop 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 -- 172.21.15.52:0/3095277296 shutdown_connections 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c10f0 0x555f141c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 --2- 172.21.15.52:0/3095277296 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b6240 0x555f141c0bb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 -- 172.21.15.52:0/3095277296 >> 172.21.15.52:0/3095277296 conn(0x555f140e46a0 msgr2=0x555f141b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 -- 172.21.15.52:0/3095277296 shutdown_connections 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.033+0000 7fbe41365b80 1 -- 172.21.15.52:0/3095277296 wait complete. 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.034+0000 7fbe41365b80 1 -- start start 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.035+0000 7fbe41365b80 1 Processor -- start 2024-04-26T02:03:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b6240 0x555f141bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141c10f0 0x555f141be090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555f141c3bb0 con 0x555f141c10f0 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555f140c2960 con 0x555f141b6240 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555f141c3d30 con 0x555f141b5920 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe41365b80 10 client.0 _ll_register_callbacks cb 0x555f140a1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe2f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe2f400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33312/0 (socket says 172.21.15.52:33312) 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe2f400700 1 -- 172.21.15.52:0/2133351714 learned_addr learned my addr 172.21.15.52:0/2133351714 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.036+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b6240 0x555f141bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141c10f0 0x555f141be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 -- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b6240 msgr2=0x555f141bdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b6240 0x555f141bdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 -- 172.21.15.52:0/2133351714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141c10f0 msgr2=0x555f141be090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141c10f0 0x555f141be090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.037+0000 7fbe2f400700 1 -- 172.21.15.52:0/2133351714 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f140569d0 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.038+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009e50 tx=0x7fbe1c009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97050a61bbc78cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.038+0000 7fbe1a000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c011b80 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7fbe1c011d20 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe200054f0 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fbe1c004020 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe41365b80 10 client.27271 fetch_fsmap learned FSMap version 68 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.039+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x555f1409fbf0 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe200054f0 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 68) ==== 8346+0+0 (secure 0 0 0) 0x7fbe1c00f040 con 0x555f141b5920 2024-04-26T02:03:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe41365b80 10 client.27271 fetch_fsmap finished waiting for FSMap version 68 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe41365b80 10 client.27271 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe41365b80 20 client.27271 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.040+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f141d0940 0x555f141d2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.041+0000 7fbe41365b80 4 client.27271 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.041+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f14116780 con 0x555f141d0940 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.041+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f141d0940 0x555f141d2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.041+0000 7fbe1a000700 10 client.27271 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.042+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f141d0940 0x555f141d2d30 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11161522bd33584c server_cookie=93e95b5eab382814 in_seq=0 out_seq=0 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.043+0000 7fbe1a000700 1 -- 172.21.15.52:0/2133351714 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f14116780 con 0x555f141d0940 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.043+0000 7fbe1a000700 10 client.27271 handle_command_reply: tid=0 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.043+0000 7fbe41365b80 1 client.27271 shutdown 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.043+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 msgr2=0x555f141c0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.043+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009e50 tx=0x7fbe1c009900 comp rx=0 tx=0).stop 2024-04-26T02:03:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 shutdown_connections 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f141d0940 0x555f141d2d30 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141c10f0 0x555f141be090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141b5920 0x555f141c0850 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2133351714 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b6240 0x555f141bdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 >> 172.21.15.52:0/2133351714 conn(0x555f140e46a0 msgr2=0x555f141b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe1a000700 0 client.27271 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 shutdown_connections 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.044+0000 7fbe41365b80 1 -- 172.21.15.52:0/2133351714 wait complete. 2024-04-26T02:03:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.045+0000 7fbe41365b80 20 client.27271 trim_cache size 0 max 16384 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.077+0000 7fbe41365b80 1 Processor -- start 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.077+0000 7fbe41365b80 1 -- start start 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b5920 0x555f141b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b62e0 con 0x555f141b6240 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b3b70 con 0x555f141b5920 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe41365b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b19e0 con 0x555f141c10f0 2024-04-26T02:03:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.078+0000 7fbe2f400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b5920 0x555f141b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.079+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.079+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56356/0 (socket says 172.21.15.52:56356) 2024-04-26T02:03:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.079+0000 7fbe2fe00700 1 -- 172.21.15.52:0/4033439318 learned_addr learned my addr 172.21.15.52:0/4033439318 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.079+0000 7fbe2f400700 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.080+0000 7fbe2fe00700 1 -- 172.21.15.52:0/4033439318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 msgr2=0x555f141d85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.080+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141d85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.080+0000 7fbe2fe00700 1 -- 172.21.15.52:0/4033439318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b5920 msgr2=0x555f141b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.080+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b5920 0x555f141b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.080+0000 7fbe2fe00700 1 -- 172.21.15.52:0/4033439318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f141b6850 con 0x555f141b6240 2024-04-26T02:03:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.081+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbe200098f0 tx=0x7fbe2000f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0221bd2fe52b71 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.081+0000 7fbe24c00700 1 -- 172.21.15.52:0/4033439318 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20027040 con 0x555f141b6240 2024-04-26T02:03:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.082+0000 7fbe24c00700 1 -- 172.21.15.52:0/4033439318 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe20032030 con 0x555f141b6240 2024-04-26T02:03:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.082+0000 7fbe24c00700 1 -- 172.21.15.52:0/4033439318 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20022720 con 0x555f141b6240 2024-04-26T02:03:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 -- 172.21.15.52:0/4033439318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 msgr2=0x555f141be040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbe200098f0 tx=0x7fbe2000f8f0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 -- 172.21.15.52:0/4033439318 shutdown_connections 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141d85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141b5920 0x555f141b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 --2- 172.21.15.52:0/4033439318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b6240 0x555f141be040 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.083+0000 7fbe41365b80 1 -- 172.21.15.52:0/4033439318 >> 172.21.15.52:0/4033439318 conn(0x555f140e46a0 msgr2=0x555f141bd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.084+0000 7fbe41365b80 1 -- 172.21.15.52:0/4033439318 shutdown_connections 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.084+0000 7fbe41365b80 1 -- 172.21.15.52:0/4033439318 wait complete. 2024-04-26T02:03:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 -- start start 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 Processor -- start 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c5940 0x555f141c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b2030 con 0x555f141b5920 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b34a0 con 0x555f141c5940 2024-04-26T02:03:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301b2700 con 0x555f141c10f0 2024-04-26T02:03:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.085+0000 7fbe41365b80 10 client.0 _ll_register_callbacks cb 0x555f140a1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.086+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.086+0000 7fbe2fe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33326/0 (socket says 172.21.15.52:33326) 2024-04-26T02:03:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.086+0000 7fbe2fe00700 1 -- 172.21.15.52:0/2501150838 learned_addr learned my addr 172.21.15.52:0/2501150838 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.086+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c5940 0x555f141c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.087+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.087+0000 7fbe2fe00700 1 -- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c5940 msgr2=0x555f141c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.087+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c5940 0x555f141c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.087+0000 7fbe2fe00700 1 -- 172.21.15.52:0/2501150838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 msgr2=0x555f141c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.088+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.088+0000 7fbe2fe00700 1 -- 172.21.15.52:0/2501150838 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f14236e30 con 0x555f141c10f0 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.088+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.088+0000 7fbe2fe00700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x555f141b3240 tx=0x7fbe2000f560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7325407858c40ab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.088+0000 7fbe1a000700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:03:49.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2003d070 con 0x555f141c10f0 2024-04-26T02:03:49.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7fbe2000b760 con 0x555f141c10f0 2024-04-26T02:03:49.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe20032030 con 0x555f141c10f0 2024-04-26T02:03:49.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fbe20047050 con 0x555f141c10f0 2024-04-26T02:03:49.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe41365b80 10 client.27277 fetch_fsmap learned FSMap version 68 2024-04-26T02:03:49.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.089+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x555f14237fd0 con 0x555f141c10f0 2024-04-26T02:03:49.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe20032030 con 0x555f141c10f0 2024-04-26T02:03:49.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 68) ==== 8346+0+0 (secure 0 0 0) 0x7fbe20021070 con 0x555f141c10f0 2024-04-26T02:03:49.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 10 client.27277 fetch_fsmap finished waiting for FSMap version 68 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 10 client.27277 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 20 client.27277 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f14238520 0x555f1423a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 4 client.27277 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f112abc", "await_for": 42.0}] 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.090+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f112abc", "await_for": 42.0}) -- 0x555f14110820 con 0x555f14238520 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.091+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f14238520 0x555f1423a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.091+0000 7fbe2f400700 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f14238520 0x555f1423a910 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb67f35bbd97ee8d server_cookie=162e1b3001419d1 in_seq=0 out_seq=0 2024-04-26T02:03:49.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.091+0000 7fbe1a000700 10 client.27277 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:49.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe1a000700 1 -- 172.21.15.52:0/2501150838 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: -110 ) ==== 8+0+592 (crc 0 0 0) 0x555f14110820 con 0x555f14238520 2024-04-26T02:03:49.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe1a000700 10 client.27277 handle_command_reply: tid=0 2024-04-26T02:03:49.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 client.27277 shutdown 2024-04-26T02:03:49.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 msgr2=0x555f141c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:49.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x555f141b3240 tx=0x7fbe2000f560 comp rx=0 tx=0).stop 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 shutdown_connections 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x555f14238520 0x555f1423a910 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555f141c5940 0x555f141c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555f141b5920 0x555f141c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 --2- 172.21.15.52:0/2501150838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f141c10f0 0x555f141c5400 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 >> 172.21.15.52:0/2501150838 conn(0x555f140e46a0 msgr2=0x555f141bf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe1a000700 0 client.27277 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.093+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 shutdown_connections 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.094+0000 7fbe41365b80 1 -- 172.21.15.52:0/2501150838 wait complete. 2024-04-26T02:03:49.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:03:49.094+0000 7fbe41365b80 20 client.27277 trim_cache size 0 max 16384 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 68, 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 50, 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": { 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "f112abc": { 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "version": 50, 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "age_ref": 10.1, 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "TIMEDOUT", 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 0.0 2024-04-26T02:03:49.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "timeout": 60.0, 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "expiration": 82.5, 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "members": { 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "file:/": { 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "excluded": false, 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "state": { 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "QUIESCING", 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: "age": 60.0 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:49.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:49.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:03:49.097 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:03:49.097 INFO:teuthology.orchestra.run.smithi052.stderr:Error ETIMEDOUT: 2024-04-26T02:03:49.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:48 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:49.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:48 smithi191 ceph-mon[32745]: pgmap v596: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:49.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:48 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:49.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:48 smithi183 ceph-mon[32752]: pgmap v596: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:49.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 99693 17.15 MB/sec execute 186 sec latency 476.891 ms 2024-04-26T02:03:50.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 100271 17.22 MB/sec execute 187 sec latency 1476.951 ms 2024-04-26T02:03:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T02:03:50.813 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout: { "epoch": 68, "leader": 24524, "set_version": 50, "sets": { "f112abc": { "version": 50, "age_ref": 10.1, "state": { "name": "TIMEDOUT", "age": 0.0 }, "timeout": 60.0, "expiration": 82.5, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCING", "age": 60.0 } } } } } } 2024-04-26T02:03:50.813 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 354, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 310, in do_quiesce raise RuntimeError(f"Error quiescing set '{set_id}': {rcinfo(rc)}") RuntimeError: Error quiescing set 'f112abc': 110 (ETIMEDOUT) 2024-04-26T02:03:51.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:51 smithi191 ceph-mon[32745]: pgmap v597: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:51.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:51 smithi183 ceph-mon[32752]: pgmap v597: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:51 smithi052 ceph-mon[27851]: pgmap v597: 97 pgs: 97 active+clean; 886 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.7 MiB/s wr, 288 op/s 2024-04-26T02:03:51.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 100914 17.33 MB/sec execute 188 sec latency 2477.006 ms 2024-04-26T02:03:52.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.426+0000 7f73b2a00640 1 -- 172.21.15.52:0/1132362186 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a00318a0 con 0x7f73b4104d60 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 -- 172.21.15.52:0/1132362186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4104d60 msgr2=0x7f73b4108da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 --2- 172.21.15.52:0/1132362186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4104d60 0x7f73b4108da0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009900 tx=0x7f73a002a3f0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 -- 172.21.15.52:0/1132362186 shutdown_connections 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 --2- 172.21.15.52:0/1132362186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b4109480 0x7f73b4111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 --2- 172.21.15.52:0/1132362186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4104d60 0x7f73b4108da0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 --2- 172.21.15.52:0/1132362186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b4104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 -- 172.21.15.52:0/1132362186 >> 172.21.15.52:0/1132362186 conn(0x7f73b40ffc50 msgr2=0x7f73b4102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 -- 172.21.15.52:0/1132362186 shutdown_connections 2024-04-26T02:03:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.427+0000 7f73bac00640 1 -- 172.21.15.52:0/1132362186 wait complete. 2024-04-26T02:03:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.428+0000 7f73bac00640 1 Processor -- start 2024-04-26T02:03:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.428+0000 7f73bac00640 1 -- start start 2024-04-26T02:03:52.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b419a6e0 0x7f73b419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41143c0 con 0x7f73b41043b0 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4114240 con 0x7f73b4109480 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73bac00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4114540 con 0x7f73b419a6e0 2024-04-26T02:03:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73b3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73b3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35558/0 (socket says 172.21.15.52:35558) 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73b3400640 1 -- 172.21.15.52:0/1847630972 learned_addr learned my addr 172.21.15.52:0/1847630972 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73b3e00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.429+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b419a6e0 0x7f73b419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3400640 1 -- 172.21.15.52:0/1847630972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b419a6e0 msgr2=0x7f73b419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3400640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b419a6e0 0x7f73b419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3400640 1 -- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 msgr2=0x7f73b41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3400640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3400640 1 -- 172.21.15.52:0/1847630972 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b419b2e0 con 0x7f73b4109480 2024-04-26T02:03:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.430+0000 7f73b3e00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:03:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73b3400640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f73a0006010 tx=0x7f73a0004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73b0c00640 1 -- 172.21.15.52:0/1847630972 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a003f030 con 0x7f73b4109480 2024-04-26T02:03:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73b0c00640 1 -- 172.21.15.52:0/1847630972 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a0003aa0 con 0x7f73b4109480 2024-04-26T02:03:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73b0c00640 1 -- 172.21.15.52:0/1847630972 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0035070 con 0x7f73b4109480 2024-04-26T02:03:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b419b570 con 0x7f73b4109480 2024-04-26T02:03:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.431+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b41a7660 con 0x7f73b4109480 2024-04-26T02:03:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.437+0000 7f73bdb6a740 1 Processor -- start 2024-04-26T02:03:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 -- start start 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73b0c00640 1 -- 172.21.15.52:0/1847630972 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f73a0042030 con 0x7f73b4109480 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 0x55c547529c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c5475344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c54748f900 con 0x55c54752a140 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c547435ef0 con 0x55c547534a00 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.438+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c54748b8c0 con 0x55c547529820 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.439+0000 7f73b3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.439+0000 7f73b8c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 0x55c547529c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.439+0000 7f73b3400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:35572/0 (socket says 172.21.15.52:35572) 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.439+0000 7f73b3400640 1 -- 172.21.15.52:0/2370325370 learned_addr learned my addr 172.21.15.52:0/2370325370 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.439+0000 7f73b3e00640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c5475344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 -- 172.21.15.52:0/2370325370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 msgr2=0x55c547529c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 0x55c547529c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 -- 172.21.15.52:0/2370325370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 msgr2=0x55c5475344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c5475344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 -- 172.21.15.52:0/2370325370 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5471a2210 con 0x55c547534a00 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b8c00640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 0x55c547529c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3e00640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c5475344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:03:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.440+0000 7f73b3400640 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f73a00ee1d0 tx=0x7f73a00f4de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef24056c8f57c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.441+0000 7f73a9200640 1 -- 172.21.15.52:0/2370325370 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0100020 con 0x55c547534a00 2024-04-26T02:03:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.442+0000 7f73a9200640 1 -- 172.21.15.52:0/2370325370 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a00fc400 con 0x55c547534a00 2024-04-26T02:03:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.442+0000 7f73a9200640 1 -- 172.21.15.52:0/2370325370 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a00f5040 con 0x55c547534a00 2024-04-26T02:03:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.442+0000 7f73b0c00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 0x7f73980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.442+0000 7f73b0c00640 1 -- 172.21.15.52:0/1847630972 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f73a00f0b10 con 0x7f73b4109480 2024-04-26T02:03:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 0x7f73980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2370325370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 msgr2=0x55c547536df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f73a00ee1d0 tx=0x7f73a00f4de0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2370325370 shutdown_connections 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547534a00 0x55c547536df0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c5475344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2370325370 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547529820 0x55c547529c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2370325370 >> 172.21.15.52:0/2370325370 conn(0x55c547525d30 msgr2=0x55c547526110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2370325370 shutdown_connections 2024-04-26T02:03:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.443+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2370325370 wait complete. 2024-04-26T02:03:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.444+0000 7f73bdb6a740 1 -- start start 2024-04-26T02:03:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.444+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 0x7f73980a6300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005fd0 tx=0x7f73ac009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.444+0000 7f73bdb6a740 1 Processor -- start 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547529820 0x55c547531aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c547531fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 0x55c547548fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c54753fc00 con 0x55c547529820 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55c547449310 con 0x55c547532520 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c54753fd80 con 0x55c54752a140 2024-04-26T02:03:52.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.445+0000 7f73bdb6a740 10 client.0 _ll_register_callbacks cb 0x55c547417cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 0x55c547548fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c547531fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c547531fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33354/0 (socket says 172.21.15.52:33354) 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b3400640 1 -- 172.21.15.52:0/1469172010 learned_addr learned my addr 172.21.15.52:0/1469172010 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b3e00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547529820 0x55c547531aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 -- 172.21.15.52:0/1469172010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 msgr2=0x55c547531fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c547531fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 -- 172.21.15.52:0/1469172010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547529820 msgr2=0x55c547531aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547529820 0x55c547531aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.446+0000 7f73b8c00640 1 -- 172.21.15.52:0/1469172010 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54726e620 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.447+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 0x55c547548fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0042c0 tx=0x7f73ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bf7515204440f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.447+0000 7f739d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.447+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac011de0 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.447+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f73ac004b20 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.448+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73a00fc400 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.448+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f73ac004840 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.448+0000 7f73bdb6a740 10 client.27272 fetch_fsmap learned FSMap version 68 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.448+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x55c54754c320 con 0x55c547532520 2024-04-26T02:03:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.448+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a00fc400 con 0x55c547532520 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 68) ==== 8346+0+0 (secure 0 0 0) 0x7f73ac032420 con 0x55c547532520 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 10 client.27272 fetch_fsmap finished waiting for FSMap version 68 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 10 client.27272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 20 client.27272 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c54754d7e0 0x55c54754fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 4 client.27272 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5474a38b0 con 0x55c54754d7e0 2024-04-26T02:03:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.449+0000 7f73b3e00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c54754d7e0 0x55c54754fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.450+0000 7f73b3e00640 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c54754d7e0 0x55c54754fbd0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc9d73f797df42f2 server_cookie=44d291a9a56924b0 in_seq=0 out_seq=0 2024-04-26T02:03:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.450+0000 7f739d600640 10 client.27272 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.452+0000 7f739d600640 1 -- 172.21.15.52:0/1469172010 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5474a38b0 con 0x55c54754d7e0 2024-04-26T02:03:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.452+0000 7f739d600640 10 client.27272 handle_command_reply: tid=0 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.452+0000 7f73bdb6a740 1 client.27272 shutdown 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 msgr2=0x55c547548fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 0x55c547548fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0042c0 tx=0x7f73ac004a40 comp rx=0 tx=0).stop 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 shutdown_connections 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c54754d7e0 0x55c54754fbd0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547532520 0x55c547548fe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c547531fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1469172010 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547529820 0x55c547531aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 >> 172.21.15.52:0/1469172010 conn(0x55c547525d30 msgr2=0x55c54752bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f739d600640 0 client.27272 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 shutdown_connections 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1469172010 wait complete. 2024-04-26T02:03:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.453+0000 7f73bdb6a740 20 client.27272 trim_cache size 0 max 16384 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.476+0000 7f73bdb6a740 1 Processor -- start 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.476+0000 7f73bdb6a740 1 -- start start 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c54752c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 0x55c547555a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41ae2a0 con 0x55c547532520 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41abb70 con 0x55c547529820 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41a99c0 con 0x55c54752a140 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73b3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73b3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c54752c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73b3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33364/0 (socket says 172.21.15.52:33364) 2024-04-26T02:03:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73b3400640 1 -- 172.21.15.52:0/1474757186 learned_addr learned my addr 172.21.15.52:0/1474757186 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.477+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 0x55c547555a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 -- 172.21.15.52:0/1474757186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 msgr2=0x55c54752c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c54752c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 -- 172.21.15.52:0/1474757186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 msgr2=0x55c547555a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 0x55c547555a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 -- 172.21.15.52:0/1474757186 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54752fff0 con 0x55c54752a140 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b8c00640 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 0x55c547555a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f73b3400640 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f73a00ee860 tx=0x7f73a0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4724ea68f4a0cced server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.478+0000 7f739fe00640 1 -- 172.21.15.52:0/1474757186 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0100030 con 0x55c54752a140 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.479+0000 7f739fe00640 1 -- 172.21.15.52:0/1474757186 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a003b460 con 0x55c54752a140 2024-04-26T02:03:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.479+0000 7f739fe00640 1 -- 172.21.15.52:0/1474757186 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a00bc410 con 0x55c54752a140 2024-04-26T02:03:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.479+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1474757186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 msgr2=0x55c5475554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.479+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f73a00ee860 tx=0x7f73a0000bc0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1474757186 shutdown_connections 2024-04-26T02:03:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c547532520 0x55c547555a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c54752a140 0x55c5475554e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/1474757186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c54752c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1474757186 >> 172.21.15.52:0/1474757186 conn(0x55c547525d30 msgr2=0x55c54752cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1474757186 shutdown_connections 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 -- 172.21.15.52:0/1474757186 wait complete. 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.480+0000 7f73bdb6a740 1 -- start start 2024-04-26T02:03:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 Processor -- start 2024-04-26T02:03:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c547544f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c547545490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41aa030 con 0x55c54752a140 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41ab4a0 con 0x55c547529820 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41aa700 con 0x55c547532520 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.481+0000 7f73bdb6a740 10 client.0 _ll_register_callbacks cb 0x7f73ac003e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c547544f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33376/0 (socket says 172.21.15.52:33376) 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 -- 172.21.15.52:0/2905310684 learned_addr learned my addr 172.21.15.52:0/2905310684 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:03:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b3400640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c547545490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 -- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 msgr2=0x55c547544f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c547544f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 -- 172.21.15.52:0/2905310684 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 msgr2=0x55c547545490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c547545490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 -- 172.21.15.52:0/2905310684 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c547540cc0 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f73b8c00640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00e3d0 tx=0x7f73ac010c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=613455d0610bea13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.482+0000 7f739d600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac004840 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8521+0+0 (secure 0 0 0) 0x7f73ac011be0 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c54749d320 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f73ac010e80 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 10 client.27292 fetch_fsmap learned FSMap version 68 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) -- 0x55c5475b39f0 con 0x55c547532520 2024-04-26T02:03:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c54749d320 con 0x55c547532520 2024-04-26T02:03:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 68) ==== 8346+0+0 (secure 0 0 0) 0x7f73ac03a070 con 0x55c547532520 2024-04-26T02:03:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 10 client.27292 fetch_fsmap finished waiting for FSMap version 68 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 10 client.27292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.483+0000 7f73bdb6a740 20 client.27292 populate_metadata read hostname 'smithi052' 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.484+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c5475b4100 0x55c5475b6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.484+0000 7f73bdb6a740 4 client.27292 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.484+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c547497190 con 0x55c5475b4100 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.484+0000 7f73b3e00640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c5475b4100 0x55c5475b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.485+0000 7f73b3e00640 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c5475b4100 0x55c5475b6520 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33bb1ff6b5735ac3 server_cookie=35f5e39aec8cd674 in_seq=0 out_seq=0 2024-04-26T02:03:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.485+0000 7f739d600640 10 client.27292 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:52.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.690+0000 7f739d600640 1 -- 172.21.15.52:0/2905310684 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c547497190 con 0x55c5475b4100 2024-04-26T02:03:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.690+0000 7f739d600640 10 client.27292 handle_command_reply: tid=0 2024-04-26T02:03:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.690+0000 7f73bdb6a740 1 client.27292 shutdown 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.691+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 msgr2=0x55c5475404c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.691+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00e3d0 tx=0x7f73ac010c30 comp rx=0 tx=0).stop 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.691+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 shutdown_connections 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.691+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55c5475b4100 0x55c5475b6520 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c547532520 0x55c5475404c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c54752a140 0x55c547545490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 --2- 172.21.15.52:0/2905310684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55c547529820 0x55c547544f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 >> 172.21.15.52:0/2905310684 conn(0x55c547525d30 msgr2=0x55c5475332d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f739d600640 0 client.27292 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 shutdown_connections 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 1 -- 172.21.15.52:0/2905310684 wait complete. 2024-04-26T02:03:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.692+0000 7f73bdb6a740 20 client.27292 trim_cache size 0 max 16384 2024-04-26T02:03:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:03:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:03:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:03:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 msgr2=0x7f73980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 0x7f73980a6300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005fd0 tx=0x7f73ac009500 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 msgr2=0x7f73b41a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f73a0006010 tx=0x7f73a0004060 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 shutdown_connections 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73980a3e40 0x7f73980a6300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b419a6e0 0x7f73b419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b4109480 0x7f73b41a0b60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 --2- 172.21.15.52:0/1847630972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b41043b0 0x7f73b41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:03:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.694+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 >> 172.21.15.52:0/1847630972 conn(0x7f73b40ffc50 msgr2=0x7f73b4101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:03:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.695+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 shutdown_connections 2024-04-26T02:03:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:03:52.695+0000 7f73bac00640 1 -- 172.21.15.52:0/1847630972 wait complete. 2024-04-26T02:03:52.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 101356 17.39 MB/sec execute 189 sec latency 3477.054 ms 2024-04-26T02:03:53.175 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:03:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T02:03:53.176 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:03:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T02:03:53.176 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:03:52 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/2426421266 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558029a29a40 con 0x5580299fcc00 2024-04-26T02:03:53.176 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:03:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T02:03:53.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:53 smithi191 ceph-mon[32745]: pgmap v598: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.2 MiB/s wr, 455 op/s 2024-04-26T02:03:53.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:53 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T02:03:53.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:53 smithi183 ceph-mon[32752]: pgmap v598: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.2 MiB/s wr, 455 op/s 2024-04-26T02:03:53.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:53 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T02:03:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:53 smithi052 ceph-mon[27851]: pgmap v598: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 6.2 MiB/s wr, 455 op/s 2024-04-26T02:03:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:53 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T02:03:53.744 INFO:tasks.workunit.client.0.smithi052.stdout: 10 101727 17.38 MB/sec execute 190 sec latency 3686.512 ms 2024-04-26T02:03:54.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:54 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:03:54.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:54 smithi183 ceph-mon[32752]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:54.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:54 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/2453995259,v1:172.21.15.183:6837/2453995259] up:boot 2024-04-26T02:03:54.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:54 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:03:54.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:54 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:54.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:54 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:03:54.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:54 smithi052 ceph-mon[27851]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:54.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:54 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/2453995259,v1:172.21.15.183:6837/2453995259] up:boot 2024-04-26T02:03:54.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:54 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:03:54.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:54 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:54.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 102405 17.47 MB/sec execute 191 sec latency 86.591 ms 2024-04-26T02:03:54.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:54 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:03:54.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:54 smithi191 ceph-mon[32745]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T02:03:54.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:54 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/2453995259,v1:172.21.15.183:6837/2453995259] up:boot 2024-04-26T02:03:54.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:54 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T02:03:54.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:54 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:03:55.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:55 smithi183 ceph-mon[32752]: pgmap v600: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.2 MiB/s wr, 450 op/s 2024-04-26T02:03:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:55 smithi052 ceph-mon[27851]: pgmap v600: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.2 MiB/s wr, 450 op/s 2024-04-26T02:03:55.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 102996 17.52 MB/sec execute 192 sec latency 113.115 ms 2024-04-26T02:03:55.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:55 smithi191 ceph-mon[32745]: pgmap v600: 97 pgs: 97 active+clean; 913 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.2 MiB/s wr, 450 op/s 2024-04-26T02:03:56.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 103426 17.56 MB/sec execute 193 sec latency 373.545 ms 2024-04-26T02:03:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:57 smithi183 ceph-mon[32752]: pgmap v601: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:03:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:57 smithi052 ceph-mon[27851]: pgmap v601: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:03:57.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 104130 17.71 MB/sec execute 194 sec latency 70.659 ms 2024-04-26T02:03:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:57 smithi191 ceph-mon[32745]: pgmap v601: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:03:58.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 104608 17.71 MB/sec execute 195 sec latency 107.835 ms 2024-04-26T02:03:59.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:03:59 smithi183 ceph-mon[32752]: pgmap v602: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:03:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:03:59 smithi052 ceph-mon[27851]: pgmap v602: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:03:59.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105120 17.78 MB/sec execute 196 sec latency 173.733 ms 2024-04-26T02:03:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:03:59 smithi191 ceph-mon[32745]: pgmap v602: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:04:00.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105527 17.82 MB/sec execute 197 sec latency 357.966 ms 2024-04-26T02:04:01.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:01 smithi183 ceph-mon[32752]: pgmap v603: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:04:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:01 smithi052 ceph-mon[27851]: pgmap v603: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:04:01.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105527 17.73 MB/sec execute 198 sec latency 1202.925 ms 2024-04-26T02:04:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:01 smithi191 ceph-mon[32745]: pgmap v603: 97 pgs: 97 active+clean; 922 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.7 MiB/s wr, 524 op/s 2024-04-26T02:04:02.746 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105527 17.64 MB/sec execute 199 sec latency 2203.099 ms 2024-04-26T02:04:03.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:03 smithi191 ceph-mon[32745]: pgmap v604: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 612 op/s 2024-04-26T02:04:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:03 smithi052 ceph-mon[27851]: pgmap v604: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 612 op/s 2024-04-26T02:04:03.745 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105637 17.59 MB/sec execute 200 sec latency 2630.414 ms 2024-04-26T02:04:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:03 smithi183 ceph-mon[32752]: pgmap v604: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.5 MiB/s wr, 612 op/s 2024-04-26T02:04:04.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105639 17.51 MB/sec execute 201 sec latency 1372.353 ms 2024-04-26T02:04:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:05 smithi052 ceph-mon[27851]: pgmap v605: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.7 MiB/s wr, 559 op/s 2024-04-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi052.stdout: 10 105745 17.42 MB/sec execute 202 sec latency 2099.276 ms 2024-04-26T02:04:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.894+0000 7f90aaa00640 1 -- 172.21.15.52:0/3215631609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a410b070 msgr2=0x7f90a410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.894+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3215631609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a410b070 0x7f90a410d460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f909c009be0 tx=0x7f909c02a6c0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 -- 172.21.15.52:0/3215631609 shutdown_connections 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3215631609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a410b070 0x7f90a410d460 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3215631609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90a4100d00 0x7f90a4105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3215631609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a4104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 -- 172.21.15.52:0/3215631609 >> 172.21.15.52:0/3215631609 conn(0x7f90a4076f50 msgr2=0x7f90a40fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 -- 172.21.15.52:0/3215631609 shutdown_connections 2024-04-26T02:04:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.895+0000 7f90aaa00640 1 -- 172.21.15.52:0/3215631609 wait complete. 2024-04-26T02:04:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.897+0000 7f90aaa00640 1 Processor -- start 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.897+0000 7f90aaa00640 1 -- start start 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.897+0000 7f90aaa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90aaa00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a4100d00 0x7f90a41a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90aaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90a410b070 0x7f90a4109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90aaa00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41121e0 con 0x7f90a4100d00 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90aaa00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90a4112060 con 0x7f90a4068920 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90aaa00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90a4112360 con 0x7f90a410b070 2024-04-26T02:04:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41474/0 (socket says 172.21.15.52:41474) 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90a3e00640 1 -- 172.21.15.52:0/3532392153 learned_addr learned my addr 172.21.15.52:0/3532392153 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.898+0000 7f90a3400640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a4100d00 0x7f90a41a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.899+0000 7f90a3e00640 1 -- 172.21.15.52:0/3532392153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90a410b070 msgr2=0x7f90a4109d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.899+0000 7f90a3e00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90a410b070 0x7f90a4109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.899+0000 7f90a3e00640 1 -- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a4100d00 msgr2=0x7f90a41a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.899+0000 7f90a3e00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a4100d00 0x7f90a41a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.899+0000 7f90a3e00640 1 -- 172.21.15.52:0/3532392153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a410a600 con 0x7f90a4068920 2024-04-26T02:04:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.900+0000 7f90a3e00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f909400e9c0 tx=0x7f909400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.901+0000 7f90a0c00640 1 -- 172.21.15.52:0/3532392153 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909400cd60 con 0x7f90a4068920 2024-04-26T02:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.901+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a410a8f0 con 0x7f90a4068920 2024-04-26T02:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.901+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a41a75e0 con 0x7f90a4068920 2024-04-26T02:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.901+0000 7f90a0c00640 1 -- 172.21.15.52:0/3532392153 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90940035f0 con 0x7f90a4068920 2024-04-26T02:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.901+0000 7f90a0c00640 1 -- 172.21.15.52:0/3532392153 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094010670 con 0x7f90a4068920 2024-04-26T02:04:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:05 smithi191 ceph-mon[32745]: pgmap v605: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.7 MiB/s wr, 559 op/s 2024-04-26T02:04:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.908+0000 7f90a0c00640 1 -- 172.21.15.52:0/3532392153 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9094004000 con 0x7f90a4068920 2024-04-26T02:04:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.910+0000 7f90ad776740 1 Processor -- start 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 -- start start 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bc760 0x557e968bcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557e96822900 con 0x557e968bd080 2024-04-26T02:04:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557e967c8ef0 con 0x557e968bc760 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90ad776740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557e9681e8c0 con 0x557e968d0210 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90a3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90a3400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33178/0 (socket says 172.21.15.52:33178) 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.911+0000 7f90a3400640 1 -- 172.21.15.52:0/62824563 learned_addr learned my addr 172.21.15.52:0/62824563 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.912+0000 7f90a3400640 1 -- 172.21.15.52:0/62824563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bc760 msgr2=0x557e968bcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.912+0000 7f90a3400640 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bc760 0x557e968bcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.912+0000 7f90a3400640 1 -- 172.21.15.52:0/62824563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 msgr2=0x557e968cfcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:04:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.913+0000 7f90a0c00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 0x7f90880a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.913+0000 7f90a0c00640 1 -- 172.21.15.52:0/3532392153 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8477+0+0 (secure 0 0 0) 0x7f90940d0a60 con 0x7f90a4068920 2024-04-26T02:04:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.913+0000 7f90a3e00640 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.914+0000 7f90a3400640 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.914+0000 7f90a3400640 1 -- 172.21.15.52:0/62824563 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e96535210 con 0x557e968d0210 2024-04-26T02:04:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.914+0000 7f90a3e00640 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.914+0000 7f90a8a00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 0x7f90880a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.914+0000 7f90a3400640 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9090009ba0 tx=0x7f9090011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=602a537bf4f5ff15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.915+0000 7f9099200640 1 -- 172.21.15.52:0/62824563 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090004230 con 0x557e968d0210 2024-04-26T02:04:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.915+0000 7f9099200640 1 -- 172.21.15.52:0/62824563 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9090009e10 con 0x557e968d0210 2024-04-26T02:04:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.915+0000 7f9099200640 1 -- 172.21.15.52:0/62824563 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090004d00 con 0x557e968d0210 2024-04-26T02:04:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.916+0000 7f90ad776740 1 -- 172.21.15.52:0/62824563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 msgr2=0x557e968d2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.916+0000 7f90ad776740 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9090009ba0 tx=0x7f9090011810 comp rx=0 tx=0).stop 2024-04-26T02:04:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90ad776740 1 -- 172.21.15.52:0/62824563 shutdown_connections 2024-04-26T02:04:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90ad776740 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968d0210 0x557e968d2600 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90ad776740 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90ad776740 1 --2- 172.21.15.52:0/62824563 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bc760 0x557e968bcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90ad776740 1 -- 172.21.15.52:0/62824563 >> 172.21.15.52:0/62824563 conn(0x557e968b8d30 msgr2=0x557e968b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.917+0000 7f90a8a00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 0x7f90880a63d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f909c034040 tx=0x7f909c0098d0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.918+0000 7f90ad776740 1 -- 172.21.15.52:0/62824563 shutdown_connections 2024-04-26T02:04:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.918+0000 7f90ad776740 1 -- 172.21.15.52:0/62824563 wait complete. 2024-04-26T02:04:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.919+0000 7f90ad776740 1 -- start start 2024-04-26T02:04:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.919+0000 7f90ad776740 1 Processor -- start 2024-04-26T02:04:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.920+0000 7f90ad776740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.920+0000 7f90ad776740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968ca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.921+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.921+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33186/0 (socket says 172.21.15.52:33186) 2024-04-26T02:04:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.921+0000 7f90a3400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.921+0000 7f90ad776740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968d0210 0x557e968cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90ad776740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557e968bf700 con 0x557e968bd080 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90ad776740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557e967dc310 con 0x557e968d0210 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90ad776740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557e968bf880 con 0x557e968bc760 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90a3e00640 1 -- 172.21.15.52:0/445174651 learned_addr learned my addr 172.21.15.52:0/445174651 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90ad776740 10 client.0 _ll_register_callbacks cb 0x557e967aacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90a8a00640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968d0210 0x557e968cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90a3e00640 1 -- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968d0210 msgr2=0x557e968cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.922+0000 7f90a3e00640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968d0210 0x557e968cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.923+0000 7f90a3e00640 1 -- 172.21.15.52:0/445174651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 msgr2=0x557e968ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.923+0000 7f90a3e00640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.923+0000 7f90a3e00640 1 -- 172.21.15.52:0/445174651 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e96601620 con 0x557e968bc760 2024-04-26T02:04:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.923+0000 7f90a3400640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.924+0000 7f90a3e00640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f909400c6a0 tx=0x7f909409c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11fb4a0249830052 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.924+0000 7f908d600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.925+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909409c9d0 con 0x557e968bc760 2024-04-26T02:04:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.925+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8477+0+0 (secure 0 0 0) 0x7f9094099040 con 0x557e968bc760 2024-04-26T02:04:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.926+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9090009e10 con 0x557e968bc760 2024-04-26T02:04:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.926+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f90940a0d80 con 0x557e968bc760 2024-04-26T02:04:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.926+0000 7f90ad776740 10 client.27307 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.926+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x557e968e78c0 con 0x557e968bc760 2024-04-26T02:04:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:05 smithi183 ceph-mon[32752]: pgmap v605: 97 pgs: 97 active+clean; 948 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 8.7 MiB/s wr, 559 op/s 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.926+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9090009e10 con 0x557e968bc760 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f9094087070 con 0x557e968bc760 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 10 client.27307 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 10 client.27307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 20 client.27307 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e968e8d30 0x557e968eb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 4 client.27307 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e968368b0 con 0x557e968e8d30 2024-04-26T02:04:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.927+0000 7f90a3400640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e968e8d30 0x557e968eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.929+0000 7f90a3400640 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e968e8d30 0x557e968eb120 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ea3a7fa5a448176 server_cookie=4fa498583f8a69ba in_seq=0 out_seq=0 2024-04-26T02:04:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.929+0000 7f908d600640 10 client.27307 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.932+0000 7f908d600640 1 -- 172.21.15.52:0/445174651 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e968368b0 con 0x557e968e8d30 2024-04-26T02:04:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.932+0000 7f908d600640 10 client.27307 handle_command_reply: tid=0 2024-04-26T02:04:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 client.27307 shutdown 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 msgr2=0x557e968c9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f909400c6a0 tx=0x7f909409c420 comp rx=0 tx=0).stop 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 shutdown_connections 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e968e8d30 0x557e968eb120 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968d0210 0x557e968cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968bd080 0x557e968ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 --2- 172.21.15.52:0/445174651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bc760 0x557e968c9dc0 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f909400c6a0 tx=0x7f909409c420 comp rx=0 tx=0).stop 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 >> 172.21.15.52:0/445174651 conn(0x557e968b8d30 msgr2=0x557e968beab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f908d600640 0 client.27307 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.933+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 shutdown_connections 2024-04-26T02:04:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.934+0000 7f90ad776740 1 -- 172.21.15.52:0/445174651 wait complete. 2024-04-26T02:04:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.934+0000 7f90ad776740 20 client.27307 trim_cache size 0 max 16384 2024-04-26T02:04:05.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.971+0000 7f90ad776740 1 Processor -- start 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 -- start start 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bd080 0x557e968b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968f0f90 0x557e968c9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41ae220 con 0x557e968d0210 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41abaf0 con 0x557e968bd080 2024-04-26T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.972+0000 7f90ad776740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41a9980 con 0x557e968f0f90 2024-04-26T02:04:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.973+0000 7f90a3400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.973+0000 7f90a3400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45990/0 (socket says 172.21.15.52:45990) 2024-04-26T02:04:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.973+0000 7f90a3400640 1 -- 172.21.15.52:0/2132736440 learned_addr learned my addr 172.21.15.52:0/2132736440 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 -- 172.21.15.52:0/2132736440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968f0f90 msgr2=0x557e968c9970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:04:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968f0f90 0x557e968c9970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 -- 172.21.15.52:0/2132736440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bd080 msgr2=0x557e968b7d10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bd080 0x557e968b7d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 -- 172.21.15.52:0/2132736440 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e968b4690 con 0x557e968d0210 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f90a3400640 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9090009b40 tx=0x7f90900048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c80d91ccc6fdb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.974+0000 7f908fe00640 1 -- 172.21.15.52:0/2132736440 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090030070 con 0x557e968d0210 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.975+0000 7f908fe00640 1 -- 172.21.15.52:0/2132736440 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9090004630 con 0x557e968d0210 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.975+0000 7f908fe00640 1 -- 172.21.15.52:0/2132736440 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909001e040 con 0x557e968d0210 2024-04-26T02:04:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.976+0000 7f90ad776740 1 -- 172.21.15.52:0/2132736440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 msgr2=0x557e968f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.976+0000 7f90ad776740 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9090009b40 tx=0x7f90900048e0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.976+0000 7f90ad776740 1 -- 172.21.15.52:0/2132736440 shutdown_connections 2024-04-26T02:04:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.977+0000 7f90ad776740 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968f0f90 0x557e968c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.977+0000 7f90ad776740 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e968f0a50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.977+0000 7f90ad776740 1 --2- 172.21.15.52:0/2132736440 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968bd080 0x557e968b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.977+0000 7f90ad776740 1 -- 172.21.15.52:0/2132736440 >> 172.21.15.52:0/2132736440 conn(0x557e968b8d30 msgr2=0x557e968b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:05.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.977+0000 7f90ad776740 1 -- 172.21.15.52:0/2132736440 shutdown_connections 2024-04-26T02:04:05.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.978+0000 7f90ad776740 1 -- 172.21.15.52:0/2132736440 wait complete. 2024-04-26T02:04:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.979+0000 7f90ad776740 1 -- start start 2024-04-26T02:04:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.979+0000 7f90ad776740 1 Processor -- start 2024-04-26T02:04:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 0x557e96949d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e9694a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 0x557e9694a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41a9fd0 con 0x557e968d0210 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41ab420 con 0x557e968f0f90 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41aa680 con 0x557e968bd080 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90ad776740 10 client.0 _ll_register_callbacks cb 0x7f909409c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 0x557e96949d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a8a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 0x557e9694a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 0x557e96949d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33208/0 (socket says 172.21.15.52:33208) 2024-04-26T02:04:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a3e00640 1 -- 172.21.15.52:0/3417998555 learned_addr learned my addr 172.21.15.52:0/3417998555 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a8a00640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 0x557e9694a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:41512/0 (socket says 172.21.15.52:41512) 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.980+0000 7f90a3400640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e9694a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.981+0000 7f90a3400640 1 -- 172.21.15.52:0/3417998555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 msgr2=0x557e96949d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.981+0000 7f90a3400640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 0x557e96949d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.981+0000 7f90a3400640 1 -- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 msgr2=0x557e9694a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.981+0000 7f90a3400640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 0x557e9694a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.981+0000 7f90a3400640 1 -- 172.21.15.52:0/3417998555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e968f9980 con 0x557e968d0210 2024-04-26T02:04:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.982+0000 7f90a3400640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e9694a2d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x557e968b9a20 tx=0x7f9090009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b72d572326db598 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.982+0000 7f908d600640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:04:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.983+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090030040 con 0x557e968d0210 2024-04-26T02:04:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.983+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8477+0+0 (secure 0 0 0) 0x7f909002a4b0 con 0x557e968d0210 2024-04-26T02:04:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.983+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9090004630 con 0x557e968d0210 2024-04-26T02:04:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.983+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f909003a050 con 0x557e968d0210 2024-04-26T02:04:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.984+0000 7f90ad776740 10 client.17133 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.984+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x557e9695ffc0 con 0x557e968d0210 2024-04-26T02:04:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.985+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9090004630 con 0x557e968d0210 2024-04-26T02:04:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.985+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f9090018070 con 0x557e968d0210 2024-04-26T02:04:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.985+0000 7f90ad776740 10 client.17133 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.985+0000 7f90ad776740 10 client.17133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.985+0000 7f90ad776740 20 client.17133 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.986+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e969602f0 0x557e969626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:05.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.986+0000 7f90ad776740 4 client.17133 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:05.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.986+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e96830320 con 0x557e969602f0 2024-04-26T02:04:05.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.986+0000 7f90a3e00640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e969602f0 0x557e969626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.987+0000 7f90a3e00640 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e969602f0 0x557e969626e0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d5a4f0b328e1b96 server_cookie=3d65576a1b078690 in_seq=0 out_seq=0 2024-04-26T02:04:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:05.987+0000 7f908d600640 10 client.17133 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.189+0000 7f908d600640 1 -- 172.21.15.52:0/3417998555 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e96830320 con 0x557e969602f0 2024-04-26T02:04:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.189+0000 7f908d600640 10 client.17133 handle_command_reply: tid=0 2024-04-26T02:04:06.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.189+0000 7f90ad776740 1 client.17133 shutdown 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.190+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 msgr2=0x557e9694a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.190+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e9694a2d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x557e968b9a20 tx=0x7f9090009e80 comp rx=0 tx=0).stop 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 shutdown_connections 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557e969602f0 0x557e969626e0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557e968f0f90 0x557e9694a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557e968d0210 0x557e9694a2d0 secure :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0x557e968b9a20 tx=0x7f9090009e80 comp rx=0 tx=0).stop 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 --2- 172.21.15.52:0/3417998555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557e968bd080 0x557e96949d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 >> 172.21.15.52:0/3417998555 conn(0x557e968b8d30 msgr2=0x557e968b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f908d600640 0 client.17133 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 shutdown_connections 2024-04-26T02:04:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.191+0000 7f90ad776740 1 -- 172.21.15.52:0/3417998555 wait complete. 2024-04-26T02:04:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.192+0000 7f90ad776740 20 client.17133 trim_cache size 0 max 16384 2024-04-26T02:04:06.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 msgr2=0x7f90880a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 0x7f90880a63d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f909c034040 tx=0x7f909c0098d0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 msgr2=0x7f90a41a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f909400e9c0 tx=0x7f909400ee80 comp rx=0 tx=0).stop 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 shutdown_connections 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f90880a3f10 0x7f90880a63d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90a410b070 0x7f90a4109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90a4100d00 0x7f90a41a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 --2- 172.21.15.52:0/3532392153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90a4068920 0x7f90a41a0660 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 >> 172.21.15.52:0/3532392153 conn(0x7f90a4076f50 msgr2=0x7f90a410c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 shutdown_connections 2024-04-26T02:04:06.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:06.193+0000 7f90aaa00640 1 -- 172.21.15.52:0/3532392153 wait complete. 2024-04-26T02:04:06.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 106007 17.37 MB/sec execute 203 sec latency 238.288 ms 2024-04-26T02:04:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:07 smithi052 ceph-mon[27851]: pgmap v606: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-04-26T02:04:07.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:07 smithi052 ceph-mon[27851]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:07.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 106301 17.35 MB/sec execute 204 sec latency 262.294 ms 2024-04-26T02:04:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:07 smithi191 ceph-mon[32745]: pgmap v606: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-04-26T02:04:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:07 smithi191 ceph-mon[32745]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:07 smithi183 ceph-mon[32752]: pgmap v606: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 743 op/s 2024-04-26T02:04:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:07 smithi183 ceph-mon[32752]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T02:04:08.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 106722 17.41 MB/sec execute 205 sec latency 152.581 ms 2024-04-26T02:04:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:09 smithi052 ceph-mon[27851]: pgmap v608: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:09.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 107177 17.46 MB/sec execute 206 sec latency 78.174 ms 2024-04-26T02:04:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:09 smithi191 ceph-mon[32745]: pgmap v608: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:09 smithi183 ceph-mon[32752]: pgmap v608: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:10.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 107686 17.48 MB/sec execute 207 sec latency 90.456 ms 2024-04-26T02:04:11.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:11 smithi052 ceph-mon[27851]: pgmap v609: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:11.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 107983 17.48 MB/sec execute 208 sec latency 453.537 ms 2024-04-26T02:04:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:11 smithi191 ceph-mon[32745]: pgmap v609: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:11 smithi183 ceph-mon[32752]: pgmap v609: 97 pgs: 97 active+clean; 992 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 14 MiB/s wr, 567 op/s 2024-04-26T02:04:12.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 108691 17.56 MB/sec execute 209 sec latency 64.368 ms 2024-04-26T02:04:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:13 smithi052 ceph-mon[27851]: pgmap v610: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 637 op/s 2024-04-26T02:04:13.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:13 smithi052 ceph-mon[27851]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:13.727 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:13 smithi183 ceph-mon[32752]: pgmap v610: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 637 op/s 2024-04-26T02:04:13.727 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:13 smithi183 ceph-mon[32752]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:13.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 109378 17.69 MB/sec execute 210 sec latency 130.176 ms 2024-04-26T02:04:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:13 smithi191 ceph-mon[32745]: pgmap v610: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 637 op/s 2024-04-26T02:04:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:13 smithi191 ceph-mon[32745]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T02:04:14.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 109936 17.76 MB/sec execute 211 sec latency 88.347 ms 2024-04-26T02:04:15.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 110426 17.81 MB/sec execute 212 sec latency 123.239 ms 2024-04-26T02:04:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:15 smithi191 ceph-mon[32745]: pgmap v612: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-04-26T02:04:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:15 smithi183 ceph-mon[32752]: pgmap v612: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-04-26T02:04:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:15 smithi052 ceph-mon[27851]: pgmap v612: 97 pgs: 97 active+clean; 947 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.8 MiB/s wr, 447 op/s 2024-04-26T02:04:16.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 110845 17.82 MB/sec execute 213 sec latency 236.009 ms 2024-04-26T02:04:17.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 111354 17.84 MB/sec execute 214 sec latency 273.174 ms 2024-04-26T02:04:17.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:17 smithi052 ceph-mon[27851]: pgmap v613: 97 pgs: 97 active+clean; 976 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.2 MiB/s wr, 552 op/s 2024-04-26T02:04:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:17 smithi191 ceph-mon[32745]: pgmap v613: 97 pgs: 97 active+clean; 976 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.2 MiB/s wr, 552 op/s 2024-04-26T02:04:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:17 smithi183 ceph-mon[32752]: pgmap v613: 97 pgs: 97 active+clean; 976 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.2 MiB/s wr, 552 op/s 2024-04-26T02:04:18.748 INFO:tasks.workunit.client.0.smithi052.stdout: 10 111901 17.88 MB/sec execute 215 sec latency 100.839 ms 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 -- 172.21.15.52:0/1311760501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad01031e0 msgr2=0x7fbad01035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 --2- 172.21.15.52:0/1311760501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad01031e0 0x7fbad01035c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009900 tx=0x7fbac402a510 comp rx=0 tx=0).stop 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 -- 172.21.15.52:0/1311760501 shutdown_connections 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 --2- 172.21.15.52:0/1311760501 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad010c520 0x7fbad010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 --2- 172.21.15.52:0/1311760501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad0103b00 0x7fbad010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 --2- 172.21.15.52:0/1311760501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad01031e0 0x7fbad01035c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 -- 172.21.15.52:0/1311760501 >> 172.21.15.52:0/1311760501 conn(0x7fbad0077fd0 msgr2=0x7fbad00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 -- 172.21.15.52:0/1311760501 shutdown_connections 2024-04-26T02:04:19.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.392+0000 7fbad7800640 1 -- 172.21.15.52:0/1311760501 wait complete. 2024-04-26T02:04:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.393+0000 7fbad7800640 1 Processor -- start 2024-04-26T02:04:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.393+0000 7fbad7800640 1 -- start start 2024-04-26T02:04:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad01031e0 0x7fbad0100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad010c520 0x7fbad01018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01121e0 con 0x7fbad01031e0 2024-04-26T02:04:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0112060 con 0x7fbad010c520 2024-04-26T02:04:19.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad7800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0112360 con 0x7fbad0103b00 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37960/0 (socket says 172.21.15.52:37960) 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbacfe00640 1 -- 172.21.15.52:0/906986929 learned_addr learned my addr 172.21.15.52:0/906986929 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad5800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad010c520 0x7fbad01018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.394+0000 7fbad4e00640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad01031e0 0x7fbad0100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 -- 172.21.15.52:0/906986929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad010c520 msgr2=0x7fbad01018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad010c520 0x7fbad01018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 -- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad01031e0 msgr2=0x7fbad0100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad01031e0 0x7fbad0100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 -- 172.21.15.52:0/906986929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad01a6c00 con 0x7fbad0103b00 2024-04-26T02:04:19.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.395+0000 7fbacfe00640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00e9f0 tx=0x7fbabc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.396+0000 7fbacd600640 1 -- 172.21.15.52:0/906986929 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00cd80 con 0x7fbad0103b00 2024-04-26T02:04:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.396+0000 7fbacd600640 1 -- 172.21.15.52:0/906986929 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc0108e0 con 0x7fbad0103b00 2024-04-26T02:04:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.396+0000 7fbacd600640 1 -- 172.21.15.52:0/906986929 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc018860 con 0x7fbad0103b00 2024-04-26T02:04:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.396+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad01a6ef0 con 0x7fbad0103b00 2024-04-26T02:04:19.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.396+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad01a7300 con 0x7fbad0103b00 2024-04-26T02:04:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.403+0000 7fbada79b740 1 Processor -- start 2024-04-26T02:04:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.403+0000 7fbacd600640 1 -- 172.21.15.52:0/906986929 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbabc010430 con 0x7fbad0103b00 2024-04-26T02:04:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.403+0000 7fbada79b740 1 -- start start 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a053410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cd29fb0900 con 0x55cd2a053030 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cd29f56ef0 con 0x55cd2a053950 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.404+0000 7fbada79b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cd29fac8c0 con 0x55cd2a05e210 2024-04-26T02:04:19.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbad5800640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a053410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37968/0 (socket says 172.21.15.52:37968) 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 -- 172.21.15.52:0/1956529713 learned_addr learned my addr 172.21.15.52:0/1956529713 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 -- 172.21.15.52:0/1956529713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 msgr2=0x55cd2a05dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 -- 172.21.15.52:0/1956529713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 msgr2=0x55cd2a053410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a053410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.405+0000 7fbacfe00640 1 -- 172.21.15.52:0/1956529713 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd29cc3210 con 0x55cd2a05e210 2024-04-26T02:04:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.406+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbabc0ce0c0 tx=0x7fbabc0d9540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c78bffd752665652 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.406+0000 7fbac2000640 1 -- 172.21.15.52:0/1956529713 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0da040 con 0x55cd2a05e210 2024-04-26T02:04:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.407+0000 7fbac2000640 1 -- 172.21.15.52:0/1956529713 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc0d9c20 con 0x55cd2a05e210 2024-04-26T02:04:19.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.407+0000 7fbacd600640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 0x7fbab80a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.407+0000 7fbacd600640 1 -- 172.21.15.52:0/906986929 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbabc0d0a00 con 0x7fbad0103b00 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.407+0000 7fbada79b740 1 -- 172.21.15.52:0/1956529713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 msgr2=0x55cd2a060600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.407+0000 7fbada79b740 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbabc0ce0c0 tx=0x7fbabc0d9540 comp rx=0 tx=0).stop 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 -- 172.21.15.52:0/1956529713 shutdown_connections 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a060600 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 --2- 172.21.15.52:0/1956529713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a053410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 -- 172.21.15.52:0/1956529713 >> 172.21.15.52:0/1956529713 conn(0x55cd2a04f540 msgr2=0x55cd2a04f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbad5800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 0x7fbab80a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 -- 172.21.15.52:0/1956529713 shutdown_connections 2024-04-26T02:04:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.408+0000 7fbada79b740 1 -- 172.21.15.52:0/1956529713 wait complete. 2024-04-26T02:04:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.409+0000 7fbad5800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 0x7fbab80a6420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005fd0 tx=0x7fbac80096a0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.409+0000 7fbada79b740 1 -- start start 2024-04-26T02:04:19.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 Processor -- start 2024-04-26T02:04:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a05b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cd2a060e90 con 0x55cd2a053030 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55cd29f6a310 con 0x55cd2a053950 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.410+0000 7fbada79b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55cd2a061010 con 0x55cd2a05e210 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbada79b740 10 client.0 _ll_register_callbacks cb 0x55cd29f38cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37982/0 (socket says 172.21.15.52:37982) 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 -- 172.21.15.52:0/3574079042 learned_addr learned my addr 172.21.15.52:0/3574079042 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:19.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad4e00640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a05b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 -- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 msgr2=0x55cd2a05b590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 -- 172.21.15.52:0/3574079042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 msgr2=0x55cd2a05b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a05b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.411+0000 7fbad5800640 1 -- 172.21.15.52:0/3574079042 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd29d8f620 con 0x55cd2a05e210 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.412+0000 7fbad5800640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbac801f700 tx=0x7fbac8004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecfad57b5bc3ad6b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.412+0000 7fbab6a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:19.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.412+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac802c060 con 0x55cd2a05e210 2024-04-26T02:04:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.412+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbac8017030 con 0x55cd2a05e210 2024-04-26T02:04:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.412+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbabc0d9c20 con 0x55cd2a05e210 2024-04-26T02:04:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.413+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fbac8010880 con 0x55cd2a05e210 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.413+0000 7fbada79b740 10 client.27322 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.413+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55cd2a06d0e0 con 0x55cd2a05e210 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.413+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc0d9c20 con 0x55cd2a05e210 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fbac8027bb0 con 0x55cd2a05e210 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 10 client.27322 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 10 client.27322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 20 client.27322 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a06e550 0x55cd2a0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 4 client.27322 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:19.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd29fc48b0 con 0x55cd2a06e550 2024-04-26T02:04:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.414+0000 7fbad4e00640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a06e550 0x55cd2a0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.415+0000 7fbad4e00640 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a06e550 0x55cd2a0709a0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2869cbd9d219f4d server_cookie=5ddfef78ba478ade in_seq=0 out_seq=0 2024-04-26T02:04:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.415+0000 7fbab6a00640 10 client.27322 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.416+0000 7fbab6a00640 1 -- 172.21.15.52:0/3574079042 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd29fc48b0 con 0x55cd2a06e550 2024-04-26T02:04:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.416+0000 7fbab6a00640 10 client.27322 handle_command_reply: tid=0 2024-04-26T02:04:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.416+0000 7fbada79b740 1 client.27322 shutdown 2024-04-26T02:04:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.416+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 msgr2=0x55cd2a05bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.416+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbac801f700 tx=0x7fbac8004ab0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 shutdown_connections 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a06e550 0x55cd2a0709a0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a05e210 0x55cd2a05bad0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a05b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 --2- 172.21.15.52:0/3574079042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a053030 0x55cd2a05b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 >> 172.21.15.52:0/3574079042 conn(0x55cd2a04f540 msgr2=0x55cd2a045e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbab6a00640 0 client.27322 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 shutdown_connections 2024-04-26T02:04:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 1 -- 172.21.15.52:0/3574079042 wait complete. 2024-04-26T02:04:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.417+0000 7fbada79b740 20 client.27322 trim_cache size 0 max 16384 2024-04-26T02:04:19.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.442+0000 7fbada79b740 1 Processor -- start 2024-04-26T02:04:19.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbada79b740 1 -- start start 2024-04-26T02:04:19.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbada79b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbada79b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a055a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbada79b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a045d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbad4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.443+0000 7fbad4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37984/0 (socket says 172.21.15.52:37984) 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 -- 172.21.15.52:0/1175304415 learned_addr learned my addr 172.21.15.52:0/1175304415 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01ae290 con 0x55cd2a05e210 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01abb60 con 0x55cd2a053950 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01a99d0 con 0x55cd2a053030 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a055a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad5800640 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a045d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 -- 172.21.15.52:0/1175304415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 msgr2=0x55cd2a055a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a055a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 -- 172.21.15.52:0/1175304415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 msgr2=0x55cd2a045d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a045d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 -- 172.21.15.52:0/1175304415 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd2a04f370 con 0x55cd2a053030 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.444+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbac4000c00 tx=0x7fbac4005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77cef60809baf973 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.445+0000 7fbac1600640 1 -- 172.21.15.52:0/1175304415 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4035040 con 0x55cd2a053030 2024-04-26T02:04:19.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.445+0000 7fbac1600640 1 -- 172.21.15.52:0/1175304415 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac4049070 con 0x55cd2a053030 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.445+0000 7fbac1600640 1 -- 172.21.15.52:0/1175304415 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac40046c0 con 0x55cd2a053030 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 msgr2=0x55cd2a069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbac4000c00 tx=0x7fbac4005b50 comp rx=0 tx=0).stop 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 shutdown_connections 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a045d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053950 0x55cd2a055a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.446+0000 7fbada79b740 1 --2- 172.21.15.52:0/1175304415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053030 0x55cd2a069da0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.447+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 >> 172.21.15.52:0/1175304415 conn(0x55cd2a04f540 msgr2=0x55cd2a045600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.447+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 shutdown_connections 2024-04-26T02:04:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.447+0000 7fbada79b740 1 -- 172.21.15.52:0/1175304415 wait complete. 2024-04-26T02:04:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.448+0000 7fbada79b740 1 -- start start 2024-04-26T02:04:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.448+0000 7fbada79b740 1 Processor -- start 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a063690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01aa020 con 0x55cd2a05e210 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01ab490 con 0x55cd2a053030 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01aa6f0 con 0x55cd2a053950 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbada79b740 10 client.0 _ll_register_callbacks cb 0x7fbac801f4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbad4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbad4e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59674/0 (socket says 172.21.15.52:59674) 2024-04-26T02:04:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:37988/0 (socket says 172.21.15.52:37988) 2024-04-26T02:04:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbad4e00640 1 -- 172.21.15.52:0/1802027279 learned_addr learned my addr 172.21.15.52:0/1802027279 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.449+0000 7fbad5800640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 -- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 msgr2=0x55cd2a0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 -- 172.21.15.52:0/1802027279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 msgr2=0x55cd2a063690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a063690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 -- 172.21.15.52:0/1802027279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd2a063dd0 con 0x55cd2a053950 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbacfe00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55cd2a05a8b0 tx=0x7fbabc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b351f8e6c825e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbab6a00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.450+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc098c40 con 0x55cd2a053950 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbabc0d4680 con 0x55cd2a053950 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbac4049070 con 0x55cd2a053950 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fbabc09c300 con 0x55cd2a053950 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbada79b740 10 client.27328 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55cd2a0d46c0 con 0x55cd2a053950 2024-04-26T02:04:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.451+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac4049070 con 0x55cd2a053950 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fbabc0dd070 con 0x55cd2a053950 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 10 client.27328 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 10 client.27328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 20 client.27328 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a0d4950 0x55cd2a0d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 4 client.27328 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.452+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd2a070d30 con 0x55cd2a0d4950 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.453+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a0d4950 0x55cd2a0d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.453+0000 7fbab6a00640 10 client.27328 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.453+0000 7fbad4e00640 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a0d4950 0x55cd2a0d6d40 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbafd97afd224063 server_cookie=1f944bcb03a76d5f in_seq=0 out_seq=0 2024-04-26T02:04:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.658+0000 7fbab6a00640 1 -- 172.21.15.52:0/1802027279 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd2a070d30 con 0x55cd2a0d4950 2024-04-26T02:04:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.658+0000 7fbab6a00640 10 client.27328 handle_command_reply: tid=0 2024-04-26T02:04:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 client.27328 shutdown 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 msgr2=0x55cd2a0cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55cd2a05a8b0 tx=0x7fbabc009310 comp rx=0 tx=0).stop 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 shutdown_connections 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55cd2a0d4950 0x55cd2a0d6d40 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cd2a05e210 0x55cd2a063690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55cd2a053950 0x55cd2a0cd420 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 --2- 172.21.15.52:0/1802027279 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55cd2a053030 0x55cd2a0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 >> 172.21.15.52:0/1802027279 conn(0x55cd2a04f540 msgr2=0x55cd2a05b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.659+0000 7fbab6a00640 0 client.27328 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.660+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 shutdown_connections 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.660+0000 7fbada79b740 1 -- 172.21.15.52:0/1802027279 wait complete. 2024-04-26T02:04:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.660+0000 7fbada79b740 20 client.27328 trim_cache size 0 max 16384 2024-04-26T02:04:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:04:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.661+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 msgr2=0x7fbab80a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.661+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 0x7fbab80a6420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005fd0 tx=0x7fbac80096a0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.661+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 msgr2=0x7fbad0101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.661+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00e9f0 tx=0x7fbabc00eeb0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 shutdown_connections 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbab80a3f60 0x7fbab80a6420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbad010c520 0x7fbad01018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbad0103b00 0x7fbad0101370 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 --2- 172.21.15.52:0/906986929 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbad01031e0 0x7fbad0100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 >> 172.21.15.52:0/906986929 conn(0x7fbad0077fd0 msgr2=0x7fbad00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 shutdown_connections 2024-04-26T02:04:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:19.662+0000 7fbad7800640 1 -- 172.21.15.52:0/906986929 wait complete. 2024-04-26T02:04:19.709 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:19 smithi052 ceph-mon[27851]: pgmap v614: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:19.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 112499 17.94 MB/sec execute 216 sec latency 176.980 ms 2024-04-26T02:04:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:19 smithi191 ceph-mon[32745]: pgmap v614: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:19 smithi183 ceph-mon[32752]: pgmap v614: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:20.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 113178 18.07 MB/sec execute 217 sec latency 88.370 ms 2024-04-26T02:04:21.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 113625 18.09 MB/sec execute 218 sec latency 130.253 ms 2024-04-26T02:04:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:21 smithi191 ceph-mon[32745]: pgmap v615: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:21 smithi183 ceph-mon[32752]: pgmap v615: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:21 smithi052 ceph-mon[27851]: pgmap v615: 97 pgs: 97 active+clean; 977 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 553 op/s 2024-04-26T02:04:22.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 114249 18.17 MB/sec execute 219 sec latency 72.189 ms 2024-04-26T02:04:23.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 114740 18.22 MB/sec execute 220 sec latency 103.863 ms 2024-04-26T02:04:23.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:23 smithi191 ceph-mon[32745]: pgmap v616: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 491 op/s 2024-04-26T02:04:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:23 smithi183 ceph-mon[32752]: pgmap v616: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 491 op/s 2024-04-26T02:04:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:23 smithi052 ceph-mon[27851]: pgmap v616: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 491 op/s 2024-04-26T02:04:24.749 INFO:tasks.workunit.client.0.smithi052.stdout: 10 115182 18.21 MB/sec execute 221 sec latency 114.027 ms 2024-04-26T02:04:25.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 115825 18.26 MB/sec execute 222 sec latency 106.089 ms 2024-04-26T02:04:25.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:25 smithi191 ceph-mon[32745]: pgmap v617: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 451 op/s 2024-04-26T02:04:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:25 smithi183 ceph-mon[32752]: pgmap v617: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 451 op/s 2024-04-26T02:04:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:25 smithi052 ceph-mon[27851]: pgmap v617: 97 pgs: 97 active+clean; 990 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 451 op/s 2024-04-26T02:04:26.755 INFO:tasks.workunit.client.0.smithi052.stdout: 10 116118 18.27 MB/sec execute 223 sec latency 448.533 ms 2024-04-26T02:04:27.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 116747 18.35 MB/sec execute 224 sec latency 110.844 ms 2024-04-26T02:04:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:27 smithi191 ceph-mon[32745]: pgmap v618: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 624 op/s 2024-04-26T02:04:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:27 smithi183 ceph-mon[32752]: pgmap v618: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 624 op/s 2024-04-26T02:04:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:27 smithi052 ceph-mon[27851]: pgmap v618: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 624 op/s 2024-04-26T02:04:28.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 117368 18.42 MB/sec execute 225 sec latency 70.443 ms 2024-04-26T02:04:29.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 117962 18.50 MB/sec execute 226 sec latency 82.365 ms 2024-04-26T02:04:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:29 smithi191 ceph-mon[32745]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 484 op/s 2024-04-26T02:04:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:29 smithi183 ceph-mon[32752]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 484 op/s 2024-04-26T02:04:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:29 smithi052 ceph-mon[27851]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 484 op/s 2024-04-26T02:04:30.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 118502 18.53 MB/sec execute 227 sec latency 78.188 ms 2024-04-26T02:04:31.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 118733 18.52 MB/sec execute 228 sec latency 703.729 ms 2024-04-26T02:04:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:31 smithi191 ceph-mon[32745]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 478 op/s 2024-04-26T02:04:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:31 smithi183 ceph-mon[32752]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 478 op/s 2024-04-26T02:04:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:31 smithi052 ceph-mon[27851]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.5 MiB/s wr, 478 op/s 2024-04-26T02:04:32.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 119114 18.51 MB/sec execute 229 sec latency 270.350 ms 2024-04-26T02:04:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.871+0000 7fba42200640 1 -- 172.21.15.52:0/410164411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1043c0 msgr2=0x7fba3c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.871+0000 7fba42200640 1 --2- 172.21.15.52:0/410164411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1043c0 0x7fba3c1047a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fba24009900 tx=0x7fba2402a3f0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.872+0000 7fba42200640 1 -- 172.21.15.52:0/410164411 shutdown_connections 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.872+0000 7fba42200640 1 --2- 172.21.15.52:0/410164411 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c109490 0x7fba3c111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.872+0000 7fba42200640 1 --2- 172.21.15.52:0/410164411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c104d70 0x7fba3c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.872+0000 7fba42200640 1 --2- 172.21.15.52:0/410164411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c1043c0 0x7fba3c1047a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.872+0000 7fba42200640 1 -- 172.21.15.52:0/410164411 >> 172.21.15.52:0/410164411 conn(0x7fba3c0ffc20 msgr2=0x7fba3c102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.873+0000 7fba42200640 1 -- 172.21.15.52:0/410164411 shutdown_connections 2024-04-26T02:04:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.873+0000 7fba42200640 1 -- 172.21.15.52:0/410164411 wait complete. 2024-04-26T02:04:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.874+0000 7fba42200640 1 Processor -- start 2024-04-26T02:04:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.874+0000 7fba42200640 1 -- start start 2024-04-26T02:04:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.875+0000 7fba42200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 0x7fba3c110da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.875+0000 7fba42200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.875+0000 7fba42200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 0x7fba3c10e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34476/0 (socket says 172.21.15.52:34476) 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba42200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c06c270 con 0x7fba3c1043c0 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba42200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c06c0f0 con 0x7fba3c109490 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba42200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c06c3f0 con 0x7fba3c10de90 2024-04-26T02:04:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 0x7fba3c110da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 0x7fba3c10e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 0x7fba3c10e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59714/0 (socket says 172.21.15.52:59714) 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.876+0000 7fba3be00640 1 -- 172.21.15.52:0/3155562655 learned_addr learned my addr 172.21.15.52:0/3155562655 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3aa00640 1 -- 172.21.15.52:0/3155562655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 msgr2=0x7fba3c10e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 0x7fba3c10e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3aa00640 1 -- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 msgr2=0x7fba3c110da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 0x7fba3c110da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3aa00640 1 -- 172.21.15.52:0/3155562655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c10e880 con 0x7fba3c109490 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.877+0000 7fba3b400640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 0x7fba3c110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00ccb0 tx=0x7fba2c00a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba33e00640 1 -- 172.21.15.52:0/3155562655 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00f450 con 0x7fba3c109490 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba33e00640 1 -- 172.21.15.52:0/3155562655 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba2c00faa0 con 0x7fba3c109490 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba33e00640 1 -- 172.21.15.52:0/3155562655 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00ea10 con 0x7fba3c109490 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c10eb10 con 0x7fba3c109490 2024-04-26T02:04:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.878+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1b4240 con 0x7fba3c109490 2024-04-26T02:04:32.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.883+0000 7fba33e00640 1 -- 172.21.15.52:0/3155562655 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fba2c00f5f0 con 0x7fba3c109490 2024-04-26T02:04:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.888+0000 7fba33e00640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 0x7fba200a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.888+0000 7fba33e00640 1 -- 172.21.15.52:0/3155562655 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fba2c0d0c20 con 0x7fba3c109490 2024-04-26T02:04:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.888+0000 7fba3b400640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 0x7fba200a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.890+0000 7fba3b400640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 0x7fba200a6350 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba24000c00 tx=0x7fba24031400 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.890+0000 7fba451b8740 1 Processor -- start 2024-04-26T02:04:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.890+0000 7fba451b8740 1 -- start start 2024-04-26T02:04:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f54a00 0x5610e7f56df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5610e7eaf900 con 0x5610e7f49820 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5610e7e55ef0 con 0x5610e7f4a140 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba451b8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5610e7eab8c0 con 0x5610e7f54a00 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba3b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba3b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49818/0 (socket says 172.21.15.52:49818) 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba3be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f54a00 0x5610e7f56df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba3b400640 1 -- 172.21.15.52:0/4044081324 learned_addr learned my addr 172.21.15.52:0/4044081324 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.891+0000 7fba3aa00640 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 -- 172.21.15.52:0/4044081324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f54a00 msgr2=0x5610e7f56df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f54a00 0x5610e7f56df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 -- 172.21.15.52:0/4044081324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 msgr2=0x5610e7f49c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 -- 172.21.15.52:0/4044081324 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e7bc2210 con 0x5610e7f4a140 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3b400640 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba3aa00640 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f544c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ce370 tx=0x7fba2c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4795d041dae10775 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba2be00640 1 -- 172.21.15.52:0/4044081324 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c09d2c0 con 0x5610e7f4a140 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba2be00640 1 -- 172.21.15.52:0/4044081324 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba2c09cbc0 con 0x5610e7f4a140 2024-04-26T02:04:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.892+0000 7fba2be00640 1 -- 172.21.15.52:0/4044081324 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c099230 con 0x5610e7f4a140 2024-04-26T02:04:32.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 -- 172.21.15.52:0/4044081324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 msgr2=0x5610e7f544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f544c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ce370 tx=0x7fba2c000f40 comp rx=0 tx=0).stop 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 -- 172.21.15.52:0/4044081324 shutdown_connections 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f54a00 0x5610e7f56df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f544c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 --2- 172.21.15.52:0/4044081324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f49820 0x5610e7f49c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 -- 172.21.15.52:0/4044081324 >> 172.21.15.52:0/4044081324 conn(0x5610e7f45d30 msgr2=0x5610e7f46110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 -- 172.21.15.52:0/4044081324 shutdown_connections 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.894+0000 7fba451b8740 1 -- 172.21.15.52:0/4044081324 wait complete. 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.895+0000 7fba451b8740 1 -- start start 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.895+0000 7fba451b8740 1 Processor -- start 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.896+0000 7fba451b8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f51a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.896+0000 7fba451b8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f51f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.896+0000 7fba451b8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f52480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.896+0000 7fba451b8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5610e7f57630 con 0x5610e7f54a00 2024-04-26T02:04:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba451b8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5610e7e69310 con 0x5610e7f4a140 2024-04-26T02:04:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba451b8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5610e7f577b0 con 0x5610e7f49820 2024-04-26T02:04:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba451b8740 10 client.0 _ll_register_callbacks cb 0x5610e7e37cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f51a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f51a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59728/0 (socket says 172.21.15.52:59728) 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba3b400640 1 -- 172.21.15.52:0/1105308354 learned_addr learned my addr 172.21.15.52:0/1105308354 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.897+0000 7fba3aa00640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f51f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 -- 172.21.15.52:0/1105308354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 msgr2=0x5610e7f51a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f51a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 -- 172.21.15.52:0/1105308354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 msgr2=0x5610e7f52480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f52480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 -- 172.21.15.52:0/1105308354 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e7c8e620 con 0x5610e7f4a140 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba28c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.898+0000 7fba3aa00640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f51f40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ce3a0 tx=0x7fba2c00e420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93a5caa704a26c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.899+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0a0f70 con 0x5610e7f4a140 2024-04-26T02:04:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.899+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fba2c099230 con 0x5610e7f4a140 2024-04-26T02:04:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.899+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba2c09cbc0 con 0x5610e7f4a140 2024-04-26T02:04:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.900+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fba2c0a0f70 con 0x5610e7f4a140 2024-04-26T02:04:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.900+0000 7fba451b8740 10 client.27320 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.900+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x5610e7f5a860 con 0x5610e7f4a140 2024-04-26T02:04:32.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba2c09cbc0 con 0x5610e7f4a140 2024-04-26T02:04:32.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fba2c0a53a0 con 0x5610e7f4a140 2024-04-26T02:04:32.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba451b8740 10 client.27320 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:32.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba451b8740 10 client.27320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba451b8740 20 client.27320 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.901+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7f5bcc0 0x5610e7f5e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.902+0000 7fba451b8740 4 client.27320 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.902+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610e7ec38b0 con 0x5610e7f5bcc0 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.902+0000 7fba3b400640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7f5bcc0 0x5610e7f5e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.902+0000 7fba3b400640 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7f5bcc0 0x5610e7f5e120 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=880bbea1d5f41646 server_cookie=6ba36eb33ee54f65 in_seq=0 out_seq=0 2024-04-26T02:04:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.903+0000 7fba28c00640 10 client.27320 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.903+0000 7fba28c00640 1 -- 172.21.15.52:0/1105308354 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610e7ec38b0 con 0x5610e7f5bcc0 2024-04-26T02:04:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.904+0000 7fba28c00640 10 client.27320 handle_command_reply: tid=0 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.904+0000 7fba451b8740 1 client.27320 shutdown 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.904+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 msgr2=0x5610e7f51f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.904+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f51f40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ce3a0 tx=0x7fba2c00e420 comp rx=0 tx=0).stop 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 shutdown_connections 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7f5bcc0 0x5610e7f5e120 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f52480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f51f40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 --2- 172.21.15.52:0/1105308354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f51a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 >> 172.21.15.52:0/1105308354 conn(0x5610e7f45d30 msgr2=0x5610e7f4bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba28c00640 0 client.27320 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 shutdown_connections 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 1 -- 172.21.15.52:0/1105308354 wait complete. 2024-04-26T02:04:32.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.905+0000 7fba451b8740 20 client.27320 trim_cache size 0 max 16384 2024-04-26T02:04:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.941+0000 7fba451b8740 1 Processor -- start 2024-04-26T02:04:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.941+0000 7fba451b8740 1 -- start start 2024-04-26T02:04:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f4c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c1b6b40 con 0x5610e7f54a00 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c1b44d0 con 0x5610e7f4a140 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba451b8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10f370 con 0x5610e7f49820 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59742/0 (socket says 172.21.15.52:59742) 2024-04-26T02:04:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba3b400640 1 -- 172.21.15.52:0/1320599400 learned_addr learned my addr 172.21.15.52:0/1320599400 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.942+0000 7fba3aa00640 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f4c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3be00640 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 -- 172.21.15.52:0/1320599400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 msgr2=0x5610e7f4c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f4c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 -- 172.21.15.52:0/1320599400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 msgr2=0x5610e7f520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 -- 172.21.15.52:0/1320599400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610e7f4ff50 con 0x5610e7f49820 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.943+0000 7fba3b400640 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fba2402a9e0 tx=0x7fba2403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f94f4867a2fbf7ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.944+0000 7fba2b400640 1 -- 172.21.15.52:0/1320599400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24058070 con 0x5610e7f49820 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.944+0000 7fba2b400640 1 -- 172.21.15.52:0/1320599400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba24052770 con 0x5610e7f49820 2024-04-26T02:04:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.944+0000 7fba2b400640 1 -- 172.21.15.52:0/1320599400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24035050 con 0x5610e7f49820 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 -- 172.21.15.52:0/1320599400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 msgr2=0x5610e7f50b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fba2402a9e0 tx=0x7fba2403f000 comp rx=0 tx=0).stop 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 -- 172.21.15.52:0/1320599400 shutdown_connections 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f4c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 --2- 172.21.15.52:0/1320599400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f50b60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.945+0000 7fba451b8740 1 -- 172.21.15.52:0/1320599400 >> 172.21.15.52:0/1320599400 conn(0x5610e7f45d30 msgr2=0x5610e7f519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.946+0000 7fba451b8740 1 -- 172.21.15.52:0/1320599400 shutdown_connections 2024-04-26T02:04:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.946+0000 7fba451b8740 1 -- 172.21.15.52:0/1320599400 wait complete. 2024-04-26T02:04:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.946+0000 7fba451b8740 1 -- start start 2024-04-26T02:04:32.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.946+0000 7fba451b8740 1 Processor -- start 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f5f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f5fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c1b0450 con 0x5610e7f54a00 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c1b18c0 con 0x5610e7f4a140 2024-04-26T02:04:32.946 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c1b0b20 con 0x5610e7f49820 2024-04-26T02:04:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.947+0000 7fba451b8740 10 client.0 _ll_register_callbacks cb 0x7fba2c09ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f5f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:59758/0 (socket says 172.21.15.52:59758) 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 -- 172.21.15.52:0/3326652005 learned_addr learned my addr 172.21.15.52:0/3326652005 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 -- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 msgr2=0x5610e7f5f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f5f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 -- 172.21.15.52:0/3326652005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 msgr2=0x5610e7f5fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f5fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3b400640 1 -- 172.21.15.52:0/3326652005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610e7f60570 con 0x5610e7f49820 2024-04-26T02:04:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.948+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f5f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.950+0000 7fba3b400640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fba24003f50 tx=0x7fba24005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2689618770ca05d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:32.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.952+0000 7fba28c00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:32.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.952+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24058070 con 0x5610e7f49820 2024-04-26T02:04:32.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.952+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fba24052b20 con 0x5610e7f49820 2024-04-26T02:04:32.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.952+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba24052770 con 0x5610e7f49820 2024-04-26T02:04:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.955+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fba24035040 con 0x5610e7f49820 2024-04-26T02:04:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.955+0000 7fba451b8740 10 client.27346 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.955+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x5610e7fec840 con 0x5610e7f49820 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.955+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba24052770 con 0x5610e7f49820 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fba24031c60 con 0x5610e7f49820 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 10 client.27346 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 10 client.27346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 20 client.27346 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7fecee0 0x5610e7fef300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 4 client.27346 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.956+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610e7ebd320 con 0x5610e7fecee0 2024-04-26T02:04:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.957+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7fecee0 0x5610e7fef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.959+0000 7fba3aa00640 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7fecee0 0x5610e7fef300 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f29f91dfcd737 server_cookie=50fe51e20d6bab01 in_seq=0 out_seq=0 2024-04-26T02:04:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:32.959+0000 7fba28c00640 10 client.27346 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.127+0000 7fba28c00640 1 -- 172.21.15.52:0/3326652005 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610e7ebd320 con 0x5610e7fecee0 2024-04-26T02:04:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.127+0000 7fba28c00640 10 client.27346 handle_command_reply: tid=0 2024-04-26T02:04:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.127+0000 7fba451b8740 1 client.27346 shutdown 2024-04-26T02:04:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.128+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 msgr2=0x5610e7f61440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.128+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fba24003f50 tx=0x7fba24005d50 comp rx=0 tx=0).stop 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.128+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 shutdown_connections 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.128+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5610e7fecee0 0x5610e7fef300 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5610e7f54a00 0x5610e7f5fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5610e7f4a140 0x5610e7f5f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 --2- 172.21.15.52:0/3326652005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5610e7f49820 0x5610e7f61440 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 >> 172.21.15.52:0/3326652005 conn(0x5610e7f45d30 msgr2=0x5610e7f52250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba28c00640 0 client.27346 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 shutdown_connections 2024-04-26T02:04:33.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 1 -- 172.21.15.52:0/3326652005 wait complete. 2024-04-26T02:04:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.129+0000 7fba451b8740 20 client.27346 trim_cache size 0 max 16384 2024-04-26T02:04:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:04:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:04:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:04:33.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.130+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 msgr2=0x7fba200a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.130+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 0x7fba200a6350 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba24000c00 tx=0x7fba24031400 comp rx=0 tx=0).stop 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.130+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 msgr2=0x7fba3c073a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.130+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00ccb0 tx=0x7fba2c00a620 comp rx=0 tx=0).stop 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 shutdown_connections 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fba200a3e90 0x7fba200a6350 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba3c10de90 0x7fba3c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fba3c109490 0x7fba3c073a80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 --2- 172.21.15.52:0/3155562655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fba3c1043c0 0x7fba3c110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 >> 172.21.15.52:0/3155562655 conn(0x7fba3c0ffc20 msgr2=0x7fba3c1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 shutdown_connections 2024-04-26T02:04:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:33.131+0000 7fba42200640 1 -- 172.21.15.52:0/3155562655 wait complete. 2024-04-26T02:04:33.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 119591 18.53 MB/sec execute 230 sec latency 207.675 ms 2024-04-26T02:04:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:33 smithi191 ceph-mon[32745]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 746 op/s 2024-04-26T02:04:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:33 smithi183 ceph-mon[32752]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 746 op/s 2024-04-26T02:04:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:33 smithi052 ceph-mon[27851]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 746 op/s 2024-04-26T02:04:34.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 120263 18.62 MB/sec execute 231 sec latency 69.030 ms 2024-04-26T02:04:35.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 120819 18.67 MB/sec execute 232 sec latency 88.642 ms 2024-04-26T02:04:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:35 smithi191 ceph-mon[32745]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 500 op/s 2024-04-26T02:04:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:35 smithi183 ceph-mon[32752]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 500 op/s 2024-04-26T02:04:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:35 smithi052 ceph-mon[27851]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 12 MiB/s wr, 500 op/s 2024-04-26T02:04:36.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 121292 18.69 MB/sec execute 233 sec latency 114.692 ms 2024-04-26T02:04:37.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 121915 18.74 MB/sec execute 234 sec latency 78.318 ms 2024-04-26T02:04:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:37 smithi191 ceph-mon[32745]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 795 op/s 2024-04-26T02:04:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:37 smithi183 ceph-mon[32752]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 795 op/s 2024-04-26T02:04:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:37 smithi052 ceph-mon[27851]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 795 op/s 2024-04-26T02:04:38.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 122559 18.83 MB/sec execute 235 sec latency 104.465 ms 2024-04-26T02:04:39.756 INFO:tasks.workunit.client.0.smithi052.stdout: 10 123113 18.89 MB/sec execute 236 sec latency 111.934 ms 2024-04-26T02:04:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:39 smithi191 ceph-mon[32745]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-04-26T02:04:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:39 smithi183 ceph-mon[32752]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-04-26T02:04:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:39 smithi052 ceph-mon[27851]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 599 op/s 2024-04-26T02:04:40.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 123598 18.89 MB/sec execute 237 sec latency 110.509 ms 2024-04-26T02:04:41.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 124123 18.94 MB/sec execute 238 sec latency 185.463 ms 2024-04-26T02:04:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:41 smithi191 ceph-mon[32745]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-04-26T02:04:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:41 smithi183 ceph-mon[32752]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-04-26T02:04:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:41 smithi052 ceph-mon[27851]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 11 MiB/s wr, 582 op/s 2024-04-26T02:04:42.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 124760 19.00 MB/sec execute 239 sec latency 83.871 ms 2024-04-26T02:04:43.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 125405 19.08 MB/sec execute 240 sec latency 86.312 ms 2024-04-26T02:04:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:43 smithi191 ceph-mon[32745]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 768 op/s 2024-04-26T02:04:43.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:43 smithi183 ceph-mon[32752]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 768 op/s 2024-04-26T02:04:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:43 smithi052 ceph-mon[27851]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 768 op/s 2024-04-26T02:04:44.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 126023 19.13 MB/sec execute 241 sec latency 73.161 ms 2024-04-26T02:04:45.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 126546 19.17 MB/sec execute 242 sec latency 101.584 ms 2024-04-26T02:04:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:45 smithi191 ceph-mon[32745]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 7.2 MiB/s wr, 500 op/s 2024-04-26T02:04:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:45 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:45 smithi183 ceph-mon[32752]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 7.2 MiB/s wr, 500 op/s 2024-04-26T02:04:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:45 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:45.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:45 smithi052 ceph-mon[27851]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 7.2 MiB/s wr, 500 op/s 2024-04-26T02:04:45.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:45 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:04:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 -- 172.21.15.52:0/663337727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3134109480 msgr2=0x7f3134111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 --2- 172.21.15.52:0/663337727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3134109480 0x7f3134111050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f312c009be0 tx=0x7f312c02a6c0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 -- 172.21.15.52:0/663337727 shutdown_connections 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 --2- 172.21.15.52:0/663337727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3134109480 0x7f3134111050 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 --2- 172.21.15.52:0/663337727 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134104d60 0x7f3134108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 --2- 172.21.15.52:0/663337727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 0x7f3134104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 -- 172.21.15.52:0/663337727 >> 172.21.15.52:0/663337727 conn(0x7f31340ffc50 msgr2=0x7f3134102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.331+0000 7f313be00640 1 -- 172.21.15.52:0/663337727 shutdown_connections 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.332+0000 7f313be00640 1 -- 172.21.15.52:0/663337727 wait complete. 2024-04-26T02:04:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 Processor -- start 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 -- start start 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 0x7f31341a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134109480 0x7f31341a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f31341143d0 con 0x7f3134109480 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3134114250 con 0x7f313419a710 2024-04-26T02:04:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.333+0000 7f313be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3134114550 con 0x7f31341043b0 2024-04-26T02:04:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.334+0000 7f3139400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 0x7f31341a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.334+0000 7f3139e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.334+0000 7f3139e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59916/0 (socket says 172.21.15.52:59916) 2024-04-26T02:04:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.334+0000 7f3139e00640 1 -- 172.21.15.52:0/1384522945 learned_addr learned my addr 172.21.15.52:0/1384522945 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.334+0000 7f3138a00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134109480 0x7f31341a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 -- 172.21.15.52:0/1384522945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 msgr2=0x7f31341a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 0x7f31341a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 -- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134109480 msgr2=0x7f31341a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134109480 0x7f31341a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 -- 172.21.15.52:0/1384522945 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313419b310 con 0x7f313419a710 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3139e00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f312c004af0 tx=0x7f312c003c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.335+0000 7f3132000640 1 -- 172.21.15.52:0/1384522945 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c041020 con 0x7f313419a710 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.336+0000 7f3132000640 1 -- 172.21.15.52:0/1384522945 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f312c030970 con 0x7f313419a710 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.336+0000 7f3132000640 1 -- 172.21.15.52:0/1384522945 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c036040 con 0x7f313419a710 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.336+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313419b5a0 con 0x7f313419a710 2024-04-26T02:04:46.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.336+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31341a7640 con 0x7f313419a710 2024-04-26T02:04:46.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.339+0000 7f3132000640 1 -- 172.21.15.52:0/1384522945 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f312c044020 con 0x7f313419a710 2024-04-26T02:04:46.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.339+0000 7f313eb6c740 1 Processor -- start 2024-04-26T02:04:46.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 -- start start 2024-04-26T02:04:46.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735dde6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddfc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55735dd4c900 con 0x55735ddfa210 2024-04-26T02:04:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55735dcf2ef0 con 0x55735dde6760 2024-04-26T02:04:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.340+0000 7f313eb6c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55735dd488c0 con 0x55735dde7080 2024-04-26T02:04:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.341+0000 7f3139400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.341+0000 7f3139400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46318/0 (socket says 172.21.15.52:46318) 2024-04-26T02:04:46.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.341+0000 7f3139400640 1 -- 172.21.15.52:0/3877527570 learned_addr learned my addr 172.21.15.52:0/3877527570 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.341+0000 7f3132000640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 0x7f311c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.341+0000 7f3132000640 1 -- 172.21.15.52:0/1384522945 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f312c0f10e0 con 0x7f313419a710 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139400640 1 -- 172.21.15.52:0/3877527570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 msgr2=0x55735dde6b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139400640 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735dde6b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139400640 1 -- 172.21.15.52:0/3877527570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 msgr2=0x55735ddfc600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T02:04:46.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139e00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 0x7f311c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3138a00640 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddfc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139400640 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddfc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.342+0000 7f3139400640 1 -- 172.21.15.52:0/3877527570 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735da5f210 con 0x55735dde7080 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.343+0000 7f3138a00640 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddfc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.343+0000 7f3139e00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 0x7f311c0a6380 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f312c0c5760 tx=0x7f312c009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.343+0000 7f3139400640 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3128009cd0 tx=0x7f3128011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c403e25a25f6d591 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.344+0000 7f3126600640 1 -- 172.21.15.52:0/3877527570 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312801e070 con 0x55735dde7080 2024-04-26T02:04:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.344+0000 7f3126600640 1 -- 172.21.15.52:0/3877527570 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128019ae0 con 0x55735dde7080 2024-04-26T02:04:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.344+0000 7f3126600640 1 -- 172.21.15.52:0/3877527570 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312800ead0 con 0x55735dde7080 2024-04-26T02:04:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.345+0000 7f313eb6c740 1 -- 172.21.15.52:0/3877527570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 msgr2=0x55735ddf9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.345+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3128009cd0 tx=0x7f3128011810 comp rx=0 tx=0).stop 2024-04-26T02:04:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.346+0000 7f313eb6c740 1 -- 172.21.15.52:0/3877527570 shutdown_connections 2024-04-26T02:04:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.346+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddfc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.346+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735ddf9cd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.346+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3877527570 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735dde6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.346+0000 7f313eb6c740 1 -- 172.21.15.52:0/3877527570 >> 172.21.15.52:0/3877527570 conn(0x55735dde2d30 msgr2=0x55735dde3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.347+0000 7f313eb6c740 1 -- 172.21.15.52:0/3877527570 shutdown_connections 2024-04-26T02:04:46.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.348+0000 7f313eb6c740 1 -- 172.21.15.52:0/3877527570 wait complete. 2024-04-26T02:04:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.349+0000 7f313eb6c740 1 -- start start 2024-04-26T02:04:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.349+0000 7f313eb6c740 1 Processor -- start 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde7080 0x55735ddf7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddf7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55735ddfcde0 con 0x55735ddfa210 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55735dd06310 con 0x55735dde7080 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55735ddfcf60 con 0x55735dde6760 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f313eb6c740 10 client.0 _ll_register_callbacks cb 0x55735dcd4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f3138a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f3138a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46326/0 (socket says 172.21.15.52:46326) 2024-04-26T02:04:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f3138a00640 1 -- 172.21.15.52:0/3399806547 learned_addr learned my addr 172.21.15.52:0/3399806547 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f3138a00640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddf7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.350+0000 7f3139400640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde7080 0x55735ddf7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 -- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde7080 msgr2=0x55735ddf7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde7080 0x55735ddf7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 -- 172.21.15.52:0/3399806547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 msgr2=0x55735ddf7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddf7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 -- 172.21.15.52:0/3399806547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55735db2b620 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f3138a00640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f313419bf10 tx=0x7f312000d480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb0cf27bb1f20af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f311aa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120004030 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3128019ae0 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.351+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f31200041d0 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f31200043d0 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 10 client.27358 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55735de09130 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128019ae0 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f31200047a0 con 0x55735dde6760 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 10 client.27358 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 10 client.27358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 20 client.27358 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:46.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de0a5a0 0x55735de0c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 4 client.27358 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.352+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55735dd608b0 con 0x55735de0a5a0 2024-04-26T02:04:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.353+0000 7f3139400640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de0a5a0 0x55735de0c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.354+0000 7f3139400640 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de0a5a0 0x55735de0c990 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c967e687e3039a65 server_cookie=8dc790a31ab59517 in_seq=0 out_seq=0 2024-04-26T02:04:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.354+0000 7f311aa00640 10 client.27358 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f311aa00640 1 -- 172.21.15.52:0/3399806547 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55735dd608b0 con 0x55735de0a5a0 2024-04-26T02:04:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f311aa00640 10 client.27358 handle_command_reply: tid=0 2024-04-26T02:04:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 client.27358 shutdown 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 msgr2=0x55735ddf71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f313419bf10 tx=0x7f312000d480 comp rx=0 tx=0).stop 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 shutdown_connections 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de0a5a0 0x55735de0c990 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.355+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735ddfa210 0x55735ddf7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde7080 0x55735ddf7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3399806547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde6760 0x55735ddf71f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 >> 172.21.15.52:0/3399806547 conn(0x55735dde2d30 msgr2=0x55735dde8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f311aa00640 0 client.27358 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 shutdown_connections 2024-04-26T02:04:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 1 -- 172.21.15.52:0/3399806547 wait complete. 2024-04-26T02:04:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.356+0000 7f313eb6c740 20 client.27358 trim_cache size 0 max 16384 2024-04-26T02:04:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.379+0000 7f313eb6c740 1 Processor -- start 2024-04-26T02:04:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 -- start start 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 0x55735dde3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735ddfa210 0x55735de121b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f31341ae270 con 0x55735dde7080 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f31341abb40 con 0x55735dde6760 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f313eb6c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f31341a99b0 con 0x55735ddfa210 2024-04-26T02:04:46.379 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f3138a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f3138a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:59944/0 (socket says 172.21.15.52:59944) 2024-04-26T02:04:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.380+0000 7f3138a00640 1 -- 172.21.15.52:0/75290430 learned_addr learned my addr 172.21.15.52:0/75290430 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3138a00640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735ddfa210 0x55735de121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 0x55735dde3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 -- 172.21.15.52:0/75290430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735ddfa210 msgr2=0x55735de121b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735ddfa210 0x55735de121b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 -- 172.21.15.52:0/75290430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 msgr2=0x55735de05ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 -- 172.21.15.52:0/75290430 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735ddf6200 con 0x55735dde7080 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3138a00640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.381+0000 7f3139400640 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 0x55735dde3110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f31280098d0 tx=0x7f312800e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e7b3b307efe661 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.382+0000 7f3125c00640 1 -- 172.21.15.52:0/75290430 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312801e040 con 0x55735dde7080 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.382+0000 7f3125c00640 1 -- 172.21.15.52:0/75290430 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128005450 con 0x55735dde7080 2024-04-26T02:04:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.382+0000 7f3125c00640 1 -- 172.21.15.52:0/75290430 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312800e6a0 con 0x55735dde7080 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 -- 172.21.15.52:0/75290430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 msgr2=0x55735dde3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 0x55735dde3110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f31280098d0 tx=0x7f312800e630 comp rx=0 tx=0).stop 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 -- 172.21.15.52:0/75290430 shutdown_connections 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735ddfa210 0x55735de121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde7080 0x55735dde3110 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 --2- 172.21.15.52:0/75290430 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735dde6760 0x55735de05ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 -- 172.21.15.52:0/75290430 >> 172.21.15.52:0/75290430 conn(0x55735dde2d30 msgr2=0x55735dde1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 -- 172.21.15.52:0/75290430 shutdown_connections 2024-04-26T02:04:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.383+0000 7f313eb6c740 1 -- 172.21.15.52:0/75290430 wait complete. 2024-04-26T02:04:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.384+0000 7f313eb6c740 1 -- start start 2024-04-26T02:04:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.384+0000 7f313eb6c740 1 Processor -- start 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 0x55735ddffe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735ddfa210 0x55735de008b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f31341aa000 con 0x55735dde6760 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f31341ab470 con 0x55735ddfa210 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.385+0000 7f313eb6c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f31341aa6d0 con 0x55735dde7080 2024-04-26T02:04:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f313eb6c740 10 client.0 _ll_register_callbacks cb 0x7f31200126c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:46346/0 (socket says 172.21.15.52:46346) 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 -- 172.21.15.52:0/3154009710 learned_addr learned my addr 172.21.15.52:0/3154009710 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3138a00640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 0x55735ddffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3138a00640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735ddfa210 0x55735de008b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 -- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735ddfa210 msgr2=0x55735de008b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735ddfa210 0x55735de008b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 -- 172.21.15.52:0/3154009710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 msgr2=0x55735ddffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 0x55735ddffe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3139400640 1 -- 172.21.15.52:0/3154009710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55735ddf1fe0 con 0x55735dde7080 2024-04-26T02:04:46.386 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.386+0000 7f3138a00640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 0x55735ddffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:04:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.387+0000 7f3139400640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f3128009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a871a74359745c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.387+0000 7f311aa00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.387+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128030030 con 0x55735dde7080 2024-04-26T02:04:46.387 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.387+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3128005450 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.387+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f3128005250 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f312801e040 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 10 client.27364 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55735de70d60 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128005450 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f312800e7e0 con 0x55735dde7080 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 10 client.27364 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 10 client.27364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 20 client.27364 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de71180 0x55735de73570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 4 client.27364 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.388+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55735dd5a320 con 0x55735de71180 2024-04-26T02:04:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.389+0000 7f3138a00640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de71180 0x55735de73570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.389+0000 7f3138a00640 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de71180 0x55735de73570 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1017f783ea966a51 server_cookie=45801a6324899781 in_seq=0 out_seq=0 2024-04-26T02:04:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.389+0000 7f311aa00640 10 client.27364 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:46.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f311aa00640 1 -- 172.21.15.52:0/3154009710 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55735dd5a320 con 0x55735de71180 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f311aa00640 10 client.27364 handle_command_reply: tid=0 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 client.27364 shutdown 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 msgr2=0x55735de00370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3128006010 tx=0x7f3128009f40 comp rx=0 tx=0).stop 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 shutdown_connections 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55735de71180 0x55735de73570 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.630+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55735ddfa210 0x55735de008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55735dde7080 0x55735de00370 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f313eb6c740 1 --2- 172.21.15.52:0/3154009710 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55735dde6760 0x55735ddffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 >> 172.21.15.52:0/3154009710 conn(0x55735dde2d30 msgr2=0x55735dde1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f311aa00640 0 client.27364 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:46.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 shutdown_connections 2024-04-26T02:04:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.631+0000 7f313eb6c740 1 -- 172.21.15.52:0/3154009710 wait complete. 2024-04-26T02:04:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.632+0000 7f313eb6c740 20 client.27364 trim_cache size 0 max 16384 2024-04-26T02:04:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:04:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:04:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:04:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:04:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 msgr2=0x7f311c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 0x7f311c0a6380 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f312c0c5760 tx=0x7f312c009500 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 msgr2=0x7f313419ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f312c004af0 tx=0x7f312c003c90 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 shutdown_connections 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f311c0a3ec0 0x7f311c0a6380 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f313419a710 0x7f313419ab90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3134109480 0x7f31341a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 --2- 172.21.15.52:0/1384522945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31341043b0 0x7f31341a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.637+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 >> 172.21.15.52:0/1384522945 conn(0x7f31340ffc50 msgr2=0x7f3134101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.638+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 shutdown_connections 2024-04-26T02:04:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:46.638+0000 7f313be00640 1 -- 172.21.15.52:0/1384522945 wait complete. 2024-04-26T02:04:46.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 126887 19.17 MB/sec execute 243 sec latency 275.669 ms 2024-04-26T02:04:47.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 127479 19.21 MB/sec execute 244 sec latency 169.077 ms 2024-04-26T02:04:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:47 smithi191 ceph-mon[32745]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 9.8 MiB/s wr, 666 op/s 2024-04-26T02:04:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:47 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:47 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:47 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:04:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:47 smithi183 ceph-mon[32752]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 9.8 MiB/s wr, 666 op/s 2024-04-26T02:04:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:47 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:47 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:47 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:04:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:47 smithi052 ceph-mon[27851]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 9.8 MiB/s wr, 666 op/s 2024-04-26T02:04:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:47 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:04:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:47 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:04:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:47 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:04:48.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 128171 19.30 MB/sec execute 245 sec latency 88.129 ms 2024-04-26T02:04:49.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 128772 19.36 MB/sec execute 246 sec latency 67.141 ms 2024-04-26T02:04:49.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:49 smithi191 ceph-mon[32745]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 389 op/s 2024-04-26T02:04:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:49 smithi183 ceph-mon[32752]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 389 op/s 2024-04-26T02:04:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:49 smithi052 ceph-mon[27851]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 389 op/s 2024-04-26T02:04:50.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 129388 19.42 MB/sec execute 247 sec latency 62.729 ms 2024-04-26T02:04:51.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 129899 19.44 MB/sec execute 248 sec latency 87.940 ms 2024-04-26T02:04:51.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:51 smithi191 ceph-mon[32745]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 369 op/s 2024-04-26T02:04:51.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:51 smithi183 ceph-mon[32752]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 369 op/s 2024-04-26T02:04:51.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:51 smithi052 ceph-mon[27851]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 369 op/s 2024-04-26T02:04:52.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 130380 19.43 MB/sec execute 249 sec latency 256.158 ms 2024-04-26T02:04:53.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 131024 19.50 MB/sec execute 250 sec latency 93.817 ms 2024-04-26T02:04:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:53 smithi191 ceph-mon[32745]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 652 op/s 2024-04-26T02:04:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:53 smithi183 ceph-mon[32752]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 652 op/s 2024-04-26T02:04:53.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:53 smithi052 ceph-mon[27851]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 9.1 MiB/s wr, 652 op/s 2024-04-26T02:04:54.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 131703 19.59 MB/sec execute 251 sec latency 80.096 ms 2024-04-26T02:04:55.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 132002 19.59 MB/sec execute 252 sec latency 432.172 ms 2024-04-26T02:04:55.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:55 smithi191 ceph-mon[32745]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 466 op/s 2024-04-26T02:04:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:55 smithi183 ceph-mon[32752]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 466 op/s 2024-04-26T02:04:55.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:55 smithi052 ceph-mon[27851]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 466 op/s 2024-04-26T02:04:56.757 INFO:tasks.workunit.client.0.smithi052.stdout: 10 132502 19.61 MB/sec execute 253 sec latency 116.068 ms 2024-04-26T02:04:57.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 132611 19.56 MB/sec execute 254 sec latency 572.396 ms 2024-04-26T02:04:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:57 smithi191 ceph-mon[32745]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-04-26T02:04:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:57 smithi183 ceph-mon[32752]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-04-26T02:04:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:57 smithi052 ceph-mon[27851]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 7.7 MiB/s wr, 594 op/s 2024-04-26T02:04:58.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 133082 19.58 MB/sec execute 255 sec latency 140.120 ms 2024-04-26T02:04:59.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 133553 19.60 MB/sec execute 256 sec latency 110.721 ms 2024-04-26T02:04:59.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 -- 172.21.15.52:0/2758060798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff0109490 msgr2=0x7fbff0111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 --2- 172.21.15.52:0/2758060798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff0109490 0x7fbff0111060 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009900 tx=0x7fbfe402a3f0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 -- 172.21.15.52:0/2758060798 shutdown_connections 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 --2- 172.21.15.52:0/2758060798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff0109490 0x7fbff0111060 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 --2- 172.21.15.52:0/2758060798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0104d70 0x7fbff0108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 --2- 172.21.15.52:0/2758060798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff01043c0 0x7fbff01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.824+0000 7fbff8600640 1 -- 172.21.15.52:0/2758060798 >> 172.21.15.52:0/2758060798 conn(0x7fbff00ffc20 msgr2=0x7fbff0102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.825+0000 7fbff8600640 1 -- 172.21.15.52:0/2758060798 shutdown_connections 2024-04-26T02:04:59.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.825+0000 7fbff8600640 1 -- 172.21.15.52:0/2758060798 wait complete. 2024-04-26T02:04:59.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 Processor -- start 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 -- start start 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff01043c0 0x7fbff01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0114430 con 0x7fbff0109490 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01142b0 con 0x7fbff019a6b0 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.826+0000 7fbff8600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01145b0 con 0x7fbff01043c0 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff5200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37738/0 (socket says 172.21.15.52:37738) 2024-04-26T02:04:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 -- 172.21.15.52:0/1228791483 learned_addr learned my addr 172.21.15.52:0/1228791483 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff5200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44000/0 (socket says 172.21.15.52:44000) 2024-04-26T02:04:59.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff5c00640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff01043c0 0x7fbff01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 -- 172.21.15.52:0/1228791483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff01043c0 msgr2=0x7fbff01a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff01043c0 0x7fbff01a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.827+0000 7fbff6600640 1 -- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 msgr2=0x7fbff01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff6600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff6600640 1 -- 172.21.15.52:0/1228791483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff019b280 con 0x7fbff019a6b0 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff5200640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff6600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4003cd0 tx=0x7fbfe4003d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbfee000640 1 -- 172.21.15.52:0/1228791483 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe403f070 con 0x7fbff019a6b0 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff019b510 con 0x7fbff019a6b0 2024-04-26T02:04:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbfee000640 1 -- 172.21.15.52:0/1228791483 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe4003ea0 con 0x7fbff019a6b0 2024-04-26T02:04:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbfee000640 1 -- 172.21.15.52:0/1228791483 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4035030 con 0x7fbff019a6b0 2024-04-26T02:04:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.828+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff006bcb0 con 0x7fbff019a6b0 2024-04-26T02:04:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.832+0000 7fbffb5bd740 1 Processor -- start 2024-04-26T02:04:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbfee000640 1 -- 172.21.15.52:0/1228791483 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbfe4042070 con 0x7fbff019a6b0 2024-04-26T02:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 -- start start 2024-04-26T02:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee98066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98067140 0x55ee980714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ee97fcc900 con 0x55ee98066820 2024-04-26T02:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ee97f72ef0 con 0x55ee98067140 2024-04-26T02:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.833+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ee97fc88c0 con 0x55ee98071a00 2024-04-26T02:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98067140 0x55ee980714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36354/0 (socket says 172.21.15.52:36354) 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 -- 172.21.15.52:0/3834219390 learned_addr learned my addr 172.21.15.52:0/3834219390 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 -- 172.21.15.52:0/3834219390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98067140 msgr2=0x55ee980714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98067140 0x55ee980714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 -- 172.21.15.52:0/3834219390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 msgr2=0x55ee98066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee98066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.834+0000 7fbff5200640 1 -- 172.21.15.52:0/3834219390 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee97cdf210 con 0x55ee98071a00 2024-04-26T02:04:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbff5200640 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc012440 tx=0x7fbfdc012900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492e6e3469d20a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbfee000640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 0x7fbfd80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbfe2a00640 1 -- 172.21.15.52:0/3834219390 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc004220 con 0x55ee98071a00 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbfee000640 1 -- 172.21.15.52:0/1228791483 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbfe40f0f50 con 0x7fbff019a6b0 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbfe2a00640 1 -- 172.21.15.52:0/3834219390 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfdc0043c0 con 0x55ee98071a00 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.835+0000 7fbfe2a00640 1 -- 172.21.15.52:0/3834219390 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0109b0 con 0x55ee98071a00 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbff5c00640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 0x7fbfd80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbffb5bd740 1 -- 172.21.15.52:0/3834219390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 msgr2=0x55ee98073df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc012440 tx=0x7fbfdc012900 comp rx=0 tx=0).stop 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbffb5bd740 1 -- 172.21.15.52:0/3834219390 shutdown_connections 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98071a00 0x55ee98073df0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.836+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98067140 0x55ee980714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.837+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/3834219390 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee98066c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.837+0000 7fbffb5bd740 1 -- 172.21.15.52:0/3834219390 >> 172.21.15.52:0/3834219390 conn(0x55ee98062d30 msgr2=0x55ee98063110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.837+0000 7fbff5c00640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 0x7fbfd80a6380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55ee98067dc0 tx=0x7fbfe80075b0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.837+0000 7fbffb5bd740 1 -- 172.21.15.52:0/3834219390 shutdown_connections 2024-04-26T02:04:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.838+0000 7fbffb5bd740 1 -- 172.21.15.52:0/3834219390 wait complete. 2024-04-26T02:04:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.839+0000 7fbffb5bd740 1 -- start start 2024-04-26T02:04:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.839+0000 7fbffb5bd740 1 Processor -- start 2024-04-26T02:04:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.842+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee9806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 0x55ee9806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbff5c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee9806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36362/0 (socket says 172.21.15.52:36362) 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbff5200640 1 -- 172.21.15.52:0/819643672 learned_addr learned my addr 172.21.15.52:0/819643672 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ee98074630 con 0x55ee98066820 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55ee97f86310 con 0x55ee98071a00 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ee980747b0 con 0x55ee98067140 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.843+0000 7fbff6600640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 0x55ee9806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbffb5bd740 10 client.0 _ll_register_callbacks cb 0x55ee97f54cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5200640 1 -- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 msgr2=0x55ee9806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5200640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 0x55ee9806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5200640 1 -- 172.21.15.52:0/819643672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 msgr2=0x55ee9806e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5200640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee9806e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5200640 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee97dab620 con 0x55ee98067140 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff6600640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 0x55ee9806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbff5c00640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee9806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.844+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfdc0043c0 con 0x55ee98067140 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbff5200640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00bfd0 tx=0x7fbfdc0042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d55fe960c2298466 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbfd7400640 10 client.27376 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbfd7400640 1 -- 172.21.15.52:0/819643672 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc028020 con 0x55ee98067140 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbfd7400640 1 -- 172.21.15.52:0/819643672 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbfdc003930 con 0x55ee98067140 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbfd7400640 1 -- 172.21.15.52:0/819643672 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fbfdc01e040 con 0x55ee98067140 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbffb5bd740 10 client.27376 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.845+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({fsmap=70}) -- 0x55ee98088df0 con 0x55ee98067140 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbfd7400640 1 -- 172.21.15.52:0/819643672 <== mon.1 v2:172.21.15.191:3300/0 4 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fbfdc021030 con 0x55ee98067140 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 10 client.27376 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 10 client.27376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 20 client.27376 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee9808a140 0x55ee9808c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 4 client.27376 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee97fe08b0 con 0x55ee9808a140 2024-04-26T02:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.847+0000 7fbff5c00640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee9808a140 0x55ee9808c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.848+0000 7fbff5c00640 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee9808a140 0x55ee9808c530 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f92c9a17d9cb1b1 server_cookie=ae090a636ef0aaf8 in_seq=0 out_seq=0 2024-04-26T02:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.848+0000 7fbfd7400640 10 client.27376 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.849+0000 7fbfd7400640 1 -- 172.21.15.52:0/819643672 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee97fe08b0 con 0x55ee9808a140 2024-04-26T02:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.849+0000 7fbfd7400640 10 client.27376 handle_command_reply: tid=0 2024-04-26T02:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.849+0000 7fbffb5bd740 1 client.27376 shutdown 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.849+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 msgr2=0x55ee9806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.849+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00bfd0 tx=0x7fbfdc0042d0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 shutdown_connections 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee9808a140 0x55ee9808c530 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98071a00 0x55ee9806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee98067140 0x55ee9806ef00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/819643672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98066820 0x55ee9806e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 >> 172.21.15.52:0/819643672 conn(0x55ee98062d30 msgr2=0x55ee98068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbfd7400640 0 client.27376 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 shutdown_connections 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 1 -- 172.21.15.52:0/819643672 wait complete. 2024-04-26T02:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.850+0000 7fbffb5bd740 20 client.27376 trim_cache size 0 max 16384 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.875+0000 7fbffb5bd740 1 Processor -- start 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 -- start start 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98085d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee98091c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0072750 con 0x55ee98071a00 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0070020 con 0x55ee98066820 2024-04-26T02:04:59.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbff006de90 con 0x55ee9806d710 2024-04-26T02:04:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbff5200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbff5200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44034/0 (socket says 172.21.15.52:44034) 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.876+0000 7fbff5200640 1 -- 172.21.15.52:0/2386015004 learned_addr learned my addr 172.21.15.52:0/2386015004 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff6600640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee98091c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5c00640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98085d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 -- 172.21.15.52:0/2386015004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 msgr2=0x55ee98091c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee98091c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 -- 172.21.15.52:0/2386015004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 msgr2=0x55ee98085d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98085d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 -- 172.21.15.52:0/2386015004 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee98063ec0 con 0x55ee98071a00 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff6600640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee98091c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5c00640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98085d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.877+0000 7fbff5200640 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc005d20 tx=0x7fbfdc024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecb9bed37a819fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.878+0000 7fbfe2000640 1 -- 172.21.15.52:0/2386015004 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc028070 con 0x55ee98071a00 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.878+0000 7fbfe2000640 1 -- 172.21.15.52:0/2386015004 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfdc0043c0 con 0x55ee98071a00 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.878+0000 7fbfe2000640 1 -- 172.21.15.52:0/2386015004 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc003990 con 0x55ee98071a00 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 -- 172.21.15.52:0/2386015004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 msgr2=0x55ee9806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc005d20 tx=0x7fbfdc024000 comp rx=0 tx=0).stop 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 -- 172.21.15.52:0/2386015004 shutdown_connections 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee98091c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee9806f260 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/2386015004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98085d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.879+0000 7fbffb5bd740 1 -- 172.21.15.52:0/2386015004 >> 172.21.15.52:0/2386015004 conn(0x55ee98062d30 msgr2=0x55ee980701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:04:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.880+0000 7fbffb5bd740 1 -- 172.21.15.52:0/2386015004 shutdown_connections 2024-04-26T02:04:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.880+0000 7fbffb5bd740 1 -- 172.21.15.52:0/2386015004 wait complete. 2024-04-26T02:04:59.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.880+0000 7fbffb5bd740 1 -- start start 2024-04-26T02:04:59.880 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.881+0000 7fbffb5bd740 1 Processor -- start 2024-04-26T02:04:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.882+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98080090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee980ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbff006e4e0 con 0x55ee98071a00 2024-04-26T02:04:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbff006f950 con 0x55ee98066820 2024-04-26T02:04:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbff006ebb0 con 0x55ee9806d710 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbffb5bd740 10 client.0 _ll_register_callbacks cb 0x7fbfdc005870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbff5200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:36384/0 (socket says 172.21.15.52:36384) 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbff5200640 1 -- 172.21.15.52:0/603719897 learned_addr learned my addr 172.21.15.52:0/603719897 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbff6600640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee980ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.883+0000 7fbff5c00640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98080090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 -- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 msgr2=0x55ee98080090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98080090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 -- 172.21.15.52:0/603719897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 msgr2=0x55ee980ef4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee980ef4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 -- 172.21.15.52:0/603719897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee980ef9e0 con 0x55ee9806d710 2024-04-26T02:04:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff6600640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee980ef4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5c00640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98080090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbff5200640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b6e0 tx=0x7fbfdc003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bc8f4b35d4ef48a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbfd7400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.884+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc028040 con 0x55ee9806d710 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7fbfdc010900 con 0x55ee9806d710 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfdc0043c0 con 0x55ee9806d710 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fbfdc010b00 con 0x55ee9806d710 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbffb5bd740 10 client.27382 fetch_fsmap learned FSMap version 70 2024-04-26T02:04:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55ee980f0b50 con 0x55ee9806d710 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.885+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfdc010ca0 con 0x55ee9806d710 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7fbfdc021070 con 0x55ee9806d710 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 10 client.27382 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 10 client.27382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 20 client.27382 populate_metadata read hostname 'smithi052' 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee980f1160 0x55ee980f3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 4 client.27382 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fbfdc0043c0 con 0x55ee980f1160 2024-04-26T02:04:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.886+0000 7fbff5c00640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee980f1160 0x55ee980f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:04:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.888+0000 7fbff5c00640 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee980f1160 0x55ee980f3550 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b81c6b94c347221d server_cookie=9c245f753f5f242c in_seq=0 out_seq=0 2024-04-26T02:04:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:04:59.888+0000 7fbfd7400640 10 client.27382 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:04:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:04:59 smithi191 ceph-mon[32745]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 468 op/s 2024-04-26T02:04:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:04:59 smithi183 ceph-mon[32752]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 468 op/s 2024-04-26T02:04:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:04:59 smithi052 ceph-mon[27851]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 468 op/s 2024-04-26T02:05:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.109+0000 7fbfd7400640 1 -- 172.21.15.52:0/603719897 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fbfdc0043c0 con 0x55ee980f1160 2024-04-26T02:05:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.109+0000 7fbfd7400640 10 client.27382 handle_command_reply: tid=0 2024-04-26T02:05:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.109+0000 7fbffb5bd740 1 client.27382 shutdown 2024-04-26T02:05:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.110+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 msgr2=0x55ee980805d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.110+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b6e0 tx=0x7fbfdc003960 comp rx=0 tx=0).stop 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 shutdown_connections 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55ee980f1160 0x55ee980f3550 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ee98071a00 0x55ee980ef4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ee9806d710 0x55ee980805d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 --2- 172.21.15.52:0/603719897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55ee98066820 0x55ee98080090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbfd7400640 0 client.27382 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 >> 172.21.15.52:0/603719897 conn(0x55ee98062d30 msgr2=0x55ee98070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 shutdown_connections 2024-04-26T02:05:00.112 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 1 -- 172.21.15.52:0/603719897 wait complete. 2024-04-26T02:05:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.113+0000 7fbffb5bd740 20 client.27382 trim_cache size 0 max 16384 2024-04-26T02:05:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:05:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:05:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:05:00.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:05:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 msgr2=0x7fbfd80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 0x7fbfd80a6380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55ee98067dc0 tx=0x7fbfe80075b0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 msgr2=0x7fbff019ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4003cd0 tx=0x7fbfe4003d00 comp rx=0 tx=0).stop 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 shutdown_connections 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fbfd80a3ec0 0x7fbfd80a6380 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbff019a6b0 0x7fbff019ab30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbff0109490 0x7fbff01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 --2- 172.21.15.52:0/1228791483 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbff01043c0 0x7fbff01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 >> 172.21.15.52:0/1228791483 conn(0x7fbff00ffc20 msgr2=0x7fbff01006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 shutdown_connections 2024-04-26T02:05:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:00.116+0000 7fbff8600640 1 -- 172.21.15.52:0/1228791483 wait complete. 2024-04-26T02:05:00.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 133767 19.57 MB/sec execute 257 sec latency 413.785 ms 2024-04-26T02:05:01.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 134034 19.53 MB/sec execute 258 sec latency 676.921 ms 2024-04-26T02:05:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:01 smithi191 ceph-mon[32745]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 450 op/s 2024-04-26T02:05:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:01 smithi183 ceph-mon[32752]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 450 op/s 2024-04-26T02:05:01.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:01 smithi052 ceph-mon[27851]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 450 op/s 2024-04-26T02:05:02.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 134354 19.49 MB/sec execute 259 sec latency 301.824 ms 2024-04-26T02:05:03.758 INFO:tasks.workunit.client.0.smithi052.stdout: 10 134481 19.44 MB/sec execute 260 sec latency 861.527 ms 2024-04-26T02:05:03.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:03 smithi191 ceph-mon[32745]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 839 op/s 2024-04-26T02:05:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:03 smithi183 ceph-mon[32752]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 839 op/s 2024-04-26T02:05:03.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:03 smithi052 ceph-mon[27851]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 839 op/s 2024-04-26T02:05:04.759 INFO:tasks.workunit.client.0.smithi052.stdout: 10 134759 19.44 MB/sec execute 261 sec latency 360.847 ms 2024-04-26T02:05:05.760 INFO:tasks.workunit.client.0.smithi052.stdout: 10 135134 19.46 MB/sec execute 262 sec latency 196.875 ms 2024-04-26T02:05:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:05 smithi191 ceph-mon[32745]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 556 op/s 2024-04-26T02:05:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:05 smithi183 ceph-mon[32752]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 556 op/s 2024-04-26T02:05:05.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:05 smithi052 ceph-mon[27851]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 10 MiB/s wr, 556 op/s 2024-04-26T02:05:06.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 135692 19.51 MB/sec execute 263 sec latency 343.075 ms 2024-04-26T02:05:07.759 INFO:tasks.workunit.client.0.smithi052.stdout: 10 136356 19.58 MB/sec execute 264 sec latency 67.272 ms 2024-04-26T02:05:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:07 smithi191 ceph-mon[32745]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.05k op/s 2024-04-26T02:05:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:07 smithi183 ceph-mon[32752]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.05k op/s 2024-04-26T02:05:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:07 smithi052 ceph-mon[27851]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.05k op/s 2024-04-26T02:05:08.760 INFO:tasks.workunit.client.0.smithi052.stdout: 10 136918 19.64 MB/sec execute 265 sec latency 89.906 ms 2024-04-26T02:05:09.760 INFO:tasks.workunit.client.0.smithi052.stdout: 10 137363 19.63 MB/sec execute 266 sec latency 98.585 ms 2024-04-26T02:05:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:09 smithi191 ceph-mon[32745]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-04-26T02:05:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:09 smithi183 ceph-mon[32752]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-04-26T02:05:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:09 smithi052 ceph-mon[27851]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 18 MiB/s wr, 960 op/s 2024-04-26T02:05:10.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 137990 19.68 MB/sec execute 267 sec latency 96.811 ms 2024-04-26T02:05:11.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 138554 19.73 MB/sec execute 268 sec latency 156.400 ms 2024-04-26T02:05:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:11 smithi191 ceph-mon[32745]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 920 op/s 2024-04-26T02:05:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:11 smithi183 ceph-mon[32752]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 920 op/s 2024-04-26T02:05:11.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:11 smithi052 ceph-mon[27851]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 920 op/s 2024-04-26T02:05:12.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 139113 19.75 MB/sec execute 269 sec latency 75.054 ms 2024-04-26T02:05:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.290+0000 7f0cf0c00640 1 -- 172.21.15.52:0/2199105533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec102b80 msgr2=0x7f0cec106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.290+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/2199105533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec102b80 0x7f0cec106bc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009900 tx=0x7f0cd402a3f0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 -- 172.21.15.52:0/2199105533 shutdown_connections 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/2199105533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cec1072a0 0x7f0cec10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/2199105533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec102b80 0x7f0cec106bc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/2199105533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1021d0 0x7f0cec1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 -- 172.21.15.52:0/2199105533 >> 172.21.15.52:0/2199105533 conn(0x7f0cec0793e0 msgr2=0x7f0cec0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 -- 172.21.15.52:0/2199105533 shutdown_connections 2024-04-26T02:05:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.291+0000 7f0cf0c00640 1 -- 172.21.15.52:0/2199105533 wait complete. 2024-04-26T02:05:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.292+0000 7f0cf0c00640 1 Processor -- start 2024-04-26T02:05:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.292+0000 7f0cf0c00640 1 -- start start 2024-04-26T02:05:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1072a0 0x7f0cec1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cec19a720 0x7f0cec19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec112230 con 0x7f0cec19a720 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec1120b0 con 0x7f0cec1021d0 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.293+0000 7f0cf0c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec1123b0 con 0x7f0cec1072a0 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49230/0 (socket says 172.21.15.52:49230) 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 -- 172.21.15.52:0/1135356267 learned_addr learned my addr 172.21.15.52:0/1135356267 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1072a0 0x7f0cec1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 -- 172.21.15.52:0/1135356267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1072a0 msgr2=0x7f0cec1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1072a0 0x7f0cec1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 -- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cec19a720 msgr2=0x7f0cec19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cec19a720 0x7f0cec19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 -- 172.21.15.52:0/1135356267 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cec19b320 con 0x7f0cec1021d0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00e9f0 tx=0x7f0cdc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce2a00640 1 -- 172.21.15.52:0/1135356267 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00cd80 con 0x7f0cec1021d0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.294+0000 7f0ce2a00640 1 -- 172.21.15.52:0/1135356267 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cdc0054d0 con 0x7f0cec1021d0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.295+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cec1a7400 con 0x7f0cec1021d0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.295+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cec1a77b0 con 0x7f0cec1021d0 2024-04-26T02:05:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.295+0000 7f0ce2a00640 1 -- 172.21.15.52:0/1135356267 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc010430 con 0x7f0cec1021d0 2024-04-26T02:05:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.299+0000 7f0cf3b65740 1 Processor -- start 2024-04-26T02:05:13.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.299+0000 7f0cf3b65740 1 -- start start 2024-04-26T02:05:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.299+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cea800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cea800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55858/0 (socket says 172.21.15.52:55858) 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cea800640 1 -- 172.21.15.52:0/3504837897 learned_addr learned my addr 172.21.15.52:0/3504837897 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563759725900 con 0x5637597c0140 2024-04-26T02:05:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637596cbef0 con 0x5637597caa00 2024-04-26T02:05:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.300+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637597218c0 con 0x5637597bf820 2024-04-26T02:05:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.301+0000 7f0ce2a00640 1 -- 172.21.15.52:0/1135356267 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0cdc005020 con 0x7f0cec1021d0 2024-04-26T02:05:13.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.301+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.302+0000 7f0ce2a00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 0x7f0cd00aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.303+0000 7f0ce2a00640 1 -- 172.21.15.52:0/1135356267 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f0cdc0d0b90 con 0x7f0cec1021d0 2024-04-26T02:05:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0cea800640 1 -- 172.21.15.52:0/3504837897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 msgr2=0x5637597ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0cea800640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0cea800640 1 -- 172.21.15.52:0/3504837897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 msgr2=0x5637597ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0cea800640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0cea800640 1 -- 172.21.15.52:0/3504837897 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563759438210 con 0x5637597bf820 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.304+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.305+0000 7f0cea800640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 0x7f0cd00aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.306+0000 7f0cea800640 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009d40 tx=0x7f0ce400d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d50536ed3eaad47 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.306+0000 7f0cea800640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 0x7f0cd00aeb10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009fd0 tx=0x7f0ce4012040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.306+0000 7f0cdb000640 1 -- 172.21.15.52:0/3504837897 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce401d630 con 0x5637597bf820 2024-04-26T02:05:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.306+0000 7f0cdb000640 1 -- 172.21.15.52:0/3504837897 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ce401dc80 con 0x5637597bf820 2024-04-26T02:05:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.306+0000 7f0cdb000640 1 -- 172.21.15.52:0/3504837897 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce401ccd0 con 0x5637597bf820 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.307+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 msgr2=0x5637597bfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.307+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009d40 tx=0x7f0ce400d6b0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 shutdown_connections 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3504837897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597bf820 0x5637597bfc00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 >> 172.21.15.52:0/3504837897 conn(0x5637597bbd30 msgr2=0x5637597bc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 shutdown_connections 2024-04-26T02:05:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.308+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3504837897 wait complete. 2024-04-26T02:05:13.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.309+0000 7f0cf3b65740 1 -- start start 2024-04-26T02:05:13.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.309+0000 7f0cf3b65740 1 Processor -- start 2024-04-26T02:05:13.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 0x5637597c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5637597cd630 con 0x5637597caa00 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5637596df310 con 0x5637597bf820 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5637597cd7b0 con 0x5637597c0140 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0cf3b65740 10 client.0 _ll_register_callbacks cb 0x5637596adcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49244/0 (socket says 172.21.15.52:49244) 2024-04-26T02:05:13.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0ce9e00640 1 -- 172.21.15.52:0/3920388539 learned_addr learned my addr 172.21.15.52:0/3920388539 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.310+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0cea800640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 0x5637597c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 -- 172.21.15.52:0/3920388539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 msgr2=0x5637597c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 -- 172.21.15.52:0/3920388539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 msgr2=0x5637597c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 0x5637597c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 -- 172.21.15.52:0/3920388539 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563759504620 con 0x5637597bf820 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0cea800640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 0x5637597c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc09c7d0 tx=0x7f0cdc0053d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9728b4af863c6fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ccf400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc09cad0 con 0x5637597bf820 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f0cdc016330 con 0x5637597bf820 2024-04-26T02:05:13.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.311+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ce401dc80 con 0x5637597bf820 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.312+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f0cdc016530 con 0x5637597bf820 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.312+0000 7f0cf3b65740 10 client.27365 fetch_fsmap learned FSMap version 70 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.312+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x5637597d98b0 con 0x5637597bf820 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.312+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ce401dc80 con 0x5637597bf820 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f0cdc0b8020 con 0x5637597bf820 2024-04-26T02:05:13.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 10 client.27365 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 10 client.27365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 20 client.27365 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637597dad70 0x5637597dd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 4 client.27365 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.313+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637597398b0 con 0x5637597dad70 2024-04-26T02:05:13.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.314+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637597dad70 0x5637597dd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.315+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637597dad70 0x5637597dd1a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97575b69f30a116c server_cookie=e0748fff876516ba in_seq=0 out_seq=0 2024-04-26T02:05:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.315+0000 7f0ccf400640 10 client.27365 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.319+0000 7f0ccf400640 1 -- 172.21.15.52:0/3920388539 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637597398b0 con 0x5637597dad70 2024-04-26T02:05:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.319+0000 7f0ccf400640 10 client.27365 handle_command_reply: tid=0 2024-04-26T02:05:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 client.27365 shutdown 2024-04-26T02:05:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 msgr2=0x5637597c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc09c7d0 tx=0x7f0cdc0053d0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 shutdown_connections 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5637597dad70 0x5637597dd1a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597caa00 0x5637597c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/3920388539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597c79a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 >> 172.21.15.52:0/3920388539 conn(0x5637597bbd30 msgr2=0x5637597c1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0ccf400640 0 client.27365 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 shutdown_connections 2024-04-26T02:05:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.320+0000 7f0cf3b65740 1 -- 172.21.15.52:0/3920388539 wait complete. 2024-04-26T02:05:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.321+0000 7f0cf3b65740 20 client.27365 trim_cache size 0 max 16384 2024-04-26T02:05:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.348+0000 7f0cf3b65740 1 Processor -- start 2024-04-26T02:05:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.348+0000 7f0cf3b65740 1 -- start start 2024-04-26T02:05:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 0x5637597e2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc8011050 con 0x5637597c0140 2024-04-26T02:05:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc800e920 con 0x5637597bf820 2024-04-26T02:05:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc800c790 con 0x5637597caa00 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:49256/0 (socket says 172.21.15.52:49256) 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0ce9e00640 1 -- 172.21.15.52:0/157940381 learned_addr learned my addr 172.21.15.52:0/157940381 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.349+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0cea800640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 0x5637597e2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0ce9e00640 1 -- 172.21.15.52:0/157940381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 msgr2=0x5637597e2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 0x5637597e2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0ce9e00640 1 -- 172.21.15.52:0/157940381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 msgr2=0x5637597c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0ce9e00640 1 -- 172.21.15.52:0/157940381 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637597c0ad0 con 0x5637597bf820 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.350+0000 7f0cea800640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 0x5637597e2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.351+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0097d0 tx=0x7f0cdc004b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f899a1f27c0d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.351+0000 7f0cda600640 1 -- 172.21.15.52:0/157940381 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0a0e80 con 0x5637597bf820 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.351+0000 7f0cda600640 1 -- 172.21.15.52:0/157940381 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cdc07f020 con 0x5637597bf820 2024-04-26T02:05:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.351+0000 7f0cda600640 1 -- 172.21.15.52:0/157940381 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0a54e0 con 0x5637597bf820 2024-04-26T02:05:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.352+0000 7f0cf3b65740 1 -- 172.21.15.52:0/157940381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 msgr2=0x5637597d6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.352+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0097d0 tx=0x7f0cdc004b60 comp rx=0 tx=0).stop 2024-04-26T02:05:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 -- 172.21.15.52:0/157940381 shutdown_connections 2024-04-26T02:05:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597caa00 0x5637597e2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597c0140 0x5637597c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/157940381 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597bf820 0x5637597d6540 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 -- 172.21.15.52:0/157940381 >> 172.21.15.52:0/157940381 conn(0x5637597bbd30 msgr2=0x5637597bae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 -- 172.21.15.52:0/157940381 shutdown_connections 2024-04-26T02:05:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 -- 172.21.15.52:0/157940381 wait complete. 2024-04-26T02:05:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.353+0000 7f0cf3b65740 1 -- start start 2024-04-26T02:05:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.354+0000 7f0cf3b65740 1 Processor -- start 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 0x5637597d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc800cde0 con 0x5637597bf820 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc800e250 con 0x5637597caa00 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc800d4b0 con 0x5637597c0140 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.356+0000 7f0cf3b65740 10 client.0 _ll_register_callbacks cb 0x7f0cdc09c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 0x5637597d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0ce9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 0x5637597d2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40700/0 (socket says 172.21.15.52:40700) 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0ce9e00640 1 -- 172.21.15.52:0/2065665947 learned_addr learned my addr 172.21.15.52:0/2065665947 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:13.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0cea800640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.357+0000 7f0ce3e00640 1 -- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 msgr2=0x5637597d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ce3e00640 1 -- 172.21.15.52:0/2065665947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 msgr2=0x5637597d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 0x5637597d2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ce3e00640 1 -- 172.21.15.52:0/2065665947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637597d0b20 con 0x5637597c0140 2024-04-26T02:05:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0cea800640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ccf400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:05:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.358+0000 7f0ce3e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597d0020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0cd402a910 tx=0x7f0cd4009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c51a756d03c0132d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4035040 con 0x5637597c0140 2024-04-26T02:05:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f0cd4005040 con 0x5637597c0140 2024-04-26T02:05:13.358 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cdc07f020 con 0x5637597c0140 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f0cd40044c0 con 0x5637597c0140 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0cf3b65740 10 client.27400 fetch_fsmap learned FSMap version 70 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.359+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x563759841170 con 0x5637597c0140 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.360+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cdc07f020 con 0x5637597c0140 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.360+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f0cd40046c0 con 0x5637597c0140 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 10 client.27400 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 10 client.27400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 20 client.27400 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563759841920 0x563759843d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 4 client.27400 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563759733320 con 0x563759841920 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.361+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563759841920 0x563759843d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.362+0000 7f0ce9e00640 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563759841920 0x563759843d40 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f615a742b6dd70ed server_cookie=10bdf20089be8878 in_seq=0 out_seq=0 2024-04-26T02:05:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.362+0000 7f0ccf400640 10 client.27400 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.597+0000 7f0ccf400640 1 -- 172.21.15.52:0/2065665947 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563759733320 con 0x563759841920 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0ccf400640 10 client.27400 handle_command_reply: tid=0 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 client.27400 shutdown 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 msgr2=0x5637597d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597d0020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0cd402a910 tx=0x7f0cd4009520 comp rx=0 tx=0).stop 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 shutdown_connections 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x563759841920 0x563759843d40 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5637597caa00 0x5637597d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5637597c0140 0x5637597d0020 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 --2- 172.21.15.52:0/2065665947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5637597bf820 0x5637597d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.598+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 >> 172.21.15.52:0/2065665947 conn(0x5637597bbd30 msgr2=0x5637597c1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.600 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.601+0000 7f0ccf400640 0 client.27400 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.602+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 shutdown_connections 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.602+0000 7f0cf3b65740 1 -- 172.21.15.52:0/2065665947 wait complete. 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.602+0000 7f0cf3b65740 20 client.27400 trim_cache size 0 max 16384 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:05:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:05:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 msgr2=0x7f0cd00aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 0x7f0cd00aeb10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009fd0 tx=0x7f0ce4012040 comp rx=0 tx=0).stop 2024-04-26T02:05:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 msgr2=0x7f0cec1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00e9f0 tx=0x7f0cdc00edf0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 shutdown_connections 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f0cd00ac650 0x7f0cd00aeb10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0cec19a720 0x7f0cec19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cec1072a0 0x7f0cec1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 --2- 172.21.15.52:0/1135356267 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0cec1021d0 0x7f0cec1a0660 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.603+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 >> 172.21.15.52:0/1135356267 conn(0x7f0cec0793e0 msgr2=0x7f0cec0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.604+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 shutdown_connections 2024-04-26T02:05:13.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:13.604+0000 7f0cf0c00640 1 -- 172.21.15.52:0/1135356267 wait complete. 2024-04-26T02:05:13.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 139666 19.77 MB/sec execute 270 sec latency 178.455 ms 2024-04-26T02:05:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:13 smithi191 ceph-mon[32745]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-26T02:05:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:13 smithi183 ceph-mon[32752]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-26T02:05:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:13 smithi052 ceph-mon[27851]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-26T02:05:14.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 140396 19.90 MB/sec execute 271 sec latency 76.310 ms 2024-04-26T02:05:15.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 140842 19.89 MB/sec execute 272 sec latency 99.939 ms 2024-04-26T02:05:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:15 smithi191 ceph-mon[32745]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 13 MiB/s wr, 715 op/s 2024-04-26T02:05:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:15 smithi183 ceph-mon[32752]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 13 MiB/s wr, 715 op/s 2024-04-26T02:05:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:15 smithi052 ceph-mon[27851]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 13 MiB/s wr, 715 op/s 2024-04-26T02:05:16.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 141401 19.93 MB/sec execute 273 sec latency 96.646 ms 2024-04-26T02:05:17.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 141900 19.97 MB/sec execute 274 sec latency 109.518 ms 2024-04-26T02:05:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:17 smithi191 ceph-mon[32745]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 971 op/s 2024-04-26T02:05:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:17 smithi183 ceph-mon[32752]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 971 op/s 2024-04-26T02:05:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:17 smithi052 ceph-mon[27851]: pgmap v643: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 16 MiB/s wr, 971 op/s 2024-04-26T02:05:18.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 142382 19.97 MB/sec execute 275 sec latency 97.925 ms 2024-04-26T02:05:19.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 142982 19.99 MB/sec execute 276 sec latency 103.251 ms 2024-04-26T02:05:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:19 smithi191 ceph-mon[32745]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 493 op/s 2024-04-26T02:05:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:19 smithi183 ceph-mon[32752]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 493 op/s 2024-04-26T02:05:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:19 smithi052 ceph-mon[27851]: pgmap v644: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 493 op/s 2024-04-26T02:05:20.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 143646 20.07 MB/sec execute 277 sec latency 116.871 ms 2024-04-26T02:05:21.761 INFO:tasks.workunit.client.0.smithi052.stdout: 10 144094 20.08 MB/sec execute 278 sec latency 236.340 ms 2024-04-26T02:05:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:21 smithi191 ceph-mon[32745]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 451 op/s 2024-04-26T02:05:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:21 smithi183 ceph-mon[32752]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 451 op/s 2024-04-26T02:05:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:21 smithi052 ceph-mon[27851]: pgmap v645: 97 pgs: 97 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 451 op/s 2024-04-26T02:05:22.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 144764 20.16 MB/sec execute 279 sec latency 74.524 ms 2024-04-26T02:05:23.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 145265 20.20 MB/sec execute 280 sec latency 238.706 ms 2024-04-26T02:05:23.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:23 smithi191 ceph-mon[32745]: pgmap v646: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.7 MiB/s wr, 645 op/s 2024-04-26T02:05:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:23 smithi183 ceph-mon[32752]: pgmap v646: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.7 MiB/s wr, 645 op/s 2024-04-26T02:05:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:23 smithi052 ceph-mon[27851]: pgmap v646: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 9.7 MiB/s wr, 645 op/s 2024-04-26T02:05:24.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 145536 20.17 MB/sec execute 281 sec latency 516.770 ms 2024-04-26T02:05:25.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 145916 20.15 MB/sec execute 282 sec latency 156.563 ms 2024-04-26T02:05:25.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:25 smithi191 ceph-mon[32745]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.4 MiB/s wr, 461 op/s 2024-04-26T02:05:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:25 smithi183 ceph-mon[32752]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.4 MiB/s wr, 461 op/s 2024-04-26T02:05:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:25 smithi052 ceph-mon[27851]: pgmap v647: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.4 MiB/s wr, 461 op/s 2024-04-26T02:05:26.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 146368 20.15 MB/sec execute 283 sec latency 347.412 ms 2024-04-26T02:05:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 -- 172.21.15.52:0/3119493231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941072a0 msgr2=0x7f1a9410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/3119493231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941072a0 0x7f1a9410ee70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009ca0 tx=0x7f1a8c02a8c0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 -- 172.21.15.52:0/3119493231 shutdown_connections 2024-04-26T02:05:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/3119493231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941072a0 0x7f1a9410ee70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/3119493231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a94102b80 0x7f1a94106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/3119493231 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a941021d0 0x7f1a941025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 -- 172.21.15.52:0/3119493231 >> 172.21.15.52:0/3119493231 conn(0x7f1a940793e0 msgr2=0x7f1a940797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.781+0000 7f1a9ac00640 1 -- 172.21.15.52:0/3119493231 shutdown_connections 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.782+0000 7f1a9ac00640 1 -- 172.21.15.52:0/3119493231 wait complete. 2024-04-26T02:05:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.783+0000 7f1a9ac00640 1 Processor -- start 2024-04-26T02:05:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.783+0000 7f1a9ac00640 1 -- start start 2024-04-26T02:05:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.784+0000 7f1a9ac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941021d0 0x7f1a9410edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.784+0000 7f1a9ac00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.784+0000 7f1a9ac00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 0x7f1a9410c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.784+0000 7f1a93400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.784+0000 7f1a93400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54900/0 (socket says 172.21.15.52:54900) 2024-04-26T02:05:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94075d30 con 0x7f1a9410c3b0 2024-04-26T02:05:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 -- 172.21.15.52:0/2593151208 learned_addr learned my addr 172.21.15.52:0/2593151208 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94075bb0 con 0x7f1a941072a0 2024-04-26T02:05:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94075eb0 con 0x7f1a941021d0 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93e00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941021d0 0x7f1a9410edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 0x7f1a9410c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 -- 172.21.15.52:0/2593151208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941021d0 msgr2=0x7f1a9410edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941021d0 0x7f1a9410edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 -- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 msgr2=0x7f1a9410c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 0x7f1a9410c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a93400640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941ab2b0 con 0x7f1a941072a0 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.786+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 0x7f1a9410c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.787+0000 7f1a93400640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e9a0 tx=0x7f1a8000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.788+0000 7f1a90c00640 1 -- 172.21.15.52:0/2593151208 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000cd80 con 0x7f1a941072a0 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.788+0000 7f1a90c00640 1 -- 172.21.15.52:0/2593151208 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a800035f0 con 0x7f1a941072a0 2024-04-26T02:05:26.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.788+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a941ab5a0 con 0x7f1a941072a0 2024-04-26T02:05:26.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.788+0000 7f1a90c00640 1 -- 172.21.15.52:0/2593151208 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80010690 con 0x7f1a941072a0 2024-04-26T02:05:26.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.788+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941abaf0 con 0x7f1a941072a0 2024-04-26T02:05:26.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.793+0000 7f1a9d982740 1 Processor -- start 2024-04-26T02:05:26.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.793+0000 7f1a9d982740 1 -- start start 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.793+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b069c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a9d982740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557b4afcf900 con 0x557b4b069820 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a9d982740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557b4af75ef0 con 0x557b4b06a140 2024-04-26T02:05:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a9d982740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557b4afcb8c0 con 0x557b4b074a00 2024-04-26T02:05:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a90c00640 1 -- 172.21.15.52:0/2593151208 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1a80004000 con 0x7f1a941072a0 2024-04-26T02:05:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a93400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a93400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44510/0 (socket says 172.21.15.52:44510) 2024-04-26T02:05:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a93400640 1 -- 172.21.15.52:0/2124554855 learned_addr learned my addr 172.21.15.52:0/2124554855 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a93e00640 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.794+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b069c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 -- 172.21.15.52:0/2124554855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 msgr2=0x557b4b0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 -- 172.21.15.52:0/2124554855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 msgr2=0x557b4b069c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b069c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 -- 172.21.15.52:0/2124554855 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4ace2210 con 0x557b4b074a00 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a93400640 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1a800ce0f0 tx=0x7f1a800d4b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2e7133875e2cec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.795+0000 7f1a89200640 1 -- 172.21.15.52:0/2124554855 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a800d9990 con 0x557b4b074a00 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.796+0000 7f1a89200640 1 -- 172.21.15.52:0/2124554855 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a800ef440 con 0x557b4b074a00 2024-04-26T02:05:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.796+0000 7f1a89200640 1 -- 172.21.15.52:0/2124554855 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a800ee640 con 0x557b4b074a00 2024-04-26T02:05:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.796+0000 7f1a9d982740 1 -- 172.21.15.52:0/2124554855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 msgr2=0x557b4b076df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1a800ce0f0 tx=0x7f1a800d4b60 comp rx=0 tx=0).stop 2024-04-26T02:05:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 -- 172.21.15.52:0/2124554855 shutdown_connections 2024-04-26T02:05:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b076df0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2124554855 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b069c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 -- 172.21.15.52:0/2124554855 >> 172.21.15.52:0/2124554855 conn(0x557b4b065d30 msgr2=0x557b4b066110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 -- 172.21.15.52:0/2124554855 shutdown_connections 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a9d982740 1 -- 172.21.15.52:0/2124554855 wait complete. 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a90c00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 0x7f1a780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.797+0000 7f1a90c00640 1 -- 172.21.15.52:0/2593151208 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f1a800d0ac0 con 0x7f1a941072a0 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.798+0000 7f1a9d982740 1 -- start start 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.798+0000 7f1a9d982740 1 Processor -- start 2024-04-26T02:05:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.798+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 0x7f1a780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557b4b077630 con 0x557b4b069820 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557b4af89310 con 0x557b4b06a140 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557b4b0777b0 con 0x557b4b074a00 2024-04-26T02:05:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a9d982740 10 client.0 _ll_register_callbacks cb 0x557b4af57cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a93400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.799+0000 7f1a93e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54914/0 (socket says 172.21.15.52:54914) 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 -- 172.21.15.52:0/476927045 learned_addr learned my addr 172.21.15.52:0/476927045 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 -- 172.21.15.52:0/476927045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 msgr2=0x557b4b072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a98c00640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 -- 172.21.15.52:0/476927045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 msgr2=0x557b4b0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93400640 1 -- 172.21.15.52:0/476927045 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b4adae620 con 0x557b4b06a140 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.800+0000 7f1a93e00640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a98c00640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a93400640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1a80098e60 tx=0x7f1a80003e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ec3f07928b62b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a7d600640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80098ca0 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 0x7f1a780a6300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c004af0 tx=0x7f1a8c004a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.801+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f1a800037a0 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.802+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a800ef440 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.802+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f1a8009c360 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.802+0000 7f1a9d982740 10 client.27383 fetch_fsmap learned FSMap version 70 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.802+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x557b4b0838b0 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.802+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a800ef440 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f1a80099070 con 0x557b4b06a140 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 10 client.27383 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 10 client.27383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 20 client.27383 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b084c00 0x557b4b086ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 4 client.27383 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b4afe38b0 con 0x557b4b084c00 2024-04-26T02:05:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.803+0000 7f1a93e00640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b084c00 0x557b4b086ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.804+0000 7f1a93e00640 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b084c00 0x557b4b086ff0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20afb4a5dc3b8d2a server_cookie=faf84a7d99e7d5fe in_seq=0 out_seq=0 2024-04-26T02:05:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.804+0000 7f1a7d600640 10 client.27383 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.805+0000 7f1a7d600640 1 -- 172.21.15.52:0/476927045 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b4afe38b0 con 0x557b4b084c00 2024-04-26T02:05:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.805+0000 7f1a7d600640 10 client.27383 handle_command_reply: tid=0 2024-04-26T02:05:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.805+0000 7f1a9d982740 1 client.27383 shutdown 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.805+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 msgr2=0x557b4b071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.805+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1a80098e60 tx=0x7f1a80003e70 comp rx=0 tx=0).stop 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 shutdown_connections 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b084c00 0x557b4b086ff0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b071ef0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 --2- 172.21.15.52:0/476927045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 >> 172.21.15.52:0/476927045 conn(0x557b4b065d30 msgr2=0x557b4b06bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a7d600640 0 client.27383 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 shutdown_connections 2024-04-26T02:05:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 1 -- 172.21.15.52:0/476927045 wait complete. 2024-04-26T02:05:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.806+0000 7f1a9d982740 20 client.27383 trim_cache size 0 max 16384 2024-04-26T02:05:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 Processor -- start 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 -- start start 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1a941ae330 con 0x557b4b069820 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94074000 con 0x557b4b06a140 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.830+0000 7f1a9d982740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410d3a0 con 0x557b4b074a00 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a98c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57596/0 (socket says 172.21.15.52:57596) 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a98c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44536/0 (socket says 172.21.15.52:44536) 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 -- 172.21.15.52:0/2438917219 learned_addr learned my addr 172.21.15.52:0/2438917219 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:26.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93400640 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 -- 172.21.15.52:0/2438917219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 msgr2=0x557b4b072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 -- 172.21.15.52:0/2438917219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 msgr2=0x557b4b06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b06c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a93e00640 1 -- 172.21.15.52:0/2438917219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4b0709a0 con 0x557b4b069820 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.831+0000 7f1a98c00640 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.832+0000 7f1a93e00640 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009640 tx=0x7f1a84005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622a1ab06bd573cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.832+0000 7f1a7fe00640 1 -- 172.21.15.52:0/2438917219 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004220 con 0x557b4b069820 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.832+0000 7f1a7fe00640 1 -- 172.21.15.52:0/2438917219 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a84024030 con 0x557b4b069820 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.832+0000 7f1a7fe00640 1 -- 172.21.15.52:0/2438917219 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004ce0 con 0x557b4b069820 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 -- 172.21.15.52:0/2438917219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 msgr2=0x557b4b0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009640 tx=0x7f1a84005f90 comp rx=0 tx=0).stop 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 -- 172.21.15.52:0/2438917219 shutdown_connections 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b074a00 0x557b4b072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b06a140 0x557b4b06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 --2- 172.21.15.52:0/2438917219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b069820 0x557b4b0702c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 -- 172.21.15.52:0/2438917219 >> 172.21.15.52:0/2438917219 conn(0x557b4b065d30 msgr2=0x557b4b080290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.834+0000 7f1a9d982740 1 -- 172.21.15.52:0/2438917219 shutdown_connections 2024-04-26T02:05:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.835+0000 7f1a9d982740 1 -- 172.21.15.52:0/2438917219 wait complete. 2024-04-26T02:05:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.835+0000 7f1a9d982740 1 -- start start 2024-04-26T02:05:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.835+0000 7f1a9d982740 1 Processor -- start 2024-04-26T02:05:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.836+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b069820 0x557b4b089490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.836+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.836+0000 7f1a9d982740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b074a00 0x557b4b0e2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44542/0 (socket says 172.21.15.52:44542) 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93400640 1 -- 172.21.15.52:0/3774956004 learned_addr learned my addr 172.21.15.52:0/3774956004 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a98c00640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b074a00 0x557b4b0e2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940724e0 con 0x557b4b074a00 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93e00640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b069820 0x557b4b089490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94073930 con 0x557b4b069820 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94072b90 con 0x557b4b06a140 2024-04-26T02:05:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a9d982740 10 client.0 _ll_register_callbacks cb 0x7f1a8000ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93400640 1 -- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b069820 msgr2=0x557b4b089490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.837+0000 7f1a93400640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b069820 0x557b4b089490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a93400640 1 -- 172.21.15.52:0/3774956004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b074a00 msgr2=0x557b4b0e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a93400640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b074a00 0x557b4b0e2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a93400640 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b4b0e26e0 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a93400640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1a8009c990 tx=0x7f1a8009af10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d0ffe21f5ad3811 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a7d600640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8009c540 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.838+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f1a800a53e0 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a84024030 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f1a8009c7d0 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 10 client.27415 fetch_fsmap learned FSMap version 70 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x557b4b0eac90 con 0x557b4b06a140 2024-04-26T02:05:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a84024030 con 0x557b4b06a140 2024-04-26T02:05:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 70) ==== 8346+0+0 (secure 0 0 0) 0x7f1a8009cc60 con 0x557b4b06a140 2024-04-26T02:05:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 10 client.27415 fetch_fsmap finished waiting for FSMap version 70 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 10 client.27415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 20 client.27415 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b0eb1a0 0x557b4b0ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 4 client.27415 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.839+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b4afdd320 con 0x557b4b0eb1a0 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.840+0000 7f1a93e00640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b0eb1a0 0x557b4b0ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.840+0000 7f1a93e00640 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b0eb1a0 0x557b4b0ed5c0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cee272adae40b69 server_cookie=e02a5d3679f54bae in_seq=0 out_seq=0 2024-04-26T02:05:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:26.840+0000 7f1a7d600640 10 client.27415 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.080+0000 7f1a7d600640 1 -- 172.21.15.52:0/3774956004 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b4afdd320 con 0x557b4b0eb1a0 2024-04-26T02:05:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.080+0000 7f1a7d600640 10 client.27415 handle_command_reply: tid=0 2024-04-26T02:05:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.080+0000 7f1a9d982740 1 client.27415 shutdown 2024-04-26T02:05:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 msgr2=0x557b4b0899d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1a8009c990 tx=0x7f1a8009af10 comp rx=0 tx=0).stop 2024-04-26T02:05:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 shutdown_connections 2024-04-26T02:05:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557b4b0eb1a0 0x557b4b0ed5c0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557b4b074a00 0x557b4b0e2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b4b06a140 0x557b4b0899d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 --2- 172.21.15.52:0/3774956004 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557b4b069820 0x557b4b089490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 >> 172.21.15.52:0/3774956004 conn(0x557b4b065d30 msgr2=0x557b4b064600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.081+0000 7f1a7d600640 0 client.27415 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.082+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 shutdown_connections 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.082+0000 7f1a9d982740 1 -- 172.21.15.52:0/3774956004 wait complete. 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.083+0000 7f1a9d982740 20 client.27415 trim_cache size 0 max 16384 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:05:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 msgr2=0x7f1a780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 0x7f1a780a6300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c004af0 tx=0x7f1a8c004a40 comp rx=0 tx=0).stop 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 msgr2=0x7f1a9410be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e9a0 tx=0x7f1a8000ee60 comp rx=0 tx=0).stop 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 shutdown_connections 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f1a780a3e40 0x7f1a780a6300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a9410c3b0 0x7f1a9410c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a941072a0 0x7f1a9410be70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.084+0000 7f1a9ac00640 1 --2- 172.21.15.52:0/2593151208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a941021d0 0x7f1a9410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:27.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.085+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 >> 172.21.15.52:0/2593151208 conn(0x7f1a940793e0 msgr2=0x7f1a940ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:27.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.085+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 shutdown_connections 2024-04-26T02:05:27.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:27.086+0000 7f1a9ac00640 1 -- 172.21.15.52:0/2593151208 wait complete. 2024-04-26T02:05:27.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 146803 20.16 MB/sec execute 284 sec latency 216.691 ms 2024-04-26T02:05:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:27 smithi191 ceph-mon[32745]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 690 op/s 2024-04-26T02:05:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:27 smithi183 ceph-mon[32752]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 690 op/s 2024-04-26T02:05:27.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:27 smithi052 ceph-mon[27851]: pgmap v648: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 690 op/s 2024-04-26T02:05:28.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 147474 20.22 MB/sec execute 285 sec latency 129.420 ms 2024-04-26T02:05:29.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 147762 20.21 MB/sec execute 286 sec latency 706.768 ms 2024-04-26T02:05:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:29 smithi191 ceph-mon[32745]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 459 op/s 2024-04-26T02:05:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:29 smithi183 ceph-mon[32752]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 459 op/s 2024-04-26T02:05:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:29 smithi052 ceph-mon[27851]: pgmap v649: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.3 MiB/s wr, 459 op/s 2024-04-26T02:05:30.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 148134 20.25 MB/sec execute 287 sec latency 206.325 ms 2024-04-26T02:05:31.762 INFO:tasks.workunit.client.0.smithi052.stdout: 10 148653 20.27 MB/sec execute 288 sec latency 80.521 ms 2024-04-26T02:05:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:31 smithi191 ceph-mon[32745]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-04-26T02:05:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:31 smithi183 ceph-mon[32752]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-04-26T02:05:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:31 smithi052 ceph-mon[27851]: pgmap v650: 97 pgs: 97 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.2 MiB/s wr, 446 op/s 2024-04-26T02:05:32.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 149078 20.26 MB/sec execute 289 sec latency 103.933 ms 2024-04-26T02:05:33.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 149514 20.26 MB/sec execute 290 sec latency 372.835 ms 2024-04-26T02:05:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:33 smithi191 ceph-mon[32745]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 877 op/s 2024-04-26T02:05:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:33 smithi183 ceph-mon[32752]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 877 op/s 2024-04-26T02:05:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:33 smithi052 ceph-mon[27851]: pgmap v651: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 877 op/s 2024-04-26T02:05:34.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 150106 20.31 MB/sec execute 291 sec latency 133.496 ms 2024-04-26T02:05:35.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 150804 20.36 MB/sec execute 292 sec latency 72.449 ms 2024-04-26T02:05:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:35 smithi191 ceph-mon[32745]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-26T02:05:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:35 smithi183 ceph-mon[32752]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-26T02:05:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:35 smithi052 ceph-mon[27851]: pgmap v652: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-26T02:05:36.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 151359 20.40 MB/sec execute 293 sec latency 106.472 ms 2024-04-26T02:05:37.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 151992 20.47 MB/sec execute 294 sec latency 75.680 ms 2024-04-26T02:05:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:37 smithi191 ceph-mon[32745]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 838 op/s 2024-04-26T02:05:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:37 smithi183 ceph-mon[32752]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 838 op/s 2024-04-26T02:05:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:37 smithi052 ceph-mon[27851]: pgmap v653: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 17 MiB/s wr, 838 op/s 2024-04-26T02:05:38.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 152475 20.47 MB/sec execute 295 sec latency 148.315 ms 2024-04-26T02:05:39.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 153054 20.51 MB/sec execute 296 sec latency 81.877 ms 2024-04-26T02:05:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:39 smithi191 ceph-mon[32745]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-04-26T02:05:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:39 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:05:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:39 smithi183 ceph-mon[32752]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-04-26T02:05:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:39 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:05:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:39 smithi052 ceph-mon[27851]: pgmap v654: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 630 op/s 2024-04-26T02:05:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:39 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.276+0000 7f2a77200640 1 -- 172.21.15.52:0/2445103879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78076da0 msgr2=0x7f2a781126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.276+0000 7f2a77200640 1 --2- 172.21.15.52:0/2445103879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78076da0 0x7f2a781126e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009930 tx=0x7f2a6002a440 comp rx=0 tx=0).stop 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.277+0000 7f2a77200640 1 -- 172.21.15.52:0/2445103879 shutdown_connections 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.277+0000 7f2a77200640 1 --2- 172.21.15.52:0/2445103879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78112c20 0x7f2a78115010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.277+0000 7f2a77200640 1 --2- 172.21.15.52:0/2445103879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78076da0 0x7f2a781126e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.277+0000 7f2a77200640 1 --2- 172.21.15.52:0/2445103879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a78076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.277+0000 7f2a77200640 1 -- 172.21.15.52:0/2445103879 >> 172.21.15.52:0/2445103879 conn(0x7f2a780ffa30 msgr2=0x7f2a78101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.278+0000 7f2a77200640 1 -- 172.21.15.52:0/2445103879 shutdown_connections 2024-04-26T02:05:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.278+0000 7f2a77200640 1 -- 172.21.15.52:0/2445103879 wait complete. 2024-04-26T02:05:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 Processor -- start 2024-04-26T02:05:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 -- start start 2024-04-26T02:05:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a781a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78076da0 0x7f2a781a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2a78118690 con 0x7f2a78076da0 2024-04-26T02:05:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2a78118510 con 0x7f2a78076480 2024-04-26T02:05:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.280+0000 7f2a77200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2a78118810 con 0x7f2a78112c20 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.281+0000 7f2a76800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.281+0000 7f2a75400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78076da0 0x7f2a781a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.281+0000 7f2a76800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39908/0 (socket says 172.21.15.52:39908) 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.281+0000 7f2a76800640 1 -- 172.21.15.52:0/6191817 learned_addr learned my addr 172.21.15.52:0/6191817 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.281+0000 7f2a75e00640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a781a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a76800640 1 -- 172.21.15.52:0/6191817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 msgr2=0x7f2a781a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a76800640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a781a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a76800640 1 -- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78076da0 msgr2=0x7f2a781a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a76800640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78076da0 0x7f2a781a4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a76800640 1 -- 172.21.15.52:0/6191817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7819f220 con 0x7f2a78112c20 2024-04-26T02:05:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.282+0000 7f2a75e00640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a781a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.283+0000 7f2a76800640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009cc0 tx=0x7f2a7000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.285+0000 7f2a6ea00640 1 -- 172.21.15.52:0/6191817 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70003b70 con 0x7f2a78112c20 2024-04-26T02:05:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.285+0000 7f2a6ea00640 1 -- 172.21.15.52:0/6191817 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a70003d10 con 0x7f2a78112c20 2024-04-26T02:05:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.285+0000 7f2a6ea00640 1 -- 172.21.15.52:0/6191817 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70010820 con 0x7f2a78112c20 2024-04-26T02:05:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.285+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7819f510 con 0x7f2a78112c20 2024-04-26T02:05:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.285+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a78103e90 con 0x7f2a78112c20 2024-04-26T02:05:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.293+0000 7f2a6ea00640 1 -- 172.21.15.52:0/6191817 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2a70004020 con 0x7f2a78112c20 2024-04-26T02:05:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.295+0000 7f2a7fdff740 1 Processor -- start 2024-04-26T02:05:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.295+0000 7f2a7fdff740 1 -- start start 2024-04-26T02:05:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a75400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a6ea00640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 0x7f2a5c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a75400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46498/0 (socket says 172.21.15.52:46498) 2024-04-26T02:05:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fff1b50900 con 0x55fff1bea9b0 2024-04-26T02:05:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fff1af6ef0 con 0x55fff1bf5b90 2024-04-26T02:05:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fff1b4c8c0 con 0x55fff1beb2d0 2024-04-26T02:05:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.296+0000 7f2a75400640 1 -- 172.21.15.52:0/2116579941 learned_addr learned my addr 172.21.15.52:0/2116579941 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.297+0000 7f2a6ea00640 1 -- 172.21.15.52:0/6191817 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f2a700d0ef0 con 0x7f2a78112c20 2024-04-26T02:05:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.297+0000 7f2a76800640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.297+0000 7f2a75e00640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 0x7f2a5c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.298+0000 7f2a75400640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.298+0000 7f2a76800640 1 -- 172.21.15.52:0/2116579941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 msgr2=0x55fff1bf5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a76800640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a76800640 1 -- 172.21.15.52:0/2116579941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 msgr2=0x55fff1bead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a76800640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a76800640 1 -- 172.21.15.52:0/2116579941 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fff1863210 con 0x55fff1bf5b90 2024-04-26T02:05:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a75400640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.299+0000 7f2a76800640 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf7f80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2a700ce5e0 tx=0x7f2a7009ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67b120499cbaa45a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.300+0000 7f2a67400640 1 -- 172.21.15.52:0/2116579941 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70009820 con 0x55fff1bf5b90 2024-04-26T02:05:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.300+0000 7f2a67400640 1 -- 172.21.15.52:0/2116579941 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a7009d530 con 0x55fff1bf5b90 2024-04-26T02:05:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.300+0000 7f2a67400640 1 -- 172.21.15.52:0/2116579941 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a700b6970 con 0x55fff1bf5b90 2024-04-26T02:05:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2116579941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 msgr2=0x55fff1bf7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf7f80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2a700ce5e0 tx=0x7f2a7009ca30 comp rx=0 tx=0).stop 2024-04-26T02:05:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a75e00640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 0x7f2a5c0a6360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005fd0 tx=0x7f2a68005ea0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2116579941 shutdown_connections 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf7f80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2116579941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2116579941 >> 172.21.15.52:0/2116579941 conn(0x55fff1be6ec0 msgr2=0x55fff1be72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.301+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2116579941 shutdown_connections 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.302+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2116579941 wait complete. 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.302+0000 7f2a7fdff740 1 -- start start 2024-04-26T02:05:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.303+0000 7f2a7fdff740 1 Processor -- start 2024-04-26T02:05:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bf2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fff1bf8760 con 0x55fff1bea9b0 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55fff1b0a310 con 0x55fff1bf5b90 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fff1bf88e0 con 0x55fff1beb2d0 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a7fdff740 10 client.0 _ll_register_callbacks cb 0x55fff1ad8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a76800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.304+0000 7f2a76800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46644/0 (socket says 172.21.15.52:46644) 2024-04-26T02:05:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.305+0000 7f2a76800640 1 -- 172.21.15.52:0/2156145425 learned_addr learned my addr 172.21.15.52:0/2156145425 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.305+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bf2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.306+0000 7f2a75400640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.307+0000 7f2a76800640 1 -- 172.21.15.52:0/2156145425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 msgr2=0x55fff1bf3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.307+0000 7f2a76800640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.307+0000 7f2a76800640 1 -- 172.21.15.52:0/2156145425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 msgr2=0x55fff1bf2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.308+0000 7f2a76800640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bf2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.308+0000 7f2a76800640 1 -- 172.21.15.52:0/2156145425 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fff192f620 con 0x55fff1bf5b90 2024-04-26T02:05:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.308+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bf2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.309+0000 7f2a76800640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000cba0 tx=0x7f2a700996b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=805e3ff5d5fb6cae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.309+0000 7f2a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.309+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a700cea70 con 0x55fff1bf5b90 2024-04-26T02:05:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.310+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f2a70086aa0 con 0x55fff1bf5b90 2024-04-26T02:05:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.310+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a7009d530 con 0x55fff1bf5b90 2024-04-26T02:05:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.310+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f2a70009820 con 0x55fff1bf5b90 2024-04-26T02:05:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.310+0000 7f2a7fdff740 10 client.27398 fetch_fsmap learned FSMap version 71 2024-04-26T02:05:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.310+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55fff1c049f0 con 0x55fff1bf5b90 2024-04-26T02:05:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.311+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a7009d530 con 0x55fff1bf5b90 2024-04-26T02:05:40.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.312+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 71) ==== 8342+0+0 (secure 0 0 0) 0x7f2a70004410 con 0x55fff1bf5b90 2024-04-26T02:05:40.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 10 client.27398 fetch_fsmap finished waiting for FSMap version 71 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 10 client.27398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 20 client.27398 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c08130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 4 client.27398 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fff1b648b0 con 0x55fff1c05d40 2024-04-26T02:05:40.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.315+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c08130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.316+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c08130 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da25d9e3ed741829 server_cookie=9976f29eba8be7ec in_seq=0 out_seq=0 2024-04-26T02:05:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.316+0000 7f2a5be00640 10 client.27398 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.318+0000 7f2a5be00640 1 -- 172.21.15.52:0/2156145425 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fff1b648b0 con 0x55fff1c05d40 2024-04-26T02:05:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.318+0000 7f2a5be00640 10 client.27398 handle_command_reply: tid=0 2024-04-26T02:05:40.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.318+0000 7f2a7fdff740 1 client.27398 shutdown 2024-04-26T02:05:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.319+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 msgr2=0x55fff1bf35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.319+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000cba0 tx=0x7f2a700996b0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 shutdown_connections 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c08130 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1bf35d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a5be00640 0 client.27398 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1bf3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2156145425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1bf2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 >> 172.21.15.52:0/2156145425 conn(0x55fff1be6ec0 msgr2=0x55fff1becc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 shutdown_connections 2024-04-26T02:05:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2156145425 wait complete. 2024-04-26T02:05:40.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.320+0000 7f2a7fdff740 20 client.27398 trim_cache size 0 max 16384 2024-04-26T02:05:40.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.353+0000 7f2a7fdff740 1 Processor -- start 2024-04-26T02:05:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.353+0000 7f2a7fdff740 1 -- start start 2024-04-26T02:05:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.353+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.353+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 0x55fff1bec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1becde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2a781bf2b0 con 0x55fff1beb2d0 2024-04-26T02:05:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2a781bcb80 con 0x55fff1bf5b90 2024-04-26T02:05:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7810c230 con 0x55fff1bea9b0 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:39940/0 (socket says 172.21.15.52:39940) 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 -- 172.21.15.52:0/2876627535 learned_addr learned my addr 172.21.15.52:0/2876627535 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75400640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 0x55fff1bec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 -- 172.21.15.52:0/2876627535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 msgr2=0x55fff1becde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a76800640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1becde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1becde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 -- 172.21.15.52:0/2876627535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 msgr2=0x55fff1bec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 0x55fff1bec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75e00640 1 -- 172.21.15.52:0/2876627535 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fff1be2820 con 0x55fff1bea9b0 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.354+0000 7f2a75400640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 0x55fff1bec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.355+0000 7f2a76800640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1becde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.355+0000 7f2a75e00640 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800a910 tx=0x7f2a68014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b05c9e515fcc3a27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.355+0000 7f2a66a00640 1 -- 172.21.15.52:0/2876627535 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68026dc0 con 0x55fff1bea9b0 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.355+0000 7f2a66a00640 1 -- 172.21.15.52:0/2876627535 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a68005020 con 0x55fff1bea9b0 2024-04-26T02:05:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.355+0000 7f2a66a00640 1 -- 172.21.15.52:0/2876627535 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68026530 con 0x55fff1bea9b0 2024-04-26T02:05:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2876627535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 msgr2=0x55fff1bec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800a910 tx=0x7f2a68014000 comp rx=0 tx=0).stop 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2876627535 shutdown_connections 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1becde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1beb2d0 0x55fff1bec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/2876627535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1bea9b0 0x55fff1bec360 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2876627535 >> 172.21.15.52:0/2876627535 conn(0x55fff1be6ec0 msgr2=0x55fff1c013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.356+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2876627535 shutdown_connections 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.357+0000 7f2a7fdff740 1 -- 172.21.15.52:0/2876627535 wait complete. 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.357+0000 7f2a7fdff740 1 -- start start 2024-04-26T02:05:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.357+0000 7f2a7fdff740 1 Processor -- start 2024-04-26T02:05:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1c656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1c0a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7810c880 con 0x55fff1bea9b0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7810dcf0 con 0x55fff1bf5b90 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7810cf50 con 0x55fff1beb2d0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a7fdff740 10 client.0 _ll_register_callbacks cb 0x7f2a7009cc00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a76800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a76800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46668/0 (socket says 172.21.15.52:46668) 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.358+0000 7f2a76800640 1 -- 172.21.15.52:0/851172192 learned_addr learned my addr 172.21.15.52:0/851172192 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a75400640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1c0a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a75e00640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1c656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 -- 172.21.15.52:0/851172192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 msgr2=0x55fff1c0a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1c0a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 -- 172.21.15.52:0/851172192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 msgr2=0x55fff1c656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1c656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 -- 172.21.15.52:0/851172192 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fff1c0b320 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a75400640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1c0a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a75e00640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1c656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a76800640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2a70005ee0 tx=0x7f2a7000cd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a53fa6a75a564c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a5be00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70009820 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.359+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f2a700b68e0 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a68005020 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f2a700aa070 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 10 client.27401 fetch_fsmap learned FSMap version 71 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55fff1c7cf20 con 0x55fff1bf5b90 2024-04-26T02:05:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a68005020 con 0x55fff1bf5b90 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 71) ==== 8342+0+0 (secure 0 0 0) 0x7f2a70004310 con 0x55fff1bf5b90 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 10 client.27401 fetch_fsmap finished waiting for FSMap version 71 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 10 client.27401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 20 client.27401 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c7f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 4 client.27401 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.360+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fff1b5e320 con 0x55fff1c05d40 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.361+0000 7f2a75e00640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c7f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.362+0000 7f2a75e00640 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c7f220 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d44b1c8dcd707d7 server_cookie=a2b6ac73830fa7fb in_seq=0 out_seq=0 2024-04-26T02:05:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.362+0000 7f2a5be00640 10 client.27401 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.604+0000 7f2a5be00640 1 -- 172.21.15.52:0/851172192 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fff1b5e320 con 0x55fff1c05d40 2024-04-26T02:05:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.604+0000 7f2a5be00640 10 client.27401 handle_command_reply: tid=0 2024-04-26T02:05:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 client.27401 shutdown 2024-04-26T02:05:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 msgr2=0x55fff1c0ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2a70005ee0 tx=0x7f2a7000cd40 comp rx=0 tx=0).stop 2024-04-26T02:05:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 shutdown_connections 2024-04-26T02:05:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55fff1c05d40 0x55fff1c7f220 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55fff1bf5b90 0x55fff1c0ac30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fff1beb2d0 0x55fff1c0a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 --2- 172.21.15.52:0/851172192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fff1bea9b0 0x55fff1c656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a5be00640 0 client.27401 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.605+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 >> 172.21.15.52:0/851172192 conn(0x55fff1be6ec0 msgr2=0x55fff1bf2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.606+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 shutdown_connections 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.606+0000 7f2a7fdff740 1 -- 172.21.15.52:0/851172192 wait complete. 2024-04-26T02:05:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.607+0000 7f2a7fdff740 20 client.27401 trim_cache size 0 max 16384 2024-04-26T02:05:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:05:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:05:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:05:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:05:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 msgr2=0x7f2a5c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 0x7f2a5c0a6360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005fd0 tx=0x7f2a68005ea0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 msgr2=0x7f2a7819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009cc0 tx=0x7f2a7000c6a0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 shutdown_connections 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2a5c0a3ea0 0x7f2a5c0a6360 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005fd0 tx=0x7f2a68005ea0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a78112c20 0x7f2a7819ead0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2a78076da0 0x7f2a781a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 --2- 172.21.15.52:0/6191817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a78076480 0x7f2a781a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 >> 172.21.15.52:0/6191817 conn(0x7f2a780ffa30 msgr2=0x7f2a78101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.608+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 shutdown_connections 2024-04-26T02:05:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:40.609+0000 7f2a77200640 1 -- 172.21.15.52:0/6191817 wait complete. 2024-04-26T02:05:40.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 153462 20.49 MB/sec execute 297 sec latency 185.107 ms 2024-04-26T02:05:41.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 154061 20.52 MB/sec execute 298 sec latency 139.991 ms 2024-04-26T02:05:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:41 smithi191 ceph-mon[32745]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 605 op/s 2024-04-26T02:05:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:41 smithi183 ceph-mon[32752]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 605 op/s 2024-04-26T02:05:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:41 smithi052 ceph-mon[27851]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 605 op/s 2024-04-26T02:05:42.763 INFO:tasks.workunit.client.0.smithi052.stdout: 10 154719 20.61 MB/sec execute 299 sec latency 85.155 ms 2024-04-26T02:05:43.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 155216 20.61 MB/sec execute 300 sec latency 94.442 ms 2024-04-26T02:05:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:43 smithi191 ceph-mon[32745]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 874 op/s 2024-04-26T02:05:43.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:43 smithi183 ceph-mon[32752]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 874 op/s 2024-04-26T02:05:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:43 smithi052 ceph-mon[27851]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 14 MiB/s wr, 874 op/s 2024-04-26T02:05:44.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 155873 20.66 MB/sec execute 301 sec latency 69.763 ms 2024-04-26T02:05:45.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 156458 20.73 MB/sec execute 302 sec latency 97.151 ms 2024-04-26T02:05:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:45 smithi191 ceph-mon[32745]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-04-26T02:05:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:45 smithi183 ceph-mon[32752]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-04-26T02:05:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:45 smithi052 ceph-mon[27851]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-04-26T02:05:46.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 156860 20.71 MB/sec execute 303 sec latency 81.777 ms 2024-04-26T02:05:47.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 157389 20.71 MB/sec execute 304 sec latency 108.126 ms 2024-04-26T02:05:47.776 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:47 smithi052 ceph-mon[27851]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.5 MiB/s wr, 609 op/s 2024-04-26T02:05:47.776 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:47 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:47 smithi191 ceph-mon[32745]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.5 MiB/s wr, 609 op/s 2024-04-26T02:05:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:47 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:47 smithi183 ceph-mon[32752]: pgmap v658: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.5 MiB/s wr, 609 op/s 2024-04-26T02:05:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:47 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:05:48.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 158037 20.78 MB/sec execute 305 sec latency 139.676 ms 2024-04-26T02:05:49.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:48 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:48 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:48 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:48 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:48 smithi052 ceph-mon[27851]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 479 op/s 2024-04-26T02:05:49.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:48 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:48 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:48 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:48 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:48 smithi191 ceph-mon[32745]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 479 op/s 2024-04-26T02:05:49.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:48 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:48 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:48 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:48 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:49.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:48 smithi183 ceph-mon[32752]: pgmap v659: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.5 MiB/s wr, 479 op/s 2024-04-26T02:05:49.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 158425 20.80 MB/sec execute 306 sec latency 175.318 ms 2024-04-26T02:05:50.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 158592 20.77 MB/sec execute 307 sec latency 731.116 ms 2024-04-26T02:05:51.653 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:51 smithi191 ceph-mon[32745]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 459 op/s 2024-04-26T02:05:51.677 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:51 smithi183 ceph-mon[32752]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 459 op/s 2024-04-26T02:05:51.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:51 smithi052 ceph-mon[27851]: pgmap v660: 97 pgs: 97 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 459 op/s 2024-04-26T02:05:51.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 158849 20.72 MB/sec execute 308 sec latency 413.024 ms 2024-04-26T02:05:52.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 159054 20.67 MB/sec execute 309 sec latency 383.630 ms 2024-04-26T02:05:53.156 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.178 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 693 op/s 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:52 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.180 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.180 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.180 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 693 op/s 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.181 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:52 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: pgmap v661: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 14 MiB/s wr, 693 op/s 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:52 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:53.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 159610 20.72 MB/sec execute 310 sec latency 475.025 ms 2024-04-26T02:05:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 -- 172.21.15.52:0/2752949111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c102b90 msgr2=0x7f9d4c106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 --2- 172.21.15.52:0/2752949111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c102b90 0x7f9d4c106bd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d3802a400 comp rx=0 tx=0).stop 2024-04-26T02:05:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 -- 172.21.15.52:0/2752949111 shutdown_connections 2024-04-26T02:05:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 --2- 172.21.15.52:0/2752949111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 0x7f9d4c10ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 --2- 172.21.15.52:0/2752949111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c102b90 0x7f9d4c106bd0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 --2- 172.21.15.52:0/2752949111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c1025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 -- 172.21.15.52:0/2752949111 >> 172.21.15.52:0/2752949111 conn(0x7f9d4c0fda00 msgr2=0x7f9d4c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 -- 172.21.15.52:0/2752949111 shutdown_connections 2024-04-26T02:05:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.790+0000 7f9d51a00640 1 -- 172.21.15.52:0/2752949111 wait complete. 2024-04-26T02:05:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.791+0000 7f9d51a00640 1 Processor -- start 2024-04-26T02:05:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.792+0000 7f9d51a00640 1 -- start start 2024-04-26T02:05:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 0x7f9d4c10be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 0x7f9d4c10c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1144a0 con 0x7f9d4c1072b0 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c114320 con 0x7f9d4c1021e0 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d51a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c114620 con 0x7f9d4c10c3c0 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d4aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d4aa00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43786/0 (socket says 172.21.15.52:43786) 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 0x7f9d4c10c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d4aa00640 1 -- 172.21.15.52:0/1486116136 learned_addr learned my addr 172.21.15.52:0/1486116136 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.793+0000 7f9d4a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 0x7f9d4c10be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.794+0000 7f9d4aa00640 1 -- 172.21.15.52:0/1486116136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 msgr2=0x7f9d4c10c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.794+0000 7f9d4aa00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 0x7f9d4c10c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.794+0000 7f9d4aa00640 1 -- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 msgr2=0x7f9d4c10be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.794+0000 7f9d4aa00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 0x7f9d4c10be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.795+0000 7f9d4aa00640 1 -- 172.21.15.52:0/1486116136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c072c00 con 0x7f9d4c1021e0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.795+0000 7f9d4b400640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 0x7f9d4c10c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d4aa00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c02f960 tx=0x7f9d3c02fe20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d43400640 1 -- 172.21.15.52:0/1486116136 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c039070 con 0x7f9d4c1021e0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d43400640 1 -- 172.21.15.52:0/1486116136 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c009c30 con 0x7f9d4c1021e0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d43400640 1 -- 172.21.15.52:0/1486116136 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c031630 con 0x7f9d4c1021e0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c072ef0 con 0x7f9d4c1021e0 2024-04-26T02:05:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.796+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c0733c0 con 0x7f9d4c1021e0 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 Processor -- start 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 -- start start 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb669dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562eb6603900 con 0x562eb669d820 2024-04-26T02:05:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562eb65a9ef0 con 0x562eb66a8a00 2024-04-26T02:05:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.800+0000 7f9d54876740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562eb65ff8c0 con 0x562eb669e140 2024-04-26T02:05:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.801+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb669dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.801+0000 7f9d4a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.801+0000 7f9d4a000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66aadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43794/0 (socket says 172.21.15.52:43794) 2024-04-26T02:05:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.801+0000 7f9d4a000640 1 -- 172.21.15.52:0/2987031063 learned_addr learned my addr 172.21.15.52:0/2987031063 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 -- 172.21.15.52:0/2987031063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 msgr2=0x562eb66a84c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T02:05:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66a84c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 -- 172.21.15.52:0/2987031063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 msgr2=0x562eb66aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 -- 172.21.15.52:0/2987031063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eb6316210 con 0x562eb669d820 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.802+0000 7f9d4b400640 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb669dc00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400a740 tx=0x7f9d44009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ddbe7d4894bb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.803+0000 7f9d37e00640 1 -- 172.21.15.52:0/2987031063 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44018020 con 0x562eb669d820 2024-04-26T02:05:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.803+0000 7f9d43400640 1 -- 172.21.15.52:0/1486116136 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9d3c032470 con 0x7f9d4c1021e0 2024-04-26T02:05:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.803+0000 7f9d37e00640 1 -- 172.21.15.52:0/2987031063 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d4400ed50 con 0x562eb669d820 2024-04-26T02:05:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.803+0000 7f9d37e00640 1 -- 172.21.15.52:0/2987031063 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44013360 con 0x562eb669d820 2024-04-26T02:05:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.804+0000 7f9d43400640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 0x7f9d300a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.805+0000 7f9d43400640 1 -- 172.21.15.52:0/1486116136 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f9d3c0f2410 con 0x7f9d4c1021e0 2024-04-26T02:05:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.805+0000 7f9d54876740 1 -- 172.21.15.52:0/2987031063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 msgr2=0x562eb669dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.805+0000 7f9d54876740 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb669dc00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400a740 tx=0x7f9d44009650 comp rx=0 tx=0).stop 2024-04-26T02:05:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.805+0000 7f9d4aa00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 0x7f9d300a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 -- 172.21.15.52:0/2987031063 shutdown_connections 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 --2- 172.21.15.52:0/2987031063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb669dc00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 -- 172.21.15.52:0/2987031063 >> 172.21.15.52:0/2987031063 conn(0x562eb6699d30 msgr2=0x562eb669a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 -- 172.21.15.52:0/2987031063 shutdown_connections 2024-04-26T02:05:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d54876740 1 -- 172.21.15.52:0/2987031063 wait complete. 2024-04-26T02:05:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.806+0000 7f9d4aa00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 0x7f9d300a63d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x562eb669edc0 tx=0x7f9d3c0ba040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.807+0000 7f9d54876740 1 -- start start 2024-04-26T02:05:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.807+0000 7f9d54876740 1 Processor -- start 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669d820 0x562eb66a5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb66a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562eb66ab630 con 0x562eb669e140 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x562eb65bd310 con 0x562eb66a8a00 2024-04-26T02:05:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562eb66ab7b0 con 0x562eb669d820 2024-04-26T02:05:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d54876740 10 client.0 _ll_register_callbacks cb 0x562eb658bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43798/0 (socket says 172.21.15.52:43798) 2024-04-26T02:05:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d4b400640 1 -- 172.21.15.52:0/972872482 learned_addr learned my addr 172.21.15.52:0/972872482 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.808+0000 7f9d4b400640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669d820 0x562eb66a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 -- 172.21.15.52:0/972872482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669d820 msgr2=0x562eb66a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669d820 0x562eb66a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 -- 172.21.15.52:0/972872482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 msgr2=0x562eb66a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4a000640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb66a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb66a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 -- 172.21.15.52:0/972872482 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eb63e2620 con 0x562eb66a8a00 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4a000640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb66a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d4b400640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9d44000f80 tx=0x7f9d44014aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5744fb1d86a9b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.809+0000 7f9d34c00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.810+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44018070 con 0x562eb66a8a00 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.810+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f9d4401f880 con 0x562eb66a8a00 2024-04-26T02:05:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.810+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d4400ed50 con 0x562eb66a8a00 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.810+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f9d4401fa80 con 0x562eb66a8a00 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d54876740 10 client.27413 fetch_fsmap learned FSMap version 71 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x562eb66c88e0 con 0x562eb66a8a00 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d4400ed50 con 0x562eb66a8a00 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 71) ==== 8342+0+0 (secure 0 0 0) 0x7f9d4401db50 con 0x562eb66a8a00 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d54876740 10 client.27413 fetch_fsmap finished waiting for FSMap version 71 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.811+0000 7f9d54876740 10 client.27413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.812+0000 7f9d54876740 20 client.27413 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.812+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb66c9da0 0x562eb66cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.812+0000 7f9d54876740 4 client.27413 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.812+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562eb66178b0 con 0x562eb66c9da0 2024-04-26T02:05:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.812+0000 7f9d4a000640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb66c9da0 0x562eb66cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.813+0000 7f9d4a000640 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb66c9da0 0x562eb66cc190 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d32a57c9b7a89d server_cookie=3d1779882bfabcec in_seq=0 out_seq=0 2024-04-26T02:05:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.813+0000 7f9d34c00640 10 client.27413 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.814+0000 7f9d34c00640 1 -- 172.21.15.52:0/972872482 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562eb66178b0 con 0x562eb66c9da0 2024-04-26T02:05:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.814+0000 7f9d34c00640 10 client.27413 handle_command_reply: tid=0 2024-04-26T02:05:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.814+0000 7f9d54876740 1 client.27413 shutdown 2024-04-26T02:05:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 msgr2=0x562eb66a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9d44000f80 tx=0x7f9d44014aa0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 shutdown_connections 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb66c9da0 0x562eb66cc190 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb66a6410 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb66a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 --2- 172.21.15.52:0/972872482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669d820 0x562eb66a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 >> 172.21.15.52:0/972872482 conn(0x562eb6699d30 msgr2=0x562eb669fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d34c00640 0 client.27413 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 shutdown_connections 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 1 -- 172.21.15.52:0/972872482 wait complete. 2024-04-26T02:05:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.815+0000 7f9d54876740 20 client.27413 trim_cache size 0 max 16384 2024-04-26T02:05:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 Processor -- start 2024-04-26T02:05:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 -- start start 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb669d820 0x562eb66b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb6698d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1a3ad0 con 0x562eb669e140 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1a13a0 con 0x562eb669d820 2024-04-26T02:05:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.836+0000 7f9d54876740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c19f210 con 0x562eb66a8a00 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb669d820 0x562eb66b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb6698d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:33240/0 (socket says 172.21.15.52:33240) 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4b400640 1 -- 172.21.15.52:0/2991137426 learned_addr learned my addr 172.21.15.52:0/2991137426 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 -- 172.21.15.52:0/2991137426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 msgr2=0x562eb66d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 -- 172.21.15.52:0/2991137426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb669d820 msgr2=0x562eb66b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb669d820 0x562eb66b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4a000640 1 -- 172.21.15.52:0/2991137426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eb669df70 con 0x562eb669e140 2024-04-26T02:05:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.837+0000 7f9d4b400640 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:05:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.838+0000 7f9d4a000640 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb6698d10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x562eb669aec0 tx=0x7f9d38000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d35220e21cbb07e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.838+0000 7f9d37400640 1 -- 172.21.15.52:0/2991137426 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3804a070 con 0x562eb669e140 2024-04-26T02:05:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.838+0000 7f9d37400640 1 -- 172.21.15.52:0/2991137426 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d38005030 con 0x562eb669e140 2024-04-26T02:05:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.838+0000 7f9d37400640 1 -- 172.21.15.52:0/2991137426 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38035040 con 0x562eb669e140 2024-04-26T02:05:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.840+0000 7f9d54876740 1 -- 172.21.15.52:0/2991137426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 msgr2=0x562eb6698d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.840+0000 7f9d54876740 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb6698d10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x562eb669aec0 tx=0x7f9d38000bc0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.841+0000 7f9d54876740 1 -- 172.21.15.52:0/2991137426 shutdown_connections 2024-04-26T02:05:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.841+0000 7f9d54876740 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb66a8a00 0x562eb66d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.841+0000 7f9d54876740 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669e140 0x562eb6698d10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.841+0000 7f9d54876740 1 --2- 172.21.15.52:0/2991137426 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb669d820 0x562eb66b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.841+0000 7f9d54876740 1 -- 172.21.15.52:0/2991137426 >> 172.21.15.52:0/2991137426 conn(0x562eb6699d30 msgr2=0x562eb6698600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.842+0000 7f9d54876740 1 -- 172.21.15.52:0/2991137426 shutdown_connections 2024-04-26T02:05:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.842+0000 7f9d54876740 1 -- 172.21.15.52:0/2991137426 wait complete. 2024-04-26T02:05:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.842+0000 7f9d54876740 1 -- start start 2024-04-26T02:05:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.843+0000 7f9d54876740 1 Processor -- start 2024-04-26T02:05:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.844+0000 7f9d54876740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb66adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.844+0000 7f9d54876740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.844+0000 7f9d54876740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:43818/0 (socket says 172.21.15.52:43818) 2024-04-26T02:05:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 -- 172.21.15.52:0/4238194965 learned_addr learned my addr 172.21.15.52:0/4238194965 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:05:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c19f860 con 0x562eb669d820 2024-04-26T02:05:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c1a0cd0 con 0x562eb66a8a00 2024-04-26T02:05:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c19ff30 con 0x562eb669e140 2024-04-26T02:05:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb66adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4a000640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 -- 172.21.15.52:0/4238194965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 msgr2=0x562eb66ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 -- 172.21.15.52:0/4238194965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 msgr2=0x562eb66adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb66adbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.845+0000 7f9d4b400640 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eb672e660 con 0x562eb66a8a00 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.846+0000 7f9d54876740 10 client.0 _ll_register_callbacks cb 0x7f9d4400e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.846+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d38005030 con 0x562eb66a8a00 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.846+0000 7f9d4b400640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9d44014b10 tx=0x7f9d4400d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342009b0f63918ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.846+0000 7f9d34c00640 10 client.27419 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.846+0000 7f9d34c00640 1 -- 172.21.15.52:0/4238194965 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44018070 con 0x562eb66a8a00 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d34c00640 1 -- 172.21.15.52:0/4238194965 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8521+0+0 (secure 0 0 0) 0x7f9d4400ed50 con 0x562eb66a8a00 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d34c00640 1 -- 172.21.15.52:0/4238194965 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f9d4401c8e0 con 0x562eb66a8a00 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d54876740 10 client.27419 fetch_fsmap learned FSMap version 71 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({fsmap=71}) -- 0x562eb672f670 con 0x562eb66a8a00 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d34c00640 1 -- 172.21.15.52:0/4238194965 <== mon.2 v2:172.21.15.183:3300/0 4 ==== fsmap(e 71) ==== 8342+0+0 (secure 0 0 0) 0x7f9d4401ca80 con 0x562eb66a8a00 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d54876740 10 client.27419 fetch_fsmap finished waiting for FSMap version 71 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d54876740 10 client.27419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.847+0000 7f9d54876740 20 client.27419 populate_metadata read hostname 'smithi052' 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.848+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb672fc00 0x562eb67320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.848+0000 7f9d54876740 4 client.27419 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:05:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.848+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562eb6611320 con 0x562eb672fc00 2024-04-26T02:05:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.848+0000 7f9d4a000640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb672fc00 0x562eb67320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:05:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.849+0000 7f9d4a000640 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb672fc00 0x562eb67320c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34099883f3d25ad5 server_cookie=6e022cc069892a3b in_seq=0 out_seq=0 2024-04-26T02:05:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:53.849+0000 7f9d34c00640 10 client.27419 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.068+0000 7f9d34c00640 1 -- 172.21.15.52:0/4238194965 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562eb6611320 con 0x562eb672fc00 2024-04-26T02:05:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.068+0000 7f9d34c00640 10 client.27419 handle_command_reply: tid=0 2024-04-26T02:05:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.068+0000 7f9d54876740 1 client.27419 shutdown 2024-04-26T02:05:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 msgr2=0x562eb672e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9d44014b10 tx=0x7f9d4400d040 comp rx=0 tx=0).stop 2024-04-26T02:05:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 shutdown_connections 2024-04-26T02:05:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x562eb672fc00 0x562eb67320c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x562eb66a8a00 0x562eb672e120 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562eb669e140 0x562eb66ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 --2- 172.21.15.52:0/4238194965 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562eb669d820 0x562eb66adbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 >> 172.21.15.52:0/4238194965 conn(0x562eb6699d30 msgr2=0x562eb6698e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d34c00640 0 client.27419 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.071+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 shutdown_connections 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.072+0000 7f9d54876740 1 -- 172.21.15.52:0/4238194965 wait complete. 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.072+0000 7f9d54876740 20 client.27419 trim_cache size 0 max 16384 2024-04-26T02:05:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:05:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:05:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:05:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 msgr2=0x7f9d300a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 0x7f9d300a63d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x562eb669edc0 tx=0x7f9d3c0ba040 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 msgr2=0x7f9d4c10ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c02f960 tx=0x7f9d3c02fe20 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 shutdown_connections 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9d300a3f10 0x7f9d300a63d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d4c10c3c0 0x7f9d4c10c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d4c1072b0 0x7f9d4c10be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 --2- 172.21.15.52:0/1486116136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9d4c1021e0 0x7f9d4c10ed90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.073+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 >> 172.21.15.52:0/1486116136 conn(0x7f9d4c0fda00 msgr2=0x7f9d4c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.074+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 shutdown_connections 2024-04-26T02:05:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:05:54.074+0000 7f9d51a00640 1 -- 172.21.15.52:0/1486116136 wait complete. 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T02:05:54.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:53 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T02:05:54.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T02:05:54.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T02:05:54.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:53 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T02:05:54.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi183 to 176.2M 2024-04-26T02:05:54.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi183 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-04-26T02:05:54.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi191 to 176.2M 2024-04-26T02:05:54.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:53 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi191 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-26T02:05:54.764 INFO:tasks.workunit.client.0.smithi052.stdout: 10 160035 20.74 MB/sec execute 311 sec latency 157.702 ms 2024-04-26T02:05:55.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:54 smithi191 ceph-mon[32745]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 425 op/s 2024-04-26T02:05:55.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:54 smithi183 ceph-mon[32752]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 425 op/s 2024-04-26T02:05:55.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:54 smithi052 ceph-mon[27851]: pgmap v662: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 425 op/s 2024-04-26T02:05:55.765 INFO:tasks.workunit.client.0.smithi052.stdout: 10 160286 20.70 MB/sec execute 312 sec latency 496.920 ms 2024-04-26T02:05:56.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 160667 20.68 MB/sec execute 313 sec latency 691.886 ms 2024-04-26T02:05:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:57 smithi183 ceph-mon[32752]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 663 op/s 2024-04-26T02:05:57.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:57 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:57.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:57 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:57 smithi052 ceph-mon[27851]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 663 op/s 2024-04-26T02:05:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:57 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:57.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:57 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:57.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 161309 20.75 MB/sec execute 314 sec latency 135.729 ms 2024-04-26T02:05:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:57 smithi191 ceph-mon[32745]: pgmap v663: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 663 op/s 2024-04-26T02:05:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:57 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:57 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:58.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:58.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:58 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:58.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 161686 20.77 MB/sec execute 315 sec latency 314.209 ms 2024-04-26T02:05:58.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:58.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:58 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:05:58.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:58 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:05:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:59 smithi052 ceph-mon[27851]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T02:05:59.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:59 smithi052 ceph-mon[27851]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:05:59.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:59 smithi052 ceph-mon[27851]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 637 op/s 2024-04-26T02:05:59.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:05:59 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:05:59.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 162114 20.75 MB/sec execute 316 sec latency 72.886 ms 2024-04-26T02:05:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:59 smithi191 ceph-mon[32745]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T02:05:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:59 smithi191 ceph-mon[32745]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:05:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:59 smithi191 ceph-mon[32745]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 637 op/s 2024-04-26T02:05:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:05:59 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:05:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:59 smithi183 ceph-mon[32752]: Adjusting osd_memory_target on smithi052 to 177.5M 2024-04-26T02:05:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:59 smithi183 ceph-mon[32752]: Unable to set osd_memory_target on smithi052 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T02:05:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:59 smithi183 ceph-mon[32752]: pgmap v664: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 MiB/s wr, 637 op/s 2024-04-26T02:05:59.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:05:59 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:00.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 162326 20.71 MB/sec execute 317 sec latency 543.092 ms 2024-04-26T02:06:01.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:01 smithi052 ceph-mon[27851]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 612 op/s 2024-04-26T02:06:01.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 162326 20.64 MB/sec execute 318 sec latency 1543.227 ms 2024-04-26T02:06:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:01 smithi191 ceph-mon[32745]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 612 op/s 2024-04-26T02:06:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:01 smithi183 ceph-mon[32752]: pgmap v665: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 14 MiB/s wr, 612 op/s 2024-04-26T02:06:02.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 162326 20.58 MB/sec execute 319 sec latency 2543.401 ms 2024-04-26T02:06:03.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:03 smithi191 ceph-mon[32745]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 817 op/s 2024-04-26T02:06:03.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:03 smithi052 ceph-mon[27851]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 817 op/s 2024-04-26T02:06:03.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 162787 20.58 MB/sec execute 320 sec latency 2762.117 ms 2024-04-26T02:06:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:03 smithi183 ceph-mon[32752]: pgmap v666: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 MiB/s wr, 817 op/s 2024-04-26T02:06:04.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 163057 20.55 MB/sec execute 321 sec latency 184.107 ms 2024-04-26T02:06:05.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:05 smithi052 ceph-mon[27851]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.8 MiB/s wr, 582 op/s 2024-04-26T02:06:05.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 163357 20.55 MB/sec execute 322 sec latency 166.323 ms 2024-04-26T02:06:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:05 smithi191 ceph-mon[32745]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.8 MiB/s wr, 582 op/s 2024-04-26T02:06:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:05 smithi183 ceph-mon[32752]: pgmap v667: 97 pgs: 97 active+clean; 1.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 8.8 MiB/s wr, 582 op/s 2024-04-26T02:06:06.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 163899 20.59 MB/sec execute 323 sec latency 142.355 ms 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.276+0000 7fcd80600640 1 -- 172.21.15.52:0/2111735132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd78109470 msgr2=0x7fcd78111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.276+0000 7fcd80600640 1 --2- 172.21.15.52:0/2111735132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd78109470 0x7fcd78111040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcd68009900 tx=0x7fcd6802a410 comp rx=0 tx=0).stop 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 -- 172.21.15.52:0/2111735132 shutdown_connections 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 --2- 172.21.15.52:0/2111735132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd78109470 0x7fcd78111040 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 --2- 172.21.15.52:0/2111735132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd78104d50 0x7fcd78108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 --2- 172.21.15.52:0/2111735132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd781043a0 0x7fcd78104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 -- 172.21.15.52:0/2111735132 >> 172.21.15.52:0/2111735132 conn(0x7fcd780ffc40 msgr2=0x7fcd78102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 -- 172.21.15.52:0/2111735132 shutdown_connections 2024-04-26T02:06:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.277+0000 7fcd80600640 1 -- 172.21.15.52:0/2111735132 wait complete. 2024-04-26T02:06:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.279+0000 7fcd80600640 1 Processor -- start 2024-04-26T02:06:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.279+0000 7fcd80600640 1 -- start start 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.279+0000 7fcd80600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.279+0000 7fcd80600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 0x7fcd7810de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.279+0000 7fcd80600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd7810e360 0x7fcd7810e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd80600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78114480 con 0x7fcd78109470 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd80600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78114300 con 0x7fcd7810e360 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd80600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78114600 con 0x7fcd781043a0 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7e600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd7810e360 0x7fcd7810e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50606/0 (socket says 172.21.15.52:50606) 2024-04-26T02:06:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 -- 172.21.15.52:0/535534748 learned_addr learned my addr 172.21.15.52:0/535534748 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7d200640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 0x7fcd7810de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 -- 172.21.15.52:0/535534748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd7810e360 msgr2=0x7fcd7810e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd7810e360 0x7fcd7810e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.280+0000 7fcd7dc00640 1 -- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 msgr2=0x7fcd7810de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.281+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 0x7fcd7810de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.281+0000 7fcd7dc00640 1 -- 172.21.15.52:0/535534748 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd781ab210 con 0x7fcd781043a0 2024-04-26T02:06:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.282+0000 7fcd7d200640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 0x7fcd7810de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.281+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000ebc0 tx=0x7fcd7000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.283+0000 7fcd76000640 1 -- 172.21.15.52:0/535534748 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70010630 con 0x7fcd781043a0 2024-04-26T02:06:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.283+0000 7fcd76000640 1 -- 172.21.15.52:0/535534748 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd70010c80 con 0x7fcd781043a0 2024-04-26T02:06:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.283+0000 7fcd76000640 1 -- 172.21.15.52:0/535534748 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70018b90 con 0x7fcd781043a0 2024-04-26T02:06:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.283+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd781ab500 con 0x7fcd781043a0 2024-04-26T02:06:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.283+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd781ab9d0 con 0x7fcd781043a0 2024-04-26T02:06:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.290+0000 7fcd76000640 1 -- 172.21.15.52:0/535534748 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fcd700107d0 con 0x7fcd781043a0 2024-04-26T02:06:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.292+0000 7fcd835e8740 1 Processor -- start 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.293+0000 7fcd76000640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 0x7fcd600a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.293+0000 7fcd835e8740 1 -- start start 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd7e600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 0x7fcd600a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd76000640 1 -- 172.21.15.52:0/535534748 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8565+0+0 (secure 0 0 0) 0x7fcd700d0da0 con 0x7fcd781043a0 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2e820 0x55f27ee2ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee394c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f27ed94900 con 0x55f27ee2e820 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f27ed3aef0 con 0x55f27ee2f140 2024-04-26T02:06:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.294+0000 7fcd835e8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f27ed908c0 con 0x55f27ee39a00 2024-04-26T02:06:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.295+0000 7fcd7d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.295+0000 7fcd7d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee394c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37114/0 (socket says 172.21.15.52:37114) 2024-04-26T02:06:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.295+0000 7fcd7e600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.295+0000 7fcd7e600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50620/0 (socket says 172.21.15.52:50620) 2024-04-26T02:06:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.295+0000 7fcd7d200640 1 -- 172.21.15.52:0/820994703 learned_addr learned my addr 172.21.15.52:0/820994703 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.296+0000 7fcd7e600640 1 -- 172.21.15.52:0/820994703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 msgr2=0x55f27ee394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.296+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2e820 0x55f27ee2ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.296+0000 7fcd7e600640 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.296+0000 7fcd7e600640 1 -- 172.21.15.52:0/820994703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2e820 msgr2=0x55f27ee2ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.297+0000 7fcd7e600640 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2e820 0x55f27ee2ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.297+0000 7fcd7e600640 1 -- 172.21.15.52:0/820994703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f27eaa7210 con 0x55f27ee39a00 2024-04-26T02:06:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.297+0000 7fcd7e600640 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcd68005ab0 tx=0x7fcd68045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f83302775905adc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.298+0000 7fcd66a00640 1 -- 172.21.15.52:0/820994703 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68035040 con 0x55f27ee39a00 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.298+0000 7fcd7e600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 0x7fcd600a6460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcd680096c0 tx=0x7fcd680041b0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.298+0000 7fcd66a00640 1 -- 172.21.15.52:0/820994703 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd68003d40 con 0x55f27ee39a00 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.298+0000 7fcd66a00640 1 -- 172.21.15.52:0/820994703 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68048710 con 0x55f27ee39a00 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 -- 172.21.15.52:0/820994703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 msgr2=0x55f27ee3bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcd68005ab0 tx=0x7fcd68045000 comp rx=0 tx=0).stop 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 -- 172.21.15.52:0/820994703 shutdown_connections 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee3bdf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 --2- 172.21.15.52:0/820994703 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2e820 0x55f27ee2ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.299+0000 7fcd835e8740 1 -- 172.21.15.52:0/820994703 >> 172.21.15.52:0/820994703 conn(0x55f27ee2ad30 msgr2=0x55f27ee2b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.300+0000 7fcd835e8740 1 -- 172.21.15.52:0/820994703 shutdown_connections 2024-04-26T02:06:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.300+0000 7fcd835e8740 1 -- 172.21.15.52:0/820994703 wait complete. 2024-04-26T02:06:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.301+0000 7fcd835e8740 1 -- start start 2024-04-26T02:06:07.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.301+0000 7fcd835e8740 1 Processor -- start 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee36f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 0x55f27ee37470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f27ee3c5d0 con 0x55f27ee2f140 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55f27ed4e310 con 0x55f27ee39a00 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55f27ee3c750 con 0x55f27ee2e820 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.303+0000 7fcd835e8740 10 client.0 _ll_register_callbacks cb 0x55f27ed1ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:07.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.304+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.304+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50626/0 (socket says 172.21.15.52:50626) 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.304+0000 7fcd7dc00640 1 -- 172.21.15.52:0/2845684898 learned_addr learned my addr 172.21.15.52:0/2845684898 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.304+0000 7fcd7d200640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee36f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 0x55f27ee37470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 -- 172.21.15.52:0/2845684898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 msgr2=0x55f27ee369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 -- 172.21.15.52:0/2845684898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 msgr2=0x55f27ee36f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee36f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7e600640 1 -- 172.21.15.52:0/2845684898 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f27eb73620 con 0x55f27ee39a00 2024-04-26T02:06:07.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.305+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.306+0000 7fcd7e600640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 0x55f27ee37470 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55f27ee2c150 tx=0x7fcd6802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e73299a9b956d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.306+0000 7fcd5b400640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.306+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd680488a0 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.307+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 8565+0+0 (secure 0 0 0) 0x7fcd680486a0 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.307+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd68003d40 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.307+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fcd680488a0 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.307+0000 7fcd835e8740 10 client.27431 fetch_fsmap learned FSMap version 72 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.307+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55f27ee3c5d0 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.308+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd68003d40 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.308+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 72) ==== 8338+0+0 (secure 0 0 0) 0x7fcd6803cbd0 con 0x55f27ee39a00 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 10 client.27431 fetch_fsmap finished waiting for FSMap version 72 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 10 client.27431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 20 client.27431 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27ee49a10 0x55f27ee4be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 4 client.27431 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f27eda88b0 con 0x55f27ee49a10 2024-04-26T02:06:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.309+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27ee49a10 0x55f27ee4be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.312+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27ee49a10 0x55f27ee4be00 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66e6d1c05c0f934b server_cookie=c30902537ccebf2f in_seq=0 out_seq=0 2024-04-26T02:06:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.312+0000 7fcd5b400640 10 client.27431 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.313+0000 7fcd5b400640 1 -- 172.21.15.52:0/2845684898 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f27eda88b0 con 0x55f27ee49a10 2024-04-26T02:06:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.313+0000 7fcd5b400640 10 client.27431 handle_command_reply: tid=0 2024-04-26T02:06:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.314+0000 7fcd835e8740 1 client.27431 shutdown 2024-04-26T02:06:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 msgr2=0x55f27ee37470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 0x55f27ee37470 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55f27ee2c150 tx=0x7fcd6802a410 comp rx=0 tx=0).stop 2024-04-26T02:06:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 shutdown_connections 2024-04-26T02:06:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27ee49a10 0x55f27ee4be00 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee39a00 0x55f27ee37470 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee36f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 --2- 172.21.15.52:0/2845684898 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 >> 172.21.15.52:0/2845684898 conn(0x55f27ee2ad30 msgr2=0x55f27ee30ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.315+0000 7fcd5b400640 0 client.27431 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.316+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 shutdown_connections 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.317+0000 7fcd835e8740 1 -- 172.21.15.52:0/2845684898 wait complete. 2024-04-26T02:06:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.317+0000 7fcd835e8740 20 client.27431 trim_cache size 0 max 16384 2024-04-26T02:06:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.350+0000 7fcd835e8740 1 Processor -- start 2024-04-26T02:06:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 -- start start 2024-04-26T02:06:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 0x55f27ee45490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee37290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee29d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b6890 con 0x55f27ee2f140 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b4160 con 0x55f27ee2e820 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.351+0000 7fcd835e8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b1fd0 con 0x55f27ee39a00 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 0x55f27ee45490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee29d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 0x55f27ee45490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37126/0 (socket says 172.21.15.52:37126) 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7dc00640 1 -- 172.21.15.52:0/1996285794 learned_addr learned my addr 172.21.15.52:0/1996285794 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7d200640 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee37290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 -- 172.21.15.52:0/1996285794 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 msgr2=0x55f27ee45490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 0x55f27ee45490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 -- 172.21.15.52:0/1996285794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 msgr2=0x55f27ee37290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee37290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.352+0000 7fcd7e600640 1 -- 172.21.15.52:0/1996285794 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f27ee3c120 con 0x55f27ee39a00 2024-04-26T02:06:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.353+0000 7fcd7e600640 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee29d10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd680573f0 tx=0x7fcd68047fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84402bd690fdc2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.355+0000 7fcd66000640 1 -- 172.21.15.52:0/1996285794 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68035070 con 0x55f27ee39a00 2024-04-26T02:06:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.355+0000 7fcd66000640 1 -- 172.21.15.52:0/1996285794 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd68054070 con 0x55f27ee39a00 2024-04-26T02:06:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.355+0000 7fcd66000640 1 -- 172.21.15.52:0/1996285794 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68035070 con 0x55f27ee39a00 2024-04-26T02:06:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.356+0000 7fcd835e8740 1 -- 172.21.15.52:0/1996285794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 msgr2=0x55f27ee29d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.356+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee29d10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd680573f0 tx=0x7fcd68047fd0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 -- 172.21.15.52:0/1996285794 shutdown_connections 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee39a00 0x55f27ee29d10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee2f140 0x55f27ee37290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1996285794 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2e820 0x55f27ee45490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 -- 172.21.15.52:0/1996285794 >> 172.21.15.52:0/1996285794 conn(0x55f27ee2ad30 msgr2=0x55f27ee29600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 -- 172.21.15.52:0/1996285794 shutdown_connections 2024-04-26T02:06:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.361+0000 7fcd835e8740 1 -- 172.21.15.52:0/1996285794 wait complete. 2024-04-26T02:06:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.362+0000 7fcd835e8740 1 -- start start 2024-04-26T02:06:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.362+0000 7fcd835e8740 1 Processor -- start 2024-04-26T02:06:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.363+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.363+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee3f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.363+0000 7fcd835e8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee39a00 0x55f27ee3f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee3f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd835e8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b2620 con 0x55f27ee39a00 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd835e8740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b3a90 con 0x55f27ee2f140 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd835e8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcd781b2cf0 con 0x55f27ee2e820 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd835e8740 10 client.0 _ll_register_callbacks cb 0x7fcd68004000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:50650/0 (socket says 172.21.15.52:50650) 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7d200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee3f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:37142/0 (socket says 172.21.15.52:37142) 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 -- 172.21.15.52:0/1772034962 learned_addr learned my addr 172.21.15.52:0/1772034962 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 -- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 msgr2=0x55f27ee3f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee3f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 -- 172.21.15.52:0/1772034962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee39a00 msgr2=0x55f27ee3f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee39a00 0x55f27ee3f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.364+0000 7fcd7dc00640 1 -- 172.21.15.52:0/1772034962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f27ee3fe10 con 0x55f27ee2e820 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.365+0000 7fcd7dc00640 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd70099680 tx=0x7fcd7001f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=363bab6435f4b37b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.365+0000 7fcd5b400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:06:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.365+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7009d3b0 con 0x55f27ee2e820 2024-04-26T02:06:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.366+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 8565+0+0 (secure 0 0 0) 0x7fcd700ce6e0 con 0x55f27ee2e820 2024-04-26T02:06:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.366+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd68054070 con 0x55f27ee2e820 2024-04-26T02:06:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.366+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fcd7009ccb0 con 0x55f27ee2e820 2024-04-26T02:06:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.366+0000 7fcd835e8740 10 client.27460 fetch_fsmap learned FSMap version 72 2024-04-26T02:06:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.367+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x55f27eeafdf0 con 0x55f27ee2e820 2024-04-26T02:06:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.367+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd68054070 con 0x55f27ee2e820 2024-04-26T02:06:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 72) ==== 8338+0+0 (secure 0 0 0) 0x7fcd700cebb0 con 0x55f27ee2e820 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 10 client.27460 fetch_fsmap finished waiting for FSMap version 72 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 10 client.27460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 20 client.27460 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27eeb0470 0x55f27eeb28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 4 client.27460 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.368+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f27eda2320 con 0x55f27eeb0470 2024-04-26T02:06:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.369+0000 7fcd7d200640 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27eeb0470 0x55f27eeb28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.371+0000 7fcd7d200640 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27eeb0470 0x55f27eeb28b0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=501e337e9d15f6d8 server_cookie=f6ce75c1fdf41f60 in_seq=0 out_seq=0 2024-04-26T02:06:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.371+0000 7fcd5b400640 10 client.27460 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.725+0000 7fcd5b400640 1 -- 172.21.15.52:0/1772034962 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f27eda2320 con 0x55f27eeb0470 2024-04-26T02:06:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.725+0000 7fcd5b400640 10 client.27460 handle_command_reply: tid=0 2024-04-26T02:06:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.725+0000 7fcd835e8740 1 client.27460 shutdown 2024-04-26T02:06:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.725+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 msgr2=0x55f27ee41d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.725+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd70099680 tx=0x7fcd7001f000 comp rx=0 tx=0).stop 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 shutdown_connections 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55f27eeb0470 0x55f27eeb28b0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd5b400640 0 client.27460 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f27ee39a00 0x55f27ee3f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55f27ee2f140 0x55f27ee3f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 --2- 172.21.15.52:0/1772034962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55f27ee2e820 0x55f27ee41d60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 >> 172.21.15.52:0/1772034962 conn(0x55f27ee2ad30 msgr2=0x55f27ee29e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.727+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 shutdown_connections 2024-04-26T02:06:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.728+0000 7fcd835e8740 1 -- 172.21.15.52:0/1772034962 wait complete. 2024-04-26T02:06:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.728+0000 7fcd835e8740 20 client.27460 trim_cache size 0 max 16384 2024-04-26T02:06:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:06:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:06:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:06:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:06:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 msgr2=0x7fcd600a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 0x7fcd600a6460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcd680096c0 tx=0x7fcd680041b0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 msgr2=0x7fcd78110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000ebc0 tx=0x7fcd7000c6a0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 shutdown_connections 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fcd600a3fa0 0x7fcd600a6460 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcd7810e360 0x7fcd7810e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd78109470 0x7fcd7810de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 --2- 172.21.15.52:0/535534748 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd781043a0 0x7fcd78110ce0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 >> 172.21.15.52:0/535534748 conn(0x7fcd780ffc40 msgr2=0x7fcd781004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 shutdown_connections 2024-04-26T02:06:07.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:07.730+0000 7fcd80600640 1 -- 172.21.15.52:0/535534748 wait complete. 2024-04-26T02:06:07.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 164133 20.58 MB/sec execute 324 sec latency 307.207 ms 2024-04-26T02:06:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:07 smithi191 ceph-mon[32745]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-04-26T02:06:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:07 smithi191 ceph-mon[32745]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:07 smithi183 ceph-mon[32752]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-04-26T02:06:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:07 smithi183 ceph-mon[32752]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:07 smithi052 ceph-mon[27851]: pgmap v668: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 758 op/s 2024-04-26T02:06:07.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:07 smithi052 ceph-mon[27851]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T02:06:08.675 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:08 smithi183 ceph-mon[32752]: daemon mds.h restarted 2024-04-26T02:06:08.676 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:06:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T02:06:08.676 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:06:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T02:06:08.676 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:06:08 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/1148160181 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ea1697a40 con 0x560ea166a800 2024-04-26T02:06:08.676 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:06:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T02:06:08.766 INFO:tasks.workunit.client.0.smithi052.stdout: 10 164396 20.57 MB/sec execute 325 sec latency 208.600 ms 2024-04-26T02:06:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:08 smithi191 ceph-mon[32745]: daemon mds.h restarted 2024-04-26T02:06:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:08 smithi052 ceph-mon[27851]: daemon mds.h restarted 2024-04-26T02:06:09.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 164886 20.59 MB/sec execute 326 sec latency 186.092 ms 2024-04-26T02:06:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:09 smithi191 ceph-mon[32745]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-26T02:06:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:09 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:06:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:09 smithi191 ceph-mon[32745]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:09 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/1603134393,v1:172.21.15.183:6837/1603134393] up:boot 2024-04-26T02:06:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:09 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:09 smithi183 ceph-mon[32752]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-26T02:06:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:09 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:06:09.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:09 smithi183 ceph-mon[32752]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:09.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:09 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/1603134393,v1:172.21.15.183:6837/1603134393] up:boot 2024-04-26T02:06:09.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:09 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:09 smithi052 ceph-mon[27851]: pgmap v670: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-26T02:06:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:09 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:06:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:09 smithi052 ceph-mon[27851]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T02:06:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:09 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/1603134393,v1:172.21.15.183:6837/1603134393] up:boot 2024-04-26T02:06:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:09 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:06:10.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 165364 20.60 MB/sec execute 327 sec latency 108.787 ms 2024-04-26T02:06:11.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 165802 20.60 MB/sec execute 328 sec latency 93.198 ms 2024-04-26T02:06:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:11 smithi191 ceph-mon[32745]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-04-26T02:06:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:11 smithi183 ceph-mon[32752]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-04-26T02:06:11.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:11 smithi052 ceph-mon[27851]: pgmap v672: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 432 op/s 2024-04-26T02:06:12.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 166324 20.62 MB/sec execute 329 sec latency 93.705 ms 2024-04-26T02:06:13.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 166914 20.65 MB/sec execute 330 sec latency 110.373 ms 2024-04-26T02:06:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:13 smithi191 ceph-mon[32745]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 750 op/s 2024-04-26T02:06:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:13 smithi191 ceph-mon[32745]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:13 smithi183 ceph-mon[32752]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 750 op/s 2024-04-26T02:06:13.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:13 smithi183 ceph-mon[32752]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:13 smithi052 ceph-mon[27851]: pgmap v673: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 6.6 MiB/s wr, 750 op/s 2024-04-26T02:06:13.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:13 smithi052 ceph-mon[27851]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T02:06:14.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 167539 20.69 MB/sec execute 331 sec latency 74.643 ms 2024-04-26T02:06:15.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 168154 20.73 MB/sec execute 332 sec latency 90.747 ms 2024-04-26T02:06:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:15 smithi191 ceph-mon[32745]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 322 op/s 2024-04-26T02:06:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:15 smithi183 ceph-mon[32752]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 322 op/s 2024-04-26T02:06:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:15 smithi052 ceph-mon[27851]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.4 MiB/s wr, 322 op/s 2024-04-26T02:06:16.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 168644 20.75 MB/sec execute 333 sec latency 137.717 ms 2024-04-26T02:06:17.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:17 smithi052 ceph-mon[27851]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 542 op/s 2024-04-26T02:06:17.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 169216 20.79 MB/sec execute 334 sec latency 192.688 ms 2024-04-26T02:06:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:17 smithi191 ceph-mon[32745]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 542 op/s 2024-04-26T02:06:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:17 smithi183 ceph-mon[32752]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.6 MiB/s wr, 542 op/s 2024-04-26T02:06:18.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 169640 20.79 MB/sec execute 335 sec latency 127.896 ms 2024-04-26T02:06:19.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 170129 20.81 MB/sec execute 336 sec latency 88.563 ms 2024-04-26T02:06:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:19 smithi191 ceph-mon[32745]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 480 op/s 2024-04-26T02:06:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:19 smithi183 ceph-mon[32752]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 480 op/s 2024-04-26T02:06:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:19 smithi052 ceph-mon[27851]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.9 MiB/s wr, 480 op/s 2024-04-26T02:06:20.768 INFO:tasks.workunit.client.0.smithi052.stdout: 10 170684 20.82 MB/sec execute 337 sec latency 74.769 ms 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.920+0000 7f4a43200640 1 -- 172.21.15.52:0/364499239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a44074300 msgr2=0x7f4a440746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.920+0000 7f4a43200640 1 --2- 172.21.15.52:0/364499239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a44074300 0x7f4a440746e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009900 tx=0x7f4a3402a480 comp rx=0 tx=0).stop 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 -- 172.21.15.52:0/364499239 shutdown_connections 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 --2- 172.21.15.52:0/364499239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a4410e630 0x7f4a44110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 --2- 172.21.15.52:0/364499239 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074c20 0x7f4a4410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 --2- 172.21.15.52:0/364499239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a44074300 0x7f4a440746e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 -- 172.21.15.52:0/364499239 >> 172.21.15.52:0/364499239 conn(0x7f4a440fd8c0 msgr2=0x7f4a440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 -- 172.21.15.52:0/364499239 shutdown_connections 2024-04-26T02:06:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.921+0000 7f4a43200640 1 -- 172.21.15.52:0/364499239 wait complete. 2024-04-26T02:06:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 Processor -- start 2024-04-26T02:06:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 -- start start 2024-04-26T02:06:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 0x7f4a441a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a44074c20 0x7f4a441a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 0x7f4a4419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44114240 con 0x7f4a44074300 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441140c0 con 0x7f4a44074c20 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a43200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441143c0 con 0x7f4a4410e630 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 0x7f4a441a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a42800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 0x7f4a4419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 0x7f4a441a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47550/0 (socket says 172.21.15.52:47550) 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a41e00640 1 -- 172.21.15.52:0/218120422 learned_addr learned my addr 172.21.15.52:0/218120422 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.922+0000 7f4a41400640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a44074c20 0x7f4a441a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 -- 172.21.15.52:0/218120422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a44074c20 msgr2=0x7f4a441a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a44074c20 0x7f4a441a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 -- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 msgr2=0x7f4a441a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 0x7f4a441a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 -- 172.21.15.52:0/218120422 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4419ad60 con 0x7f4a4410e630 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a42800640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 0x7f4a4419a5e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009fd0 tx=0x7f4a3c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a3aa00640 1 -- 172.21.15.52:0/218120422 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00ce20 con 0x7f4a4410e630 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a3aa00640 1 -- 172.21.15.52:0/218120422 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c005020 con 0x7f4a4410e630 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4419b050 con 0x7f4a4410e630 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.923+0000 7f4a3aa00640 1 -- 172.21.15.52:0/218120422 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c010690 con 0x7f4a4410e630 2024-04-26T02:06:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.924+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a441a72a0 con 0x7f4a4410e630 2024-04-26T02:06:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a3aa00640 1 -- 172.21.15.52:0/218120422 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4a3c0054d0 con 0x7f4a4410e630 2024-04-26T02:06:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 Processor -- start 2024-04-26T02:06:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 -- start start 2024-04-26T02:06:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f44650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f44b90 0x564e70f46f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564e70e9f900 con 0x564e70f44b90 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564e70e45ef0 con 0x564e70f3a2d0 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564e70e9b8c0 con 0x564e70f399b0 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a41400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a41400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58868/0 (socket says 172.21.15.52:58868) 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a41400640 1 -- 172.21.15.52:0/3714071344 learned_addr learned my addr 172.21.15.52:0/3714071344 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a41e00640 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f44650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.927+0000 7f4a41400640 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f44b90 0x564e70f46f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 -- 172.21.15.52:0/3714071344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 msgr2=0x564e70f44650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f44650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 -- 172.21.15.52:0/3714071344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f44b90 msgr2=0x564e70f46f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f44b90 0x564e70f46f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 -- 172.21.15.52:0/3714071344 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e70bb2210 con 0x564e70f399b0 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a41400640 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4a300058c0 tx=0x7f4a30005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c062fc5c43d863a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a2f000640 1 -- 172.21.15.52:0/3714071344 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30004020 con 0x564e70f399b0 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a3aa00640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 0x7f4a280a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a2f000640 1 -- 172.21.15.52:0/3714071344 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300054d0 con 0x564e70f399b0 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a2f000640 1 -- 172.21.15.52:0/3714071344 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30004c60 con 0x564e70f399b0 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.928+0000 7f4a3aa00640 1 -- 172.21.15.52:0/218120422 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f4a3c0d0e90 con 0x7f4a4410e630 2024-04-26T02:06:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a41e00640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 0x7f4a280a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3714071344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 msgr2=0x564e70f39d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4a300058c0 tx=0x7f4a30005850 comp rx=0 tx=0).stop 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3714071344 shutdown_connections 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f44b90 0x564e70f46f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f44650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3714071344 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f39d90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3714071344 >> 172.21.15.52:0/3714071344 conn(0x564e70f35ec0 msgr2=0x564e70f362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3714071344 shutdown_connections 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3714071344 wait complete. 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.929+0000 7f4a41e00640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 0x7f4a280a61d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564e70f3af50 tx=0x7f4a34005a20 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 -- start start 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 Processor -- start 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f399b0 0x564e70f41b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f42070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564e70f477c0 con 0x564e70f399b0 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x564e70e59310 con 0x564e70f3a2d0 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.930+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564e70f47940 con 0x564e70f44b90 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a4b7d5740 10 client.0 _ll_register_callbacks cb 0x564e70e27cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f399b0 0x564e70f41b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a41400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f42070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58878/0 (socket says 172.21.15.52:58878) 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 -- 172.21.15.52:0/3255668533 learned_addr learned my addr 172.21.15.52:0/3255668533 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 -- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 msgr2=0x564e70f42070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f42070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 -- 172.21.15.52:0/3255668533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f399b0 msgr2=0x564e70f41b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f399b0 0x564e70f41b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 -- 172.21.15.52:0/3255668533 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e70c7e620 con 0x564e70f44b90 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a42800640 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c000f80 tx=0x7f4a3c09cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95c5640c4d43b649 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a23400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.931+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0b6260 con 0x564e70f44b90 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f4a3c099440 con 0x564e70f44b90 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a300054d0 con 0x564e70f44b90 2024-04-26T02:06:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f4a3c0a55b0 con 0x564e70f44b90 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 10 client.27478 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x564e70f53ae0 con 0x564e70f44b90 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a300054d0 con 0x564e70f44b90 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f4a3c09bc90 con 0x564e70f44b90 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 10 client.27478 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 10 client.27478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 20 client.27478 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70f54fa0 0x564e70f57390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 4 client.27478 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.932+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e70eb38b0 con 0x564e70f54fa0 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.933+0000 7f4a41e00640 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70f54fa0 0x564e70f57390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.933+0000 7f4a41e00640 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70f54fa0 0x564e70f57390 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26dcce0ab94399b2 server_cookie=e83d60df8a541f76 in_seq=0 out_seq=0 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.933+0000 7f4a23400640 10 client.27478 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.934+0000 7f4a23400640 1 -- 172.21.15.52:0/3255668533 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e70eb38b0 con 0x564e70f54fa0 2024-04-26T02:06:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.934+0000 7f4a23400640 10 client.27478 handle_command_reply: tid=0 2024-04-26T02:06:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.934+0000 7f4a4b7d5740 1 client.27478 shutdown 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 msgr2=0x564e70f425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c000f80 tx=0x7f4a3c09cf10 comp rx=0 tx=0).stop 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 shutdown_connections 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70f54fa0 0x564e70f57390 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f425b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f3a2d0 0x564e70f42070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/3255668533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f399b0 0x564e70f41b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 >> 172.21.15.52:0/3255668533 conn(0x564e70f35ec0 msgr2=0x564e70f3bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a23400640 0 client.27478 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 shutdown_connections 2024-04-26T02:06:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/3255668533 wait complete. 2024-04-26T02:06:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.935+0000 7f4a4b7d5740 20 client.27478 trim_cache size 0 max 16384 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 Processor -- start 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 -- start start 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70f3c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441b2490 con 0x564e70f3a2d0 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441afd60 con 0x564e70f44b90 2024-04-26T02:06:20.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441adbd0 con 0x564e70f399b0 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a42800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a41400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70f3c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58894/0 (socket says 172.21.15.52:58894) 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a41e00640 1 -- 172.21.15.52:0/1225955908 learned_addr learned my addr 172.21.15.52:0/1225955908 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:20.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.951+0000 7f4a42800640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:46466/0 (socket says 172.21.15.52:46466) 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 -- 172.21.15.52:0/1225955908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 msgr2=0x564e70f42200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 -- 172.21.15.52:0/1225955908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 msgr2=0x564e70f3c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70f3c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 -- 172.21.15.52:0/1225955908 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e70f3a100 con 0x564e70f399b0 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a42800640 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a41e00640 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005ab0 tx=0x7f4a34040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82be25d09852f16c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a2e600640 1 -- 172.21.15.52:0/1225955908 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3405a070 con 0x564e70f399b0 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a2e600640 1 -- 172.21.15.52:0/1225955908 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a34054580 con 0x564e70f399b0 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.952+0000 7f4a2e600640 1 -- 172.21.15.52:0/1225955908 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34035040 con 0x564e70f399b0 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/1225955908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 msgr2=0x564e70f50680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005ab0 tx=0x7f4a34040000 comp rx=0 tx=0).stop 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/1225955908 shutdown_connections 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f44b90 0x564e70f42200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70f3c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/1225955908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f399b0 0x564e70f50680 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/1225955908 >> 172.21.15.52:0/1225955908 conn(0x564e70f35ec0 msgr2=0x564e70f41af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/1225955908 shutdown_connections 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/1225955908 wait complete. 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.953+0000 7f4a4b7d5740 1 -- start start 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 Processor -- start 2024-04-26T02:06:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f399b0 0x564e70fb3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70fb3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441ae220 con 0x564e70f3a2d0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441af690 con 0x564e70f399b0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441ae8f0 con 0x564e70f44b90 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a4b7d5740 10 client.0 _ll_register_callbacks cb 0x7f4a3c09d540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a42800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a41e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f399b0 0x564e70fb3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a41400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70fb3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a42800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:58910/0 (socket says 172.21.15.52:58910) 2024-04-26T02:06:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.954+0000 7f4a42800640 1 -- 172.21.15.52:0/2754727614 learned_addr learned my addr 172.21.15.52:0/2754727614 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 -- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f399b0 msgr2=0x564e70fb3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f399b0 0x564e70fb3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 -- 172.21.15.52:0/2754727614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 msgr2=0x564e70fb3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70fb3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 -- 172.21.15.52:0/2754727614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e70f4a760 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a42800640 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c005820 tx=0x7f4a3c0053e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f54b712c845dc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a23400640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c09be90 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f4a3c0b6060 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a34054580 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f4a3c086080 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a4b7d5740 10 client.27484 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.955+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x564e70fbb060 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a34054580 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f4a3c098c40 con 0x564e70f44b90 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 10 client.27484 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 10 client.27484 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:20.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 20 client.27484 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70fbb6a0 0x564e70fbdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 4 client.27484 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e70ead320 con 0x564e70fbb6a0 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a41e00640 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70fbb6a0 0x564e70fbdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a41e00640 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70fbb6a0 0x564e70fbdac0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=805cae4407f97ade server_cookie=2afeb73645624702 in_seq=0 out_seq=0 2024-04-26T02:06:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:20.956+0000 7f4a23400640 10 client.27484 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.508+0000 7f4a23400640 1 -- 172.21.15.52:0/2754727614 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e70ead320 con 0x564e70fbb6a0 2024-04-26T02:06:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.508+0000 7f4a23400640 10 client.27484 handle_command_reply: tid=0 2024-04-26T02:06:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.508+0000 7f4a4b7d5740 1 client.27484 shutdown 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.508+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 msgr2=0x564e70f49f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.508+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c005820 tx=0x7f4a3c0053e0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 shutdown_connections 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x564e70fbb6a0 0x564e70fbdac0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564e70f44b90 0x564e70f49f90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564e70f3a2d0 0x564e70fb3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 --2- 172.21.15.52:0/2754727614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x564e70f399b0 0x564e70fb3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 >> 172.21.15.52:0/2754727614 conn(0x564e70f35ec0 msgr2=0x564e70f42380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a23400640 0 client.27484 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 shutdown_connections 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 1 -- 172.21.15.52:0/2754727614 wait complete. 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.509+0000 7f4a4b7d5740 20 client.27484 trim_cache size 0 max 16384 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:06:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:06:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 msgr2=0x7f4a280a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 0x7f4a280a61d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564e70f3af50 tx=0x7f4a34005a20 comp rx=0 tx=0).stop 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 msgr2=0x7f4a4419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 0x7f4a4419a5e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009fd0 tx=0x7f4a3c00ef10 comp rx=0 tx=0).stop 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 shutdown_connections 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f4a280a3d10 0x7f4a280a61d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a4410e630 0x7f4a4419a5e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a44074c20 0x7f4a441a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 --2- 172.21.15.52:0/218120422 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a44074300 0x7f4a441a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.512+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 >> 172.21.15.52:0/218120422 conn(0x7f4a440fd8c0 msgr2=0x7f4a440fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.513+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 shutdown_connections 2024-04-26T02:06:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:21.513+0000 7f4a43200640 1 -- 172.21.15.52:0/218120422 wait complete. 2024-04-26T02:06:21.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 170920 20.80 MB/sec execute 338 sec latency 292.118 ms 2024-04-26T02:06:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:21 smithi191 ceph-mon[32745]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-04-26T02:06:21.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:21 smithi183 ceph-mon[32752]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-04-26T02:06:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:21 smithi052 ceph-mon[27851]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-04-26T02:06:22.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 171367 20.82 MB/sec execute 339 sec latency 344.829 ms 2024-04-26T02:06:23.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 171980 20.84 MB/sec execute 340 sec latency 112.369 ms 2024-04-26T02:06:23.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:23 smithi191 ceph-mon[32745]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 466 op/s 2024-04-26T02:06:23.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:23 smithi183 ceph-mon[32752]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 466 op/s 2024-04-26T02:06:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:23 smithi052 ceph-mon[27851]: pgmap v679: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.1 MiB/s wr, 466 op/s 2024-04-26T02:06:24.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 172586 20.89 MB/sec execute 341 sec latency 72.669 ms 2024-04-26T02:06:25.767 INFO:tasks.workunit.client.0.smithi052.stdout: 10 173019 20.91 MB/sec execute 342 sec latency 118.682 ms 2024-04-26T02:06:25.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:25 smithi191 ceph-mon[32745]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 428 op/s 2024-04-26T02:06:25.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:25 smithi183 ceph-mon[32752]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 428 op/s 2024-04-26T02:06:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:25 smithi052 ceph-mon[27851]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 428 op/s 2024-04-26T02:06:26.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 173341 20.89 MB/sec execute 343 sec latency 289.134 ms 2024-04-26T02:06:27.768 INFO:tasks.workunit.client.0.smithi052.stdout: 10 173474 20.84 MB/sec execute 344 sec latency 1071.705 ms 2024-04-26T02:06:27.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:27 smithi191 ceph-mon[32745]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.7 MiB/s wr, 557 op/s 2024-04-26T02:06:27.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:27 smithi183 ceph-mon[32752]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.7 MiB/s wr, 557 op/s 2024-04-26T02:06:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:27 smithi052 ceph-mon[27851]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.7 MiB/s wr, 557 op/s 2024-04-26T02:06:28.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 173861 20.85 MB/sec execute 345 sec latency 129.755 ms 2024-04-26T02:06:29.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 174431 20.88 MB/sec execute 346 sec latency 106.044 ms 2024-04-26T02:06:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:29 smithi191 ceph-mon[32745]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 469 op/s 2024-04-26T02:06:29.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:29 smithi183 ceph-mon[32752]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 469 op/s 2024-04-26T02:06:29.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:29 smithi052 ceph-mon[27851]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 469 op/s 2024-04-26T02:06:30.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 174839 20.88 MB/sec execute 347 sec latency 103.617 ms 2024-04-26T02:06:31.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 175378 20.90 MB/sec execute 348 sec latency 122.082 ms 2024-04-26T02:06:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:31 smithi191 ceph-mon[32745]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 435 op/s 2024-04-26T02:06:31.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:31 smithi183 ceph-mon[32752]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 435 op/s 2024-04-26T02:06:31.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:31 smithi052 ceph-mon[27851]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.2 MiB/s wr, 435 op/s 2024-04-26T02:06:32.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 175616 20.88 MB/sec execute 349 sec latency 615.468 ms 2024-04-26T02:06:33.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 175926 20.87 MB/sec execute 350 sec latency 721.548 ms 2024-04-26T02:06:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:33 smithi191 ceph-mon[32745]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:06:33.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:33 smithi183 ceph-mon[32752]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:06:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:33 smithi052 ceph-mon[27851]: pgmap v684: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 790 op/s 2024-04-26T02:06:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 -- 172.21.15.52:0/4023875942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f926810e620 msgr2=0x7f926810ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 --2- 172.21.15.52:0/4023875942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f926810e620 0x7f926810ea00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f925c009930 tx=0x7f925c02a480 comp rx=0 tx=0).stop 2024-04-26T02:06:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 -- 172.21.15.52:0/4023875942 shutdown_connections 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 --2- 172.21.15.52:0/4023875942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108f20 0x7f92681153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 --2- 172.21.15.52:0/4023875942 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108560 0x7f92681089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 --2- 172.21.15.52:0/4023875942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f926810e620 0x7f926810ea00 secure :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f925c009930 tx=0x7f925c02a480 comp rx=0 tx=0).stop 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.695+0000 7f926f600640 1 -- 172.21.15.52:0/4023875942 >> 172.21.15.52:0/4023875942 conn(0x7f92680764e0 msgr2=0x7f9268078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.696+0000 7f926f600640 1 -- 172.21.15.52:0/4023875942 shutdown_connections 2024-04-26T02:06:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.696+0000 7f926f600640 1 -- 172.21.15.52:0/4023875942 wait complete. 2024-04-26T02:06:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.697+0000 7f926f600640 1 Processor -- start 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.697+0000 7f926f600640 1 -- start start 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108560 0x7f926819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108f20 0x7f926819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9268118920 con 0x7f9268108f20 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92681187a0 con 0x7f92681a38a0 2024-04-26T02:06:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926f600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9268118aa0 con 0x7f9268108560 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926d600640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38842/0 (socket says 172.21.15.52:38842) 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926d600640 1 -- 172.21.15.52:0/3399273907 learned_addr learned my addr 172.21.15.52:0/3399273907 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f9267e00640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108f20 0x7f926819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.698+0000 7f926cc00640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108560 0x7f926819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 -- 172.21.15.52:0/3399273907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108560 msgr2=0x7f926819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108560 0x7f926819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 -- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108f20 msgr2=0x7f926819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108f20 0x7f926819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 -- 172.21.15.52:0/3399273907 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92681a4390 con 0x7f92681a38a0 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.699+0000 7f926d600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f926000b970 tx=0x7f926000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.700+0000 7f9265600640 1 -- 172.21.15.52:0/3399273907 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260019070 con 0x7f92681a38a0 2024-04-26T02:06:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.700+0000 7f9265600640 1 -- 172.21.15.52:0/3399273907 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9260014440 con 0x7f92681a38a0 2024-04-26T02:06:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.700+0000 7f9265600640 1 -- 172.21.15.52:0/3399273907 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92600075f0 con 0x7f92681a38a0 2024-04-26T02:06:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.700+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92681a4680 con 0x7f92681a38a0 2024-04-26T02:06:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.700+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92681abe40 con 0x7f92681a38a0 2024-04-26T02:06:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.704+0000 7f927237b740 1 Processor -- start 2024-04-26T02:06:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.705+0000 7f9265600640 1 -- 172.21.15.52:0/3399273907 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f926001c070 con 0x7f92681a38a0 2024-04-26T02:06:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 -- start start 2024-04-26T02:06:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b431c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b43edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55629b397900 con 0x55629b431820 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55629b33def0 con 0x55629b43ca00 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.706+0000 7f927237b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55629b3938c0 con 0x55629b432140 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34750/0 (socket says 172.21.15.52:34750) 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 -- 172.21.15.52:0/1474103471 learned_addr learned my addr 172.21.15.52:0/1474103471 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f9267e00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b43edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f9267e00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b431c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 -- 172.21.15.52:0/1474103471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 msgr2=0x55629b43edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b43edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 -- 172.21.15.52:0/1474103471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 msgr2=0x55629b431c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b431c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f926cc00640 1 -- 172.21.15.52:0/1474103471 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55629b0aa210 con 0x55629b432140 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.707+0000 7f9267e00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b431c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.708+0000 7f926cc00640 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f925c009900 tx=0x7f925c005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcfebd7cfc9c08f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.708+0000 7f9259c00640 1 -- 172.21.15.52:0/1474103471 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c040070 con 0x55629b432140 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.709+0000 7f9259c00640 1 -- 172.21.15.52:0/1474103471 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f925c0054d0 con 0x55629b432140 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.709+0000 7f9259c00640 1 -- 172.21.15.52:0/1474103471 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c035040 con 0x55629b432140 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.709+0000 7f9265600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 0x7f92500a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.710+0000 7f927237b740 1 -- 172.21.15.52:0/1474103471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 msgr2=0x55629b43c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.710+0000 7f927237b740 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f925c009900 tx=0x7f925c005c00 comp rx=0 tx=0).stop 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f9265600640 1 -- 172.21.15.52:0/3399273907 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f92600cfc30 con 0x7f92681a38a0 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 -- 172.21.15.52:0/1474103471 shutdown_connections 2024-04-26T02:06:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b43edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b43c4c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 --2- 172.21.15.52:0/1474103471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b431c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 -- 172.21.15.52:0/1474103471 >> 172.21.15.52:0/1474103471 conn(0x55629b42dd30 msgr2=0x55629b42e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.711+0000 7f927237b740 1 -- 172.21.15.52:0/1474103471 shutdown_connections 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.712+0000 7f9267e00640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 0x7f92500a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.712+0000 7f927237b740 1 -- 172.21.15.52:0/1474103471 wait complete. 2024-04-26T02:06:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.714+0000 7f927237b740 1 -- start start 2024-04-26T02:06:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.715+0000 7f927237b740 1 Processor -- start 2024-04-26T02:06:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.715+0000 7f9267e00640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 0x7f92500a6350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55629b42f150 tx=0x7f9254004700 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.723+0000 7f927237b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 0x55629b439960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.723+0000 7f927237b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.723+0000 7f927237b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b43ca00 0x55629b43a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f927237b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55629b43f5d0 con 0x55629b432140 2024-04-26T02:06:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f9267e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f926cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 0x55629b439960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f9267e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39374/0 (socket says 172.21.15.52:39374) 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f9267e00640 1 -- 172.21.15.52:0/1379889629 learned_addr learned my addr 172.21.15.52:0/1379889629 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f926cc00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 0x55629b439960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38852/0 (socket says 172.21.15.52:38852) 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55629b351310 con 0x55629b431820 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55629b43f750 con 0x55629b43ca00 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f927237b740 10 client.0 _ll_register_callbacks cb 0x55629b31fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.724+0000 7f926d600640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b43ca00 0x55629b43a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.725+0000 7f9267e00640 1 -- 172.21.15.52:0/1379889629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b43ca00 msgr2=0x55629b43a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.725+0000 7f9267e00640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b43ca00 0x55629b43a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.725+0000 7f9267e00640 1 -- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 msgr2=0x55629b439960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.725+0000 7f9267e00640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 0x55629b439960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.725+0000 7f9267e00640 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55629b176620 con 0x55629b432140 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.726+0000 7f9267e00640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f92540136e0 tx=0x7f9254004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8427ff1e2b1e6bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.726+0000 7f924e000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:06:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.726+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254024070 con 0x55629b432140 2024-04-26T02:06:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.726+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f9254012aa0 con 0x55629b432140 2024-04-26T02:06:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.727+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f925c0054d0 con 0x55629b432140 2024-04-26T02:06:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.727+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f925402e050 con 0x55629b432140 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.727+0000 7f927237b740 10 client.17277 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.727+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55629b44b800 con 0x55629b432140 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f925c0054d0 con 0x55629b432140 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f925401f070 con 0x55629b432140 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 10 client.17277 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 10 client.17277 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 20 client.17277 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b44f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 4 client.17277 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.728+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55629b3ab8b0 con 0x55629b44cd10 2024-04-26T02:06:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.729+0000 7f926cc00640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b44f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.730+0000 7f926cc00640 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b44f160 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2ec4e3fa45ce82b server_cookie=4dcffd3eecb7c7a0 in_seq=0 out_seq=0 2024-04-26T02:06:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.730+0000 7f924e000640 10 client.17277 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.735+0000 7f924e000640 1 -- 172.21.15.52:0/1379889629 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55629b3ab8b0 con 0x55629b44cd10 2024-04-26T02:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.735+0000 7f924e000640 10 client.17277 handle_command_reply: tid=0 2024-04-26T02:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.735+0000 7f927237b740 1 client.17277 shutdown 2024-04-26T02:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.735+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 msgr2=0x55629b439ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.735+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f92540136e0 tx=0x7f9254004770 comp rx=0 tx=0).stop 2024-04-26T02:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 shutdown_connections 2024-04-26T02:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b44f160 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b43ca00 0x55629b43a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f924e000640 0 client.17277 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b439ea0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 --2- 172.21.15.52:0/1379889629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b431820 0x55629b439960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 >> 172.21.15.52:0/1379889629 conn(0x55629b42dd30 msgr2=0x55629b433a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 shutdown_connections 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 1 -- 172.21.15.52:0/1379889629 wait complete. 2024-04-26T02:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.736+0000 7f927237b740 20 client.17277 trim_cache size 0 max 16384 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 Processor -- start 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 -- start start 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b42d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b448490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b7040 con 0x55629b431820 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b4910 con 0x55629b43ca00 2024-04-26T02:06:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f927237b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b2780 con 0x55629b432140 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f926cc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.763+0000 7f9267e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b42d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926cc00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39388/0 (socket says 172.21.15.52:39388) 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926cc00640 1 -- 172.21.15.52:0/291317657 learned_addr learned my addr 172.21.15.52:0/291317657 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b448490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 -- 172.21.15.52:0/291317657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 msgr2=0x55629b42d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b42d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 -- 172.21.15.52:0/291317657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 msgr2=0x55629b448f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926d600640 1 -- 172.21.15.52:0/291317657 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55629b438930 con 0x55629b43ca00 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.764+0000 7f926cc00640 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.765+0000 7f926d600640 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b448490 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92600cd430 tx=0x7f926009b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7756f378fdbb0ea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.765+0000 7f9259200640 1 -- 172.21.15.52:0/291317657 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92600a0030 con 0x55629b43ca00 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.765+0000 7f9259200640 1 -- 172.21.15.52:0/291317657 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92600a48f0 con 0x55629b43ca00 2024-04-26T02:06:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.765+0000 7f9259200640 1 -- 172.21.15.52:0/291317657 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92600bc0c0 con 0x55629b43ca00 2024-04-26T02:06:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 -- 172.21.15.52:0/291317657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 msgr2=0x55629b448490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b448490 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92600cd430 tx=0x7f926009b870 comp rx=0 tx=0).stop 2024-04-26T02:06:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 -- 172.21.15.52:0/291317657 shutdown_connections 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b448490 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b432140 0x55629b42d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 --2- 172.21.15.52:0/291317657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b431820 0x55629b448f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.766+0000 7f927237b740 1 -- 172.21.15.52:0/291317657 >> 172.21.15.52:0/291317657 conn(0x55629b42dd30 msgr2=0x55629b42e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.767+0000 7f927237b740 1 -- 172.21.15.52:0/291317657 shutdown_connections 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.767+0000 7f927237b740 1 -- 172.21.15.52:0/291317657 wait complete. 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.767+0000 7f927237b740 1 -- start start 2024-04-26T02:06:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 Processor -- start 2024-04-26T02:06:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b4428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b442e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b2dd0 con 0x55629b432140 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b4240 con 0x55629b43ca00 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f92681b34a0 con 0x55629b431820 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f927237b740 10 client.0 _ll_register_callbacks cb 0x7f9254010f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f926cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f926cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:34778/0 (socket says 172.21.15.52:34778) 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.768+0000 7f926cc00640 1 -- 172.21.15.52:0/693598253 learned_addr learned my addr 172.21.15.52:0/693598253 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926d600640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b442e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f9267e00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b4428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 -- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 msgr2=0x55629b442e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b442e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 -- 172.21.15.52:0/693598253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 msgr2=0x55629b4428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b4428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 -- 172.21.15.52:0/693598253 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55629b4b22f0 con 0x55629b431820 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f9267e00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b4428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f926cc00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f925c0057c0 tx=0x7f925c009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ba71ee66356523 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.769+0000 7f924e000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c035040 con 0x55629b431820 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f925c004f30 con 0x55629b431820 2024-04-26T02:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92600a48f0 con 0x55629b431820 2024-04-26T02:06:34.770 INFO:tasks.workunit.client.0.smithi052.stdout: 10 176393 20.87 MB/sec execute 351 sec latency 242.619 ms 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f925c040040 con 0x55629b431820 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f927237b740 10 client.27496 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55629b4b3540 con 0x55629b431820 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.770+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92600a48f0 con 0x55629b431820 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f925c0035f0 con 0x55629b431820 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 10 client.27496 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 10 client.27496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 20 client.27496 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b4b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 4 client.27496 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55629b3a5320 con 0x55629b44cd10 2024-04-26T02:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.771+0000 7f9267e00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b4b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.772+0000 7f9267e00640 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b4b58e0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf274b52fcf5ec8 server_cookie=d7b7fe730267d401 in_seq=0 out_seq=0 2024-04-26T02:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.772+0000 7f924e000640 10 client.27496 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.955+0000 7f924e000640 1 -- 172.21.15.52:0/693598253 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55629b3a5320 con 0x55629b44cd10 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.955+0000 7f924e000640 10 client.27496 handle_command_reply: tid=0 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.955+0000 7f927237b740 1 client.27496 shutdown 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.955+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 msgr2=0x55629b442380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.955+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f925c0057c0 tx=0x7f925c009830 comp rx=0 tx=0).stop 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 shutdown_connections 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55629b44cd10 0x55629b4b58e0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55629b43ca00 0x55629b442e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55629b432140 0x55629b4428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 --2- 172.21.15.52:0/693598253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55629b431820 0x55629b442380 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f924e000640 0 client.27496 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 >> 172.21.15.52:0/693598253 conn(0x55629b42dd30 msgr2=0x55629b433a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 shutdown_connections 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 1 -- 172.21.15.52:0/693598253 wait complete. 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.956+0000 7f927237b740 20 client.27496 trim_cache size 0 max 16384 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:06:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 msgr2=0x7f92500a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 0x7f92500a6350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55629b42f150 tx=0x7f9254004700 comp rx=0 tx=0).stop 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 msgr2=0x7f92681a3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f926000b970 tx=0x7f926000be30 comp rx=0 tx=0).stop 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 shutdown_connections 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f92500a3e90 0x7f92500a6350 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92681a38a0 0x7f92681a3d50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9268108f20 0x7f926819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 --2- 172.21.15.52:0/3399273907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9268108560 0x7f926819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 >> 172.21.15.52:0/3399273907 conn(0x7f92680764e0 msgr2=0x7f9268077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 shutdown_connections 2024-04-26T02:06:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:34.958+0000 7f926f600640 1 -- 172.21.15.52:0/3399273907 wait complete. 2024-04-26T02:06:35.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 176791 20.88 MB/sec execute 352 sec latency 193.035 ms 2024-04-26T02:06:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:35 smithi191 ceph-mon[32745]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-04-26T02:06:35.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:35 smithi183 ceph-mon[32752]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-04-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:35 smithi052 ceph-mon[27851]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 585 op/s 2024-04-26T02:06:36.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 177330 20.92 MB/sec execute 353 sec latency 101.669 ms 2024-04-26T02:06:37.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 177848 20.92 MB/sec execute 354 sec latency 104.714 ms 2024-04-26T02:06:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:37 smithi191 ceph-mon[32745]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-04-26T02:06:37.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:37 smithi183 ceph-mon[32752]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-04-26T02:06:37.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:37 smithi052 ceph-mon[27851]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 779 op/s 2024-04-26T02:06:38.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 178347 20.95 MB/sec execute 355 sec latency 157.199 ms 2024-04-26T02:06:39.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 178845 20.96 MB/sec execute 356 sec latency 70.260 ms 2024-04-26T02:06:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:39 smithi191 ceph-mon[32745]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 730 op/s 2024-04-26T02:06:39.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:39 smithi183 ceph-mon[32752]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 730 op/s 2024-04-26T02:06:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:39 smithi052 ceph-mon[27851]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 730 op/s 2024-04-26T02:06:40.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 179407 20.98 MB/sec execute 357 sec latency 97.855 ms 2024-04-26T02:06:41.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 179892 21.01 MB/sec execute 358 sec latency 150.344 ms 2024-04-26T02:06:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:41 smithi191 ceph-mon[32745]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 15 MiB/s wr, 669 op/s 2024-04-26T02:06:41.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:41 smithi183 ceph-mon[32752]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 15 MiB/s wr, 669 op/s 2024-04-26T02:06:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:41 smithi052 ceph-mon[27851]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 15 MiB/s wr, 669 op/s 2024-04-26T02:06:42.769 INFO:tasks.workunit.client.0.smithi052.stdout: 10 180407 21.00 MB/sec execute 359 sec latency 79.794 ms 2024-04-26T02:06:43.770 INFO:tasks.workunit.client.0.smithi052.stdout: 10 181008 21.04 MB/sec execute 360 sec latency 70.336 ms 2024-04-26T02:06:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:43 smithi191 ceph-mon[32745]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 851 op/s 2024-04-26T02:06:43.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:43 smithi183 ceph-mon[32752]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 851 op/s 2024-04-26T02:06:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:43 smithi052 ceph-mon[27851]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 851 op/s 2024-04-26T02:06:44.770 INFO:tasks.workunit.client.0.smithi052.stdout: 10 181639 21.09 MB/sec execute 361 sec latency 111.683 ms 2024-04-26T02:06:45.770 INFO:tasks.workunit.client.0.smithi052.stdout: 10 182143 21.12 MB/sec execute 362 sec latency 105.953 ms 2024-04-26T02:06:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:45 smithi191 ceph-mon[32745]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 496 op/s 2024-04-26T02:06:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:45 smithi183 ceph-mon[32752]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 496 op/s 2024-04-26T02:06:45.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:45 smithi052 ceph-mon[27851]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 10 MiB/s wr, 496 op/s 2024-04-26T02:06:46.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 182589 21.12 MB/sec execute 363 sec latency 125.104 ms 2024-04-26T02:06:47.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 182959 21.10 MB/sec execute 364 sec latency 195.726 ms 2024-04-26T02:06:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:47 smithi191 ceph-mon[32745]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-04-26T02:06:47.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:47 smithi183 ceph-mon[32752]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-04-26T02:06:47.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:47 smithi052 ceph-mon[27851]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-04-26T02:06:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.190+0000 7f73b6200640 1 -- 172.21.15.52:0/1379960524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0076da0 msgr2=0x7f73b01128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.190+0000 7f73b6200640 1 --2- 172.21.15.52:0/1379960524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0076da0 0x7f73b01128a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7398009890 tx=0x7f739802a360 comp rx=0 tx=0).stop 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 -- 172.21.15.52:0/1379960524 shutdown_connections 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 --2- 172.21.15.52:0/1379960524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0112de0 0x7f73b01151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 --2- 172.21.15.52:0/1379960524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0076da0 0x7f73b01128a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 --2- 172.21.15.52:0/1379960524 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 0x7f73b0076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 -- 172.21.15.52:0/1379960524 >> 172.21.15.52:0/1379960524 conn(0x7f73b00ffc50 msgr2=0x7f73b0102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 -- 172.21.15.52:0/1379960524 shutdown_connections 2024-04-26T02:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.191+0000 7f73b6200640 1 -- 172.21.15.52:0/1379960524 wait complete. 2024-04-26T02:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.193+0000 7f73b6200640 1 Processor -- start 2024-04-26T02:06:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.193+0000 7f73b6200640 1 -- start start 2024-04-26T02:06:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 0x7f73b01a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 0x7f73b01a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 0x7f73b019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f73b01187f0 con 0x7f73b0076da0 2024-04-26T02:06:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f73b0118670 con 0x7f73b0112de0 2024-04-26T02:06:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73b6200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f73b0118970 con 0x7f73b0076480 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.194+0000 7f73a7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 0x7f73b01a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.195+0000 7f73af400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 0x7f73b01a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.195+0000 7f73a7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 0x7f73b01a4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45376/0 (socket says 172.21.15.52:45376) 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.195+0000 7f73a7e00640 1 -- 172.21.15.52:0/4210041786 learned_addr learned my addr 172.21.15.52:0/4210041786 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.195+0000 7f73afe00640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 0x7f73b019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.196+0000 7f73afe00640 1 -- 172.21.15.52:0/4210041786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 msgr2=0x7f73b01a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.196+0000 7f73afe00640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 0x7f73b01a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.196+0000 7f73afe00640 1 -- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 msgr2=0x7f73b01a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.196+0000 7f73afe00640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 0x7f73b01a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.196+0000 7f73afe00640 1 -- 172.21.15.52:0/4210041786 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b019f1b0 con 0x7f73b0112de0 2024-04-26T02:06:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73afe00640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 0x7f73b019ea30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005fd0 tx=0x7f73a800edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73acc00640 1 -- 172.21.15.52:0/4210041786 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8004d20 con 0x7f73b0112de0 2024-04-26T02:06:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b019f4a0 con 0x7f73b0112de0 2024-04-26T02:06:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b01ab650 con 0x7f73b0112de0 2024-04-26T02:06:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73acc00640 1 -- 172.21.15.52:0/4210041786 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a8005020 con 0x7f73b0112de0 2024-04-26T02:06:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.197+0000 7f73acc00640 1 -- 172.21.15.52:0/4210041786 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a80116d0 con 0x7f73b0112de0 2024-04-26T02:06:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.206+0000 7f73acc00640 1 -- 172.21.15.52:0/4210041786 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f73a8011870 con 0x7f73b0112de0 2024-04-26T02:06:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.209+0000 7f73acc00640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 0x7f73940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.209+0000 7f73b8bfd740 1 Processor -- start 2024-04-26T02:06:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.209+0000 7f73b8bfd740 1 -- start start 2024-04-26T02:06:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a339615c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a3396204c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a33957b900 con 0x55a339615820 2024-04-26T02:06:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a339521ef0 con 0x55a339620a00 2024-04-26T02:06:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.210+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a3395778c0 con 0x55a339616140 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.211+0000 7f73a7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a3396204c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.211+0000 7f73afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.211+0000 7f73afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54326/0 (socket says 172.21.15.52:54326) 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.211+0000 7f73afe00640 1 -- 172.21.15.52:0/2261932361 learned_addr learned my addr 172.21.15.52:0/2261932361 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.211+0000 7f73acc00640 1 -- 172.21.15.52:0/4210041786 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f73a80d0710 con 0x7f73b0112de0 2024-04-26T02:06:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.212+0000 7f73af400640 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a339615c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73afe00640 1 -- 172.21.15.52:0/2261932361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 msgr2=0x55a3396204c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73afe00640 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a3396204c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73afe00640 1 -- 172.21.15.52:0/2261932361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 msgr2=0x55a339615c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73af400640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 0x7f73940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73afe00640 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a339615c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.213+0000 7f73afe00640 1 -- 172.21.15.52:0/2261932361 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a33928e210 con 0x55a339620a00 2024-04-26T02:06:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.214+0000 7f73af400640 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a339615c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:06:48.213 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.214+0000 7f73afe00640 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f73a80a5e50 tx=0x7f73a809de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a2193321c2b13b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.215+0000 7f739fe00640 1 -- 172.21.15.52:0/2261932361 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a801f040 con 0x55a339620a00 2024-04-26T02:06:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.215+0000 7f739fe00640 1 -- 172.21.15.52:0/2261932361 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a80a1ac0 con 0x55a339620a00 2024-04-26T02:06:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.215+0000 7f739fe00640 1 -- 172.21.15.52:0/2261932361 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8011b60 con 0x55a339620a00 2024-04-26T02:06:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.216+0000 7f73af400640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 0x7f73940a6300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73a0005ab0 tx=0x7f73a0009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.216+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2261932361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 msgr2=0x55a339622df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.216+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f73a80a5e50 tx=0x7f73a809de10 comp rx=0 tx=0).stop 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2261932361 shutdown_connections 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a339622df0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a3396204c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2261932361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a339615c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2261932361 >> 172.21.15.52:0/2261932361 conn(0x55a339611d30 msgr2=0x55a339612110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2261932361 shutdown_connections 2024-04-26T02:06:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.217+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2261932361 wait complete. 2024-04-26T02:06:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.218+0000 7f73b8bfd740 1 -- start start 2024-04-26T02:06:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.218+0000 7f73b8bfd740 1 Processor -- start 2024-04-26T02:06:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.220+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 0x55a33961d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.220+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339616140 0x55a33961de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.220+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73af400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 0x55a33961d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73afe00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:54340/0 (socket says 172.21.15.52:54340) 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73afe00640 1 -- 172.21.15.52:0/2325074918 learned_addr learned my addr 172.21.15.52:0/2325074918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73a7e00640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339616140 0x55a33961de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73af400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 0x55a33961d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55134/0 (socket says 172.21.15.52:55134) 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a3396235d0 con 0x55a339616140 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55a339535310 con 0x55a339620a00 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a339623750 con 0x55a339615820 2024-04-26T02:06:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.221+0000 7f73b8bfd740 10 client.0 _ll_register_callbacks cb 0x55a339503cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.222+0000 7f73afe00640 1 -- 172.21.15.52:0/2325074918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 msgr2=0x55a33961d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.222+0000 7f73afe00640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 0x55a33961d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.222+0000 7f73afe00640 1 -- 172.21.15.52:0/2325074918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339616140 msgr2=0x55a33961de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.222+0000 7f73afe00640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339616140 0x55a33961de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.222+0000 7f73afe00640 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a33935a620 con 0x55a339620a00 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.223+0000 7f73afe00640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f73a809a0e0 tx=0x7f73a80a64c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318d748d35888de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.223+0000 7f739cc00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.223+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a801f030 con 0x55a339620a00 2024-04-26T02:06:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.223+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f73a8011d60 con 0x55a339620a00 2024-04-26T02:06:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.223+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73a80a1ac0 con 0x55a339620a00 2024-04-26T02:06:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.226+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f73a8082050 con 0x55a339620a00 2024-04-26T02:06:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.226+0000 7f73b8bfd740 10 client.27488 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.226+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55a33962f7f0 con 0x55a339620a00 2024-04-26T02:06:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.227+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a80a1ac0 con 0x55a339620a00 2024-04-26T02:06:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.227+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f73a800f040 con 0x55a339620a00 2024-04-26T02:06:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.228+0000 7f73b8bfd740 10 client.27488 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.228+0000 7f73b8bfd740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.229+0000 7f73b8bfd740 20 client.27488 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.229+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a339630b40 0x55a339632f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.229+0000 7f73b8bfd740 4 client.27488 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:06:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.229+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a33958f8b0 con 0x55a339630b40 2024-04-26T02:06:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.229+0000 7f73af400640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a339630b40 0x55a339632f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.233+0000 7f73af400640 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a339630b40 0x55a339632f30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b286a66c82236d server_cookie=44b6deeb70f03fa8 in_seq=0 out_seq=0 2024-04-26T02:06:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.233+0000 7f739cc00640 10 client.27488 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.235+0000 7f739cc00640 1 -- 172.21.15.52:0/2325074918 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a33958f8b0 con 0x55a339630b40 2024-04-26T02:06:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.236+0000 7f739cc00640 10 client.27488 handle_command_reply: tid=0 2024-04-26T02:06:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.236+0000 7f73b8bfd740 1 client.27488 shutdown 2024-04-26T02:06:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 msgr2=0x55a33961e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f73a809a0e0 tx=0x7f73a80a64c0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 shutdown_connections 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a339630b40 0x55a339632f30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339620a00 0x55a33961e390 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339616140 0x55a33961de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f739cc00640 0 client.27488 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/2325074918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339615820 0x55a33961d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.237+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 >> 172.21.15.52:0/2325074918 conn(0x55a339611d30 msgr2=0x55a339617ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.240+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 shutdown_connections 2024-04-26T02:06:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.241+0000 7f73b8bfd740 1 -- 172.21.15.52:0/2325074918 wait complete. 2024-04-26T02:06:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.242+0000 7f73b8bfd740 20 client.27488 trim_cache size 0 max 16384 2024-04-26T02:06:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.294+0000 7f73b8bfd740 1 Processor -- start 2024-04-26T02:06:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 -- start start 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 0x55a33961e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a33962c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 0x55a339635f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f738c011010 con 0x55a339620a00 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f738c00e8e0 con 0x55a339615820 2024-04-26T02:06:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.295+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f738c00c750 con 0x55a339616140 2024-04-26T02:06:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.296+0000 7f73afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 0x55a339635f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.296+0000 7f73afe00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 0x55a339635f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45412/0 (socket says 172.21.15.52:45412) 2024-04-26T02:06:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.296+0000 7f73af400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 0x55a33961e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.296+0000 7f73afe00640 1 -- 172.21.15.52:0/20075824 learned_addr learned my addr 172.21.15.52:0/20075824 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.296+0000 7f73a7e00640 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a33962c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 -- 172.21.15.52:0/20075824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 msgr2=0x55a33962c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a33962c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 -- 172.21.15.52:0/20075824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 msgr2=0x55a339635f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 0x55a339635f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 -- 172.21.15.52:0/20075824 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a33961be60 con 0x55a339615820 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73a7e00640 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a33962c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.298+0000 7f73af400640 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 0x55a33961e1b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55a3396237f0 tx=0x7f73a00096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d93b38381910ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.300+0000 7f739f400640 1 -- 172.21.15.52:0/20075824 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0030070 con 0x55a339615820 2024-04-26T02:06:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.301+0000 7f739f400640 1 -- 172.21.15.52:0/20075824 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a001a070 con 0x55a339615820 2024-04-26T02:06:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.301+0000 7f739f400640 1 -- 172.21.15.52:0/20075824 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a000f710 con 0x55a339615820 2024-04-26T02:06:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.302+0000 7f73b8bfd740 1 -- 172.21.15.52:0/20075824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 msgr2=0x55a33961e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.302+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 0x55a33961e1b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55a3396237f0 tx=0x7f73a00096a0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.303+0000 7f73b8bfd740 1 -- 172.21.15.52:0/20075824 shutdown_connections 2024-04-26T02:06:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.303+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339620a00 0x55a339635f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.303+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339616140 0x55a33962c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.303+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/20075824 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339615820 0x55a33961e1b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.303+0000 7f73b8bfd740 1 -- 172.21.15.52:0/20075824 >> 172.21.15.52:0/20075824 conn(0x55a339611d30 msgr2=0x55a33962b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.304+0000 7f73b8bfd740 1 -- 172.21.15.52:0/20075824 shutdown_connections 2024-04-26T02:06:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.304+0000 7f73b8bfd740 1 -- 172.21.15.52:0/20075824 wait complete. 2024-04-26T02:06:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.305+0000 7f73b8bfd740 1 -- start start 2024-04-26T02:06:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.305+0000 7f73b8bfd740 1 Processor -- start 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a33968fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 0x55a339690240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f738c00cda0 con 0x55a339615820 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f738c00e210 con 0x55a339616140 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f738c00d470 con 0x55a339620a00 2024-04-26T02:06:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.306+0000 7f73b8bfd740 10 client.0 _ll_register_callbacks cb 0x7f73a80a6380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:06:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.307+0000 7f73afe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.307+0000 7f73afe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:55150/0 (socket says 172.21.15.52:55150) 2024-04-26T02:06:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.307+0000 7f73afe00640 1 -- 172.21.15.52:0/4044519756 learned_addr learned my addr 172.21.15.52:0/4044519756 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:06:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.307+0000 7f73af400640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a33968fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.308+0000 7f73a7e00640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 0x55a339690240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73afe00640 1 -- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 msgr2=0x55a339690240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73afe00640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 0x55a339690240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73afe00640 1 -- 172.21.15.52:0/4044519756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 msgr2=0x55a33968fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:06:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73afe00640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a33968fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73afe00640 1 -- 172.21.15.52:0/4044519756 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a33968ae30 con 0x55a339620a00 2024-04-26T02:06:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.309+0000 7f73a7e00640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 0x55a339690240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f73af400640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a33968fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:06:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f73afe00640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005980 tx=0x7f73a8099d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1850e621ed26466c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f739cc00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a801f040 con 0x55a339620a00 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f73a8099460 con 0x55a339620a00 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73a001a070 con 0x55a339620a00 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f73a80a1ac0 con 0x55a339620a00 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.310+0000 7f73b8bfd740 10 client.27514 fetch_fsmap learned FSMap version 73 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.311+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55a3396a81a0 con 0x55a339620a00 2024-04-26T02:06:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.311+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a001a070 con 0x55a339620a00 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f73a80a2c50 con 0x55a339620a00 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 10 client.27514 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 10 client.27514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 20 client.27514 populate_metadata read hostname 'smithi052' 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a3396a8620 0x55a3396aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 4 client.27514 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:06:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.312+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a339589320 con 0x55a3396a8620 2024-04-26T02:06:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.313+0000 7f73af400640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a3396a8620 0x55a3396aaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:06:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.314+0000 7f73af400640 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a3396a8620 0x55a3396aaa40 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9675f617db3420d server_cookie=bc7ccabb57d35b3b in_seq=0 out_seq=0 2024-04-26T02:06:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.314+0000 7f739cc00640 10 client.27514 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.651+0000 7f739cc00640 1 -- 172.21.15.52:0/4044519756 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a339589320 con 0x55a3396a8620 2024-04-26T02:06:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.651+0000 7f739cc00640 10 client.27514 handle_command_reply: tid=0 2024-04-26T02:06:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.651+0000 7f73b8bfd740 1 client.27514 shutdown 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 msgr2=0x55a339690780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005980 tx=0x7f73a8099d80 comp rx=0 tx=0).stop 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 shutdown_connections 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x55a3396a8620 0x55a3396aaa40 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a339620a00 0x55a339690780 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55a339616140 0x55a339690240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 --2- 172.21.15.52:0/4044519756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a339615820 0x55a33968fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 >> 172.21.15.52:0/4044519756 conn(0x55a339611d30 msgr2=0x55a33962c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.652+0000 7f739cc00640 0 client.27514 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:06:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.653+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 shutdown_connections 2024-04-26T02:06:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.653+0000 7f73b8bfd740 1 -- 172.21.15.52:0/4044519756 wait complete. 2024-04-26T02:06:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.653+0000 7f73b8bfd740 20 client.27514 trim_cache size 0 max 16384 2024-04-26T02:06:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:06:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:06:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:06:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:06:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.654+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 msgr2=0x7f73940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.654+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 0x7f73940a6300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73a0005ab0 tx=0x7f73a0009260 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.654+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 msgr2=0x7f73b019ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.654+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 0x7f73b019ea30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f73a8005fd0 tx=0x7f73a800edd0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 shutdown_connections 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f73940a3e40 0x7f73940a6300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f73b0112de0 0x7f73b019ea30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b0076da0 0x7f73b01a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 --2- 172.21.15.52:0/4210041786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73b0076480 0x7f73b01a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:06:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 >> 172.21.15.52:0/4210041786 conn(0x7f73b00ffc50 msgr2=0x7f73b0113260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:06:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 shutdown_connections 2024-04-26T02:06:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:06:48.655+0000 7f73b6200640 1 -- 172.21.15.52:0/4210041786 wait complete. 2024-04-26T02:06:48.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 183175 21.08 MB/sec execute 365 sec latency 256.838 ms 2024-04-26T02:06:49.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 183687 21.08 MB/sec execute 366 sec latency 160.316 ms 2024-04-26T02:06:49.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:49 smithi191 ceph-mon[32745]: pgmap v692: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 472 op/s 2024-04-26T02:06:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:49 smithi183 ceph-mon[32752]: pgmap v692: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 472 op/s 2024-04-26T02:06:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:49 smithi052 ceph-mon[27851]: pgmap v692: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 472 op/s 2024-04-26T02:06:50.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 184335 21.12 MB/sec execute 367 sec latency 76.108 ms 2024-04-26T02:06:51.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 184946 21.17 MB/sec execute 368 sec latency 81.512 ms 2024-04-26T02:06:51.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:51 smithi191 ceph-mon[32745]: pgmap v693: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 351 op/s 2024-04-26T02:06:51.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:51 smithi183 ceph-mon[32752]: pgmap v693: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 351 op/s 2024-04-26T02:06:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:51 smithi052 ceph-mon[27851]: pgmap v693: 97 pgs: 97 active+clean; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.3 MiB/s wr, 351 op/s 2024-04-26T02:06:52.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 185443 21.18 MB/sec execute 369 sec latency 188.711 ms 2024-04-26T02:06:53.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 185621 21.14 MB/sec execute 370 sec latency 544.974 ms 2024-04-26T02:06:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:53 smithi191 ceph-mon[32745]: pgmap v694: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 670 op/s 2024-04-26T02:06:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:53 smithi183 ceph-mon[32752]: pgmap v694: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 670 op/s 2024-04-26T02:06:53.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:53 smithi052 ceph-mon[27851]: pgmap v694: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 670 op/s 2024-04-26T02:06:54.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 186015 21.15 MB/sec execute 371 sec latency 701.733 ms 2024-04-26T02:06:55.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 186514 21.16 MB/sec execute 372 sec latency 148.623 ms 2024-04-26T02:06:55.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:55 smithi191 ceph-mon[32745]: pgmap v695: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 488 op/s 2024-04-26T02:06:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:55 smithi183 ceph-mon[32752]: pgmap v695: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 488 op/s 2024-04-26T02:06:55.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:55 smithi052 ceph-mon[27851]: pgmap v695: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.1 MiB/s wr, 488 op/s 2024-04-26T02:06:56.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 187015 21.18 MB/sec execute 373 sec latency 114.055 ms 2024-04-26T02:06:57.772 INFO:tasks.workunit.client.0.smithi052.stdout: 10 187512 21.20 MB/sec execute 374 sec latency 119.364 ms 2024-04-26T02:06:57.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:57 smithi191 ceph-mon[32745]: pgmap v696: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-04-26T02:06:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:57 smithi183 ceph-mon[32752]: pgmap v696: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-04-26T02:06:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:57 smithi052 ceph-mon[27851]: pgmap v696: 97 pgs: 97 active+clean; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-04-26T02:06:58.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 187864 21.19 MB/sec execute 375 sec latency 385.589 ms 2024-04-26T02:06:58.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:58 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:58.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:58 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:58.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:58 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:06:59.771 INFO:tasks.workunit.client.0.smithi052.stdout: 10 188274 21.19 MB/sec execute 376 sec latency 125.762 ms 2024-04-26T02:06:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:06:59 smithi191 ceph-mon[32745]: pgmap v697: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 667 op/s 2024-04-26T02:06:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:06:59 smithi183 ceph-mon[32752]: pgmap v697: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 667 op/s 2024-04-26T02:06:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:06:59 smithi052 ceph-mon[27851]: pgmap v697: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 667 op/s 2024-04-26T02:07:00.772 INFO:tasks.workunit.client.0.smithi052.stdout: 10 188480 21.15 MB/sec execute 377 sec latency 717.216 ms 2024-04-26T02:07:00.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:00.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:00 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:00.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:00 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:07:00.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:00.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:00 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:00.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:00 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:07:00.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:07:00.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:00 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:07:00.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:00 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:07:01.772 INFO:tasks.workunit.client.0.smithi052.stdout: 10 188480 21.09 MB/sec execute 378 sec latency 1717.351 ms 2024-04-26T02:07:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:01 smithi191 ceph-mon[32745]: pgmap v698: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-04-26T02:07:01.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:01 smithi183 ceph-mon[32752]: pgmap v698: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-04-26T02:07:01.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:01 smithi052 ceph-mon[27851]: pgmap v698: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 -- 172.21.15.52:0/2464011352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c104d50 msgr2=0x7f808c108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 --2- 172.21.15.52:0/2464011352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c104d50 0x7f808c108d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8078009860 tx=0x7f807802a340 comp rx=0 tx=0).stop 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 -- 172.21.15.52:0/2464011352 shutdown_connections 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 --2- 172.21.15.52:0/2464011352 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c109470 0x7f808c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 --2- 172.21.15.52:0/2464011352 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c104d50 0x7f808c108d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 --2- 172.21.15.52:0/2464011352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c1043a0 0x7f808c104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 -- 172.21.15.52:0/2464011352 >> 172.21.15.52:0/2464011352 conn(0x7f808c0ffc40 msgr2=0x7f808c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 -- 172.21.15.52:0/2464011352 shutdown_connections 2024-04-26T02:07:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.004+0000 7f8093400640 1 -- 172.21.15.52:0/2464011352 wait complete. 2024-04-26T02:07:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.005+0000 7f8093400640 1 Processor -- start 2024-04-26T02:07:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.005+0000 7f8093400640 1 -- start start 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c1043a0 0x7f808c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c19a6f0 0x7f808c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1143b0 con 0x7f808c19a6f0 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f808c114230 con 0x7f808c1043a0 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8093400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f808c114530 con 0x7f808c109470 2024-04-26T02:07:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8091400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c19a6f0 0x7f808c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f808be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f8090a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c1043a0 0x7f808c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f808be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:41722/0 (socket says 172.21.15.52:41722) 2024-04-26T02:07:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.006+0000 7f808be00640 1 -- 172.21.15.52:0/544093900 learned_addr learned my addr 172.21.15.52:0/544093900 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 -- 172.21.15.52:0/544093900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c1043a0 msgr2=0x7f808c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c1043a0 0x7f808c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 -- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c19a6f0 msgr2=0x7f808c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c19a6f0 0x7f808c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 -- 172.21.15.52:0/544093900 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c19b2f0 con 0x7f808c109470 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f808be00640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8078035040 tx=0x7f8078032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f8089600640 1 -- 172.21.15.52:0/544093900 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078037040 con 0x7f808c109470 2024-04-26T02:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.007+0000 7f8089600640 1 -- 172.21.15.52:0/544093900 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8078041070 con 0x7f808c109470 2024-04-26T02:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.008+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808c19b580 con 0x7f808c109470 2024-04-26T02:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.008+0000 7f8089600640 1 -- 172.21.15.52:0/544093900 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078003850 con 0x7f808c109470 2024-04-26T02:07:02.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.008+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808c1a76b0 con 0x7f808c109470 2024-04-26T02:07:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.012+0000 7f8096381740 1 Processor -- start 2024-04-26T02:07:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 -- start start 2024-04-26T02:07:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec4b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8089600640 1 -- 172.21.15.52:0/544093900 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f80780039f0 con 0x7f808c109470 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec4ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557cceba6900 con 0x557ccec41140 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557cceb4cef0 con 0x557ccec40820 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8096381740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557cceba28c0 con 0x557ccec4ba00 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.013+0000 7f8091400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8090a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec4b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f808be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec4ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58604/0 (socket says 172.21.15.52:58604) 2024-04-26T02:07:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 -- 172.21.15.52:0/4252765772 learned_addr learned my addr 172.21.15.52:0/4252765772 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 -- 172.21.15.52:0/4252765772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 msgr2=0x557ccec4ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec4ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 -- 172.21.15.52:0/4252765772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 msgr2=0x557ccec4b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec4b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.014+0000 7f8091400640 1 -- 172.21.15.52:0/4252765772 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cce8b9210 con 0x557ccec40820 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8091400640 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8084009c20 tx=0x7f8084011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f24191969477e3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8081c00640 1 -- 172.21.15.52:0/4252765772 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808401e070 con 0x557ccec40820 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8081c00640 1 -- 172.21.15.52:0/4252765772 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80840198e0 con 0x557ccec40820 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8081c00640 1 -- 172.21.15.52:0/4252765772 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808400e8b0 con 0x557ccec40820 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8089600640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 0x7f80700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8089600640 1 -- 172.21.15.52:0/544093900 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f80780f0160 con 0x7f808c109470 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.015+0000 7f8090a00640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 0x7f80700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 -- 172.21.15.52:0/4252765772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 msgr2=0x557ccec40c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8084009c20 tx=0x7f8084011810 comp rx=0 tx=0).stop 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 -- 172.21.15.52:0/4252765772 shutdown_connections 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec4ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec4b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 --2- 172.21.15.52:0/4252765772 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec40c00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 -- 172.21.15.52:0/4252765772 >> 172.21.15.52:0/4252765772 conn(0x557ccec3cd30 msgr2=0x557ccec3d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 -- 172.21.15.52:0/4252765772 shutdown_connections 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8096381740 1 -- 172.21.15.52:0/4252765772 wait complete. 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.016+0000 7f8090a00640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 0x7f80700a6300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x557ccec41dc0 tx=0x7f807c010040 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.017+0000 7f8096381740 1 -- start start 2024-04-26T02:07:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.017+0000 7f8096381740 1 Processor -- start 2024-04-26T02:07:02.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.017+0000 7f8096381740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec40820 0x557ccec48a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec48f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557ccec4e630 con 0x557ccec41140 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x557cceb60310 con 0x557ccec4ba00 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557ccec4e7b0 con 0x557ccec40820 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8096381740 10 client.0 _ll_register_callbacks cb 0x557cceb2ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8090a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec40820 0x557ccec48a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:58614/0 (socket says 172.21.15.52:58614) 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 -- 172.21.15.52:0/2818153153 learned_addr learned my addr 172.21.15.52:0/2818153153 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f808be00640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec48f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 -- 172.21.15.52:0/2818153153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec40820 msgr2=0x557ccec48a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec40820 0x557ccec48a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 -- 172.21.15.52:0/2818153153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 msgr2=0x557ccec48f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec48f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.018+0000 7f8091400640 1 -- 172.21.15.52:0/2818153153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cce985620 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8091400640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8084005890 tx=0x7f80840096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782e7ed2885823de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8076000640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808401e040 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f8084009dd0 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80840198e0 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.019+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f808402f050 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 10 client.27506 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x557ccec5a8f0 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80840198e0 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f8084021070 con 0x557ccec4ba00 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 10 client.27506 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 10 client.27506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 20 client.27506 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccec5bd50 0x557ccec5e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 4 client.27506 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:07:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.020+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ccebba8b0 con 0x557ccec5bd50 2024-04-26T02:07:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.021+0000 7f8090a00640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccec5bd50 0x557ccec5e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.021+0000 7f8090a00640 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccec5bd50 0x557ccec5e140 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab850074a3b07c4d server_cookie=2b51175404616a03 in_seq=0 out_seq=0 2024-04-26T02:07:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.021+0000 7f8076000640 10 client.27506 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8076000640 1 -- 172.21.15.52:0/2818153153 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ccebba8b0 con 0x557ccec5bd50 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8076000640 10 client.27506 handle_command_reply: tid=0 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 client.27506 shutdown 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 msgr2=0x557ccec49480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8084005890 tx=0x7f80840096d0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 shutdown_connections 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccec5bd50 0x557ccec5e140 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec4ba00 0x557ccec49480 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec41140 0x557ccec48f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 --2- 172.21.15.52:0/2818153153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec40820 0x557ccec48a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8076000640 0 client.27506 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.022+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 >> 172.21.15.52:0/2818153153 conn(0x557ccec3cd30 msgr2=0x557ccec42ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.023+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 shutdown_connections 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.023+0000 7f8096381740 1 -- 172.21.15.52:0/2818153153 wait complete. 2024-04-26T02:07:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.023+0000 7f8096381740 20 client.27506 trim_cache size 0 max 16384 2024-04-26T02:07:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 Processor -- start 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 -- start start 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec41140 0x557ccec574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec63880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1b2550 con 0x557ccec40820 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1afe20 con 0x557ccec41140 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.043+0000 7f8096381740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1adc90 con 0x557ccec4ba00 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39964/0 (socket says 172.21.15.52:39964) 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 -- 172.21.15.52:0/1789785766 learned_addr learned my addr 172.21.15.52:0/1789785766 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8091400640 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec63880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f808be00640 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec41140 0x557ccec574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 -- 172.21.15.52:0/1789785766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 msgr2=0x557ccec63880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec63880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 -- 172.21.15.52:0/1789785766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec41140 msgr2=0x557ccec574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec41140 0x557ccec574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.044+0000 7f8090a00640 1 -- 172.21.15.52:0/1789785766 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ccec41680 con 0x557ccec40820 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.045+0000 7f8090a00640 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f807c009f50 tx=0x7f807c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b042a0550dcc0f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.045+0000 7f8081200640 1 -- 172.21.15.52:0/1789785766 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c02b490 con 0x557ccec40820 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.045+0000 7f8081200640 1 -- 172.21.15.52:0/1789785766 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f807c031070 con 0x557ccec40820 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.045+0000 7f8081200640 1 -- 172.21.15.52:0/1789785766 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c01a450 con 0x557ccec40820 2024-04-26T02:07:02.045 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- 172.21.15.52:0/1789785766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 msgr2=0x557ccec3c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f807c009f50 tx=0x7f807c016000 comp rx=0 tx=0).stop 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- 172.21.15.52:0/1789785766 shutdown_connections 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec4ba00 0x557ccec63880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec41140 0x557ccec574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 --2- 172.21.15.52:0/1789785766 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec40820 0x557ccec3c610 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- 172.21.15.52:0/1789785766 >> 172.21.15.52:0/1789785766 conn(0x557ccec3cd30 msgr2=0x557ccec49250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- 172.21.15.52:0/1789785766 shutdown_connections 2024-04-26T02:07:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- 172.21.15.52:0/1789785766 wait complete. 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.046+0000 7f8096381740 1 -- start start 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 Processor -- start 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec51650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec41140 0x557ccec51b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1ae2e0 con 0x557ccec4ba00 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1af750 con 0x557ccec40820 2024-04-26T02:07:02.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1ae9b0 con 0x557ccec41140 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.047+0000 7f8096381740 10 client.0 _ll_register_callbacks cb 0x7f8084009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39972/0 (socket says 172.21.15.52:39972) 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 -- 172.21.15.52:0/95422403 learned_addr learned my addr 172.21.15.52:0/95422403 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f808be00640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec41140 0x557ccec51b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8090a00640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec51650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 -- 172.21.15.52:0/95422403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec41140 msgr2=0x557ccec51b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec41140 0x557ccec51b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 -- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 msgr2=0x557ccec51650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec51650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 -- 172.21.15.52:0/95422403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ccecc1300 con 0x557ccec4ba00 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8091400640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8084004c60 tx=0x7f8084019d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504eca47084a267f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:02.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.048+0000 7f8076000640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084038060 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 8521+0+0 (secure 0 0 0) 0x7f80840197b0 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f807c031070 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f808401e030 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8096381740 10 client.17316 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x557ccecc25d0 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.049+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f807c031070 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7f8084021070 con 0x557ccec4ba00 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 10 client.17316 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 10 client.17316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 20 client.17316 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccecc2c30 0x557ccecc5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 4 client.17316 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ccebb4320 con 0x557ccecc2c30 2024-04-26T02:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.050+0000 7f8090a00640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccecc2c30 0x557ccecc5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.051+0000 7f8090a00640 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccecc2c30 0x557ccecc5050 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9586c0f12e0660 server_cookie=3c0f84504d3231ad in_seq=0 out_seq=0 2024-04-26T02:07:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.051+0000 7f8076000640 10 client.17316 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.260+0000 7f8076000640 1 -- 172.21.15.52:0/95422403 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ccebb4320 con 0x557ccecc2c30 2024-04-26T02:07:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.260+0000 7f8076000640 10 client.17316 handle_command_reply: tid=0 2024-04-26T02:07:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.260+0000 7f8096381740 1 client.17316 shutdown 2024-04-26T02:07:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 msgr2=0x557ccec520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8084004c60 tx=0x7f8084019d30 comp rx=0 tx=0).stop 2024-04-26T02:07:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 shutdown_connections 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x557ccecc2c30 0x557ccecc5050 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557ccec4ba00 0x557ccec520d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557ccec41140 0x557ccec51b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 --2- 172.21.15.52:0/95422403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x557ccec40820 0x557ccec51650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 >> 172.21.15.52:0/95422403 conn(0x557ccec3cd30 msgr2=0x557ccec489a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8076000640 0 client.17316 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 shutdown_connections 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 1 -- 172.21.15.52:0/95422403 wait complete. 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.261+0000 7f8096381740 20 client.17316 trim_cache size 0 max 16384 2024-04-26T02:07:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:07:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:07:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:07:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:07:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 msgr2=0x7f80700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 0x7f80700a6300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x557ccec41dc0 tx=0x7f807c010040 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 msgr2=0x7f808c1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8078035040 tx=0x7f8078032d50 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 shutdown_connections 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f80700a3e40 0x7f80700a6300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f808c19a6f0 0x7f808c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f808c109470 0x7f808c1a0b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 --2- 172.21.15.52:0/544093900 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f808c1043a0 0x7f808c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.262+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 >> 172.21.15.52:0/544093900 conn(0x7f808c0ffc40 msgr2=0x7f808c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.263+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 shutdown_connections 2024-04-26T02:07:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:02.263+0000 7f8093400640 1 -- 172.21.15.52:0/544093900 wait complete. 2024-04-26T02:07:02.772 INFO:tasks.workunit.client.0.smithi052.stdout: 10 188480 21.04 MB/sec execute 379 sec latency 2717.486 ms 2024-04-26T02:07:03.228 INFO:tasks.workunit.client.0.smithi052.stdout:[171545] write failed on handle 13093 (Resource temporarily unavailable) 2024-04-26T02:07:03.228 INFO:tasks.workunit.client.0.smithi052.stdout:Child failed with status 1 2024-04-26T02:07:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:07:03.256 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-26T02:07:03.256 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T02:07:03.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:03 smithi191 ceph-mon[32745]: pgmap v699: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 768 op/s 2024-04-26T02:07:03.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:03 smithi183 ceph-mon[32752]: pgmap v699: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 768 op/s 2024-04-26T02:07:03.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:03 smithi052 ceph-mon[27851]: pgmap v699: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 768 op/s 2024-04-26T02:07:04.641 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T02:07:04.846 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=326fac8eca2948e9b95dfbf16bce577a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T02:07:04.849 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-26T02:07:04.860 INFO:tasks.quiescer:joining Quiescers 2024-04-26T02:07:04.861 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-26T02:07:04.861 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:07:05.436 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:05 smithi191 ceph-mon[32745]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-04-26T02:07:05.907 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:05 smithi052 ceph-mon[27851]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-04-26T02:07:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:05 smithi183 ceph-mon[32752]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 450 op/s 2024-04-26T02:07:07.628 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:07 smithi052 ceph-mon[27851]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 525 op/s 2024-04-26T02:07:07.888 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:07 smithi052 ceph-mon[27851]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:07 smithi191 ceph-mon[32745]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 525 op/s 2024-04-26T02:07:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:07 smithi191 ceph-mon[32745]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:07 smithi183 ceph-mon[32752]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 525 op/s 2024-04-26T02:07:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:07 smithi183 ceph-mon[32752]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T02:07:08.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 -- 172.21.15.52:0/897860186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6174100bc0 msgr2=0x7f6174100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 --2- 172.21.15.52:0/897860186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6174100bc0 0x7f6174100fa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6164009990 tx=0x7f6164009ca0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 -- 172.21.15.52:0/897860186 shutdown_connections 2024-04-26T02:07:08.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 --2- 172.21.15.52:0/897860186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61741138e0 0x7f6174115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 --2- 172.21.15.52:0/897860186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6174100bc0 0x7f6174100fa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 --2- 172.21.15.52:0/897860186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741014e0 0x7f61741133a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 -- 172.21.15.52:0/897860186 >> 172.21.15.52:0/897860186 conn(0x7f6174077b40 msgr2=0x7f61740feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:08.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 -- 172.21.15.52:0/897860186 shutdown_connections 2024-04-26T02:07:08.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.409+0000 7f617b200700 1 -- 172.21.15.52:0/897860186 wait complete. 2024-04-26T02:07:08.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.410+0000 7f617b200700 1 Processor -- start 2024-04-26T02:07:08.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.411+0000 7f617b200700 1 -- start start 2024-04-26T02:07:08.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6174100bc0 0x7f61741a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:08.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:08.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741138e0 0x7f617419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:08.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f6173400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:08.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f6173400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:34264/0 (socket says 172.21.15.52:34264) 2024-04-26T02:07:08.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f6173400700 1 -- 172.21.15.52:0/3853750961 learned_addr learned my addr 172.21.15.52:0/3853750961 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:08.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6174119260 con 0x7f61741138e0 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f61741190e0 con 0x7f61741014e0 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.412+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f61741193e0 con 0x7f6174100bc0 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6173400700 1 -- 172.21.15.52:0/3853750961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6174100bc0 msgr2=0x7f61741a53e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6173400700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6174100bc0 0x7f61741a53e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6173400700 1 -- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741138e0 msgr2=0x7f617419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6179200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741138e0 0x7f617419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:08.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6173400700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741138e0 0x7f617419f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.413+0000 7f6173400700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617419fd00 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.414+0000 7f6173400700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f616000c9b0 tx=0x7f616000cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.414+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160007b00 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.414+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617419fff0 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.414+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61741ac2b0 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.415+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6160004ce0 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.415+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61600055d0 con 0x7f61741014e0 2024-04-26T02:07:08.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.416+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6154005330 con 0x7f61741014e0 2024-04-26T02:07:08.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.420+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6160004750 con 0x7f61741014e0 2024-04-26T02:07:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.422+0000 7f6170c00700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 0x7f61580a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:08.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.422+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8565+0+0 (secure 0 0 0) 0x7f61600d0860 con 0x7f61741014e0 2024-04-26T02:07:08.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.424+0000 7f6173e00700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 0x7f61580a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:08.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.425+0000 7f6173e00700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 0x7f61580a62f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f61640098e0 tx=0x7f616400b610 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:08.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.426+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6160014030 con 0x7f61741014e0 2024-04-26T02:07:08.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.917+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61540058f0 con 0x7f61741014e0 2024-04-26T02:07:08.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.918+0000 7f6170c00700 1 -- 172.21.15.52:0/3853750961 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+11882 (secure 0 0 0) 0x7f616009ced0 con 0x7f61741014e0 2024-04-26T02:07:08.917 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:07:08.918 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":73,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:06:08.474351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27443":{"gid":27443,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1603134393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1603134393},{"type":"v1","addr":"172.21.15.183:6837","nonce":1603134393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 msgr2=0x7f61580a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 0x7f61580a62f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f61640098e0 tx=0x7f616400b610 comp rx=0 tx=0).stop 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 msgr2=0x7f61741a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f616000c9b0 tx=0x7f616000cd70 comp rx=0 tx=0).stop 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 shutdown_connections 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f61580a3e30 0x7f61580a62f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f61741138e0 0x7f617419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6174100bc0 0x7f61741a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 --2- 172.21.15.52:0/3853750961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61741014e0 0x7f61741a5920 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.920+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 >> 172.21.15.52:0/3853750961 conn(0x7f6174077b40 msgr2=0x7f61741121f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.921+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 shutdown_connections 2024-04-26T02:07:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:08.921+0000 7f617b200700 1 -- 172.21.15.52:0/3853750961 wait complete. 2024-04-26T02:07:08.921 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 73 2024-04-26T02:07:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:09 smithi191 ceph-mon[32745]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:09 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3853750961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:09 smithi183 ceph-mon[32752]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:09.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:09 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3853750961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:09.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:09 smithi052 ceph-mon[27851]: pgmap v703: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:09.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:09 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3853750961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:07:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:11 smithi191 ceph-mon[32745]: pgmap v704: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:11.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:11 smithi183 ceph-mon[32752]: pgmap v704: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:11.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:11 smithi052 ceph-mon[27851]: pgmap v704: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.1 MiB/s wr, 308 op/s 2024-04-26T02:07:12.856 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24524 quiesce db --cancel --all' 2024-04-26T02:07:12.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24524 quiesce db --cancel --all 2024-04-26T02:07:13.329 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:13.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:13 smithi052 ceph-mon[27851]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-04-26T02:07:13.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:13 smithi052 ceph-mon[27851]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:13 smithi191 ceph-mon[32745]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-04-26T02:07:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:13 smithi191 ceph-mon[32745]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:13.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:13 smithi183 ceph-mon[32752]: pgmap v705: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-04-26T02:07:13.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:13 smithi183 ceph-mon[32752]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T02:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 -- 172.21.15.52:0/2605537512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 msgr2=0x7fc4c8110e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/2605537512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c8110e70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009900 tx=0x7fc4c002a3f0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 -- 172.21.15.52:0/2605537512 shutdown_connections 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/2605537512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c8110e70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/2605537512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c8104b80 0x7fc4c8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/2605537512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c81041d0 0x7fc4c81045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 -- 172.21.15.52:0/2605537512 >> 172.21.15.52:0/2605537512 conn(0x7fc4c80ffa70 msgr2=0x7fc4c8101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 -- 172.21.15.52:0/2605537512 shutdown_connections 2024-04-26T02:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.463+0000 7fc4c7e00640 1 -- 172.21.15.52:0/2605537512 wait complete. 2024-04-26T02:07:15.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.464+0000 7fc4c7e00640 1 Processor -- start 2024-04-26T02:07:15.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.464+0000 7fc4c7e00640 1 -- start start 2024-04-26T02:07:15.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c81a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c819a5c0 0x7fc4c819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c8114260 con 0x7fc4c81041d0 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81140e0 con 0x7fc4c81092a0 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81143e0 con 0x7fc4c819a5c0 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c81a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.466 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50790/0 (socket says 172.21.15.52:50790) 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c6a00640 1 -- 172.21.15.52:0/1481660708 learned_addr learned my addr 172.21.15.52:0/1481660708 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.465+0000 7fc4c7400640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c819a5c0 0x7fc4c819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 -- 172.21.15.52:0/1481660708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c819a5c0 msgr2=0x7fc4c819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c819a5c0 0x7fc4c819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 -- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 msgr2=0x7fc4c81a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 -- 172.21.15.52:0/1481660708 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c819b0e0 con 0x7fc4c81092a0 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c6000640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c81a0950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e9c0 tx=0x7fc4b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4bf400640 1 -- 172.21.15.52:0/1481660708 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000ccc0 con 0x7fc4c81092a0 2024-04-26T02:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4bf400640 1 -- 172.21.15.52:0/1481660708 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b00108e0 con 0x7fc4c81092a0 2024-04-26T02:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c819b3d0 con 0x7fc4c81092a0 2024-04-26T02:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.466+0000 7fc4bf400640 1 -- 172.21.15.52:0/1481660708 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00187a0 con 0x7fc4c81092a0 2024-04-26T02:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.467+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c81a7620 con 0x7fc4c81092a0 2024-04-26T02:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 Processor -- start 2024-04-26T02:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 -- start start 2024-04-26T02:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402171d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x56540217ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5654020d7900 con 0x56540217cb90 2024-04-26T02:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56540207def0 con 0x5654021722d0 2024-04-26T02:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4d0421740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5654020d38c0 con 0x5654021719b0 2024-04-26T02:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.470+0000 7fc4bf400640 1 -- 172.21.15.52:0/1481660708 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc4b0010430 con 0x7fc4c81092a0 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44960/0 (socket says 172.21.15.52:44960) 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 -- 172.21.15.52:0/3777061006 learned_addr learned my addr 172.21.15.52:0/3777061006 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c7400640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402171d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6000640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x56540217ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 -- 172.21.15.52:0/3777061006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 msgr2=0x565402171d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402171d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 -- 172.21.15.52:0/3777061006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 msgr2=0x56540217ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x56540217ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 -- 172.21.15.52:0/3777061006 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565401dea210 con 0x5654021722d0 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c7400640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402171d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.471+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8011930 tx=0x7fc4b8011df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeda6b9d50377493 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4b7400640 1 -- 172.21.15.52:0/3777061006 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b80112f0 con 0x5654021722d0 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4b7400640 1 -- 172.21.15.52:0/3777061006 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b8010430 con 0x5654021722d0 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4b7400640 1 -- 172.21.15.52:0/3777061006 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8019640 con 0x5654021722d0 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4bf400640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 0x7fc4ac0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4bf400640 1 -- 172.21.15.52:0/1481660708 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fc4b00d0ae0 con 0x7fc4c81092a0 2024-04-26T02:07:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4d0421740 1 -- 172.21.15.52:0/3777061006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 msgr2=0x56540217c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.472+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8011930 tx=0x7fc4b8011df0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4c7400640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 0x7fc4ac0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 -- 172.21.15.52:0/3777061006 shutdown_connections 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x56540217ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x56540217c650 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3777061006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402171d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 -- 172.21.15.52:0/3777061006 >> 172.21.15.52:0/3777061006 conn(0x56540216dec0 msgr2=0x56540216e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.473+0000 7fc4d0421740 1 -- 172.21.15.52:0/3777061006 shutdown_connections 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.474+0000 7fc4d0421740 1 -- 172.21.15.52:0/3777061006 wait complete. 2024-04-26T02:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.474+0000 7fc4c7400640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 0x7fc4ac0a61d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0006010 tx=0x7fc4c0005a40 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.474+0000 7fc4d0421740 1 -- start start 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.474+0000 7fc4d0421740 1 Processor -- start 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402179a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56540217cb90 0x56540217a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56540217f7c0 con 0x5654021719b0 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x565402091310 con 0x5654021722d0 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56540217f940 con 0x56540217cb90 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4d0421740 10 client.0 _ll_register_callbacks cb 0x56540205fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44964/0 (socket says 172.21.15.52:44964) 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4c6000640 1 -- 172.21.15.52:0/2313414958 learned_addr learned my addr 172.21.15.52:0/2313414958 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4c7400640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56540217cb90 0x56540217a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.475+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402179a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 -- 172.21.15.52:0/2313414958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56540217cb90 msgr2=0x56540217a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56540217cb90 0x56540217a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 -- 172.21.15.52:0/2313414958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 msgr2=0x565402179a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402179a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 -- 172.21.15.52:0/2313414958 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565401eb6620 con 0x5654021722d0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402179a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4c6000640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc4b009c7d0 tx=0x7fc4b0010720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9ead9862b3e9ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4abe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00a0dd0 con 0x5654021722d0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fc4b0010a90 con 0x5654021722d0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4b8010430 con 0x5654021722d0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fc4b00a5690 con 0x5654021722d0 2024-04-26T02:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4d0421740 10 client.27530 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.476+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x56540218b920 con 0x5654021722d0 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b8010430 con 0x5654021722d0 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7fc4b0098820 con 0x5654021722d0 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 10 client.27530 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 10 client.27530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 20 client.27530 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56540218cd90 0x56540218f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 4 client.27530 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654020eb8b0 con 0x56540218cd90 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.477+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56540218cd90 0x56540218f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.478+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56540218cd90 0x56540218f180 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c06f701ea6a2d73 server_cookie=29b23d9bc30d0f84 in_seq=0 out_seq=0 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.478+0000 7fc4abe00640 10 client.27530 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.478+0000 7fc4abe00640 1 -- 172.21.15.52:0/2313414958 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654020eb8b0 con 0x56540218cd90 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.478+0000 7fc4abe00640 10 client.27530 handle_command_reply: tid=0 2024-04-26T02:07:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 client.27530 shutdown 2024-04-26T02:07:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 msgr2=0x565402179f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc4b009c7d0 tx=0x7fc4b0010720 comp rx=0 tx=0).stop 2024-04-26T02:07:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 shutdown_connections 2024-04-26T02:07:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56540218cd90 0x56540218f180 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56540217cb90 0x56540217a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402179f60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 --2- 172.21.15.52:0/2313414958 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402179a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 >> 172.21.15.52:0/2313414958 conn(0x56540216dec0 msgr2=0x565402173bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4abe00640 0 client.27530 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 shutdown_connections 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 1 -- 172.21.15.52:0/2313414958 wait complete. 2024-04-26T02:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.479+0000 7fc4d0421740 20 client.27530 trim_cache size 0 max 16384 2024-04-26T02:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.496+0000 7fc4d0421740 1 Processor -- start 2024-04-26T02:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.496+0000 7fc4d0421740 1 -- start start 2024-04-26T02:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402188680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81b2570 con 0x5654021719b0 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81afe40 con 0x56540217cb90 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4d0421740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81adcb0 con 0x5654021722d0 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c6a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402188680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60542/0 (socket says 172.21.15.52:60542) 2024-04-26T02:07:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44970/0 (socket says 172.21.15.52:44970) 2024-04-26T02:07:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.497+0000 7fc4c6000640 1 -- 172.21.15.52:0/379604952 learned_addr learned my addr 172.21.15.52:0/379604952 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 -- 172.21.15.52:0/379604952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 msgr2=0x56540216d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 -- 172.21.15.52:0/379604952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 msgr2=0x565402188680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402188680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 -- 172.21.15.52:0/379604952 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565402172100 con 0x56540217cb90 2024-04-26T02:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c6000640 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4c7400640 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4c002a910 tx=0x7fc4c003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bc737ff797b133 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4b6a00640 1 -- 172.21.15.52:0/379604952 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0035040 con 0x56540217cb90 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4b6a00640 1 -- 172.21.15.52:0/379604952 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4c0003e80 con 0x56540217cb90 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.498+0000 7fc4b6a00640 1 -- 172.21.15.52:0/379604952 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c003d7d0 con 0x56540217cb90 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 -- 172.21.15.52:0/379604952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 msgr2=0x565402178a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4c002a910 tx=0x7fc4c003f000 comp rx=0 tx=0).stop 2024-04-26T02:07:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 -- 172.21.15.52:0/379604952 shutdown_connections 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56540217cb90 0x565402178a00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021722d0 0x56540216d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 --2- 172.21.15.52:0/379604952 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5654021719b0 0x565402188680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 -- 172.21.15.52:0/379604952 >> 172.21.15.52:0/379604952 conn(0x56540216dec0 msgr2=0x5654021782f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 -- 172.21.15.52:0/379604952 shutdown_connections 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.499+0000 7fc4d0421740 1 -- 172.21.15.52:0/379604952 wait complete. 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.500+0000 7fc4d0421740 1 -- start start 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.500+0000 7fc4d0421740 1 Processor -- start 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402181e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x5654021f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81ae300 con 0x56540217cb90 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81af770 con 0x5654021722d0 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81ae9d0 con 0x5654021719b0 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4d0421740 10 client.0 _ll_register_callbacks cb 0x7fc4b009c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:44986/0 (socket says 172.21.15.52:44986) 2024-04-26T02:07:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 -- 172.21.15.52:0/3283842209 learned_addr learned my addr 172.21.15.52:0/3283842209 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402181e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c7400640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x5654021f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 -- 172.21.15.52:0/3283842209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 msgr2=0x565402181e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402181e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 -- 172.21.15.52:0/3283842209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 msgr2=0x5654021f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x5654021f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6000640 1 -- 172.21.15.52:0/3283842209 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654021f4110 con 0x5654021722d0 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.501+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402181e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4c6000640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00994a0 tx=0x7fc4b00ce680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1866633173953e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4abe00640 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b009bf20 con 0x5654021722d0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fc4b009c0c0 con 0x5654021722d0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5654020e5320 con 0x5654021722d0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fc4b009b990 con 0x5654021722d0 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4d0421740 10 client.27536 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5654021fb500 con 0x5654021722d0 2024-04-26T02:07:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.502+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5654020e5320 con 0x5654021722d0 2024-04-26T02:07:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7fc4b0087080 con 0x5654021722d0 2024-04-26T02:07:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 10 client.27536 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 10 client.27536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 20 client.27536 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5654021fbad0 0x5654021fdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 4 client.27536 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654020df190 con 0x5654021fbad0 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5654021fbad0 0x5654021fdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4c6a00640 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5654021fbad0 0x5654021fdef0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfcbec1b0ac7c880 server_cookie=f3dc62f8c45dd341 in_seq=0 out_seq=0 2024-04-26T02:07:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.503+0000 7fc4abe00640 10 client.27536 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4abe00640 1 -- 172.21.15.52:0/3283842209 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654020df190 con 0x5654021fbad0 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4abe00640 10 client.27536 handle_command_reply: tid=0 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 client.27536 shutdown 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 msgr2=0x565402182390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00994a0 tx=0x7fc4b00ce680 comp rx=0 tx=0).stop 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 shutdown_connections 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x5654021fbad0 0x5654021fdef0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56540217cb90 0x5654021f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x5654021722d0 0x565402182390 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4abe00640 0 client.27536 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 --2- 172.21.15.52:0/3283842209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5654021719b0 0x565402181e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 >> 172.21.15.52:0/3283842209 conn(0x56540216dec0 msgr2=0x565402173bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 shutdown_connections 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 1 -- 172.21.15.52:0/3283842209 wait complete. 2024-04-26T02:07:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.631+0000 7fc4d0421740 20 client.27536 trim_cache size 0 max 16384 2024-04-26T02:07:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-04-26T02:07:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "message": "", 2024-04-26T02:07:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-04-26T02:07:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-04-26T02:07:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 msgr2=0x7fc4ac0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 0x7fc4ac0a61d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0006010 tx=0x7fc4c0005a40 comp rx=0 tx=0).stop 2024-04-26T02:07:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 msgr2=0x7fc4c81a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c81a0950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000e9c0 tx=0x7fc4b000ee80 comp rx=0 tx=0).stop 2024-04-26T02:07:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 shutdown_connections 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fc4ac0a3d10 0x7fc4ac0a61d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4c819a5c0 0x7fc4c819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc4c81092a0 0x7fc4c81a0950 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 --2- 172.21.15.52:0/1481660708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4c81041d0 0x7fc4c81a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 >> 172.21.15.52:0/1481660708 conn(0x7fc4c80ffa70 msgr2=0x7fc4c8101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.632+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 shutdown_connections 2024-04-26T02:07:15.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-04-26T02:07:15.633+0000 7fc4c7e00640 1 -- 172.21.15.52:0/1481660708 wait complete. 2024-04-26T02:07:15.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 -- 172.21.15.52:0/1974195999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5881138f0 msgr2=0x7fb588115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 --2- 172.21.15.52:0/1974195999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5881138f0 0x7fb588115ce0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb58000b0a0 tx=0x7fb58000b3b0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 -- 172.21.15.52:0/1974195999 shutdown_connections 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 --2- 172.21.15.52:0/1974195999 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5881138f0 0x7fb588115ce0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 --2- 172.21.15.52:0/1974195999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb588076f70 0x7fb588077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 --2- 172.21.15.52:0/1974195999 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588077890 0x7fb5881133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 -- 172.21.15.52:0/1974195999 >> 172.21.15.52:0/1974195999 conn(0x7fb588100740 msgr2=0x7fb588102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 -- 172.21.15.52:0/1974195999 shutdown_connections 2024-04-26T02:07:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.665+0000 7fb590400700 1 -- 172.21.15.52:0/1974195999 wait complete. 2024-04-26T02:07:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.666+0000 7fb590400700 1 Processor -- start 2024-04-26T02:07:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.666+0000 7fb590400700 1 -- start start 2024-04-26T02:07:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588076f70 0x7fb5881a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb588077890 0x7fb5881a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb588119270 con 0x7fb588076f70 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881190f0 con 0x7fb588077890 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb590400700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881193f0 con 0x7fb5881138f0 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb58e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb58da00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588076f70 0x7fb5881a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb58e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60574/0 (socket says 172.21.15.52:60574) 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb58d000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb588077890 0x7fb5881a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.667+0000 7fb58e400700 1 -- 172.21.15.52:0/2887264439 learned_addr learned my addr 172.21.15.52:0/2887264439 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 -- 172.21.15.52:0/2887264439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb588077890 msgr2=0x7fb5881a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb588077890 0x7fb5881a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 -- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588076f70 msgr2=0x7fb5881a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588076f70 0x7fb5881a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 -- 172.21.15.52:0/2887264439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58819fcf0 con 0x7fb5881138f0 2024-04-26T02:07:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb58e400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb58000b070 tx=0x7fb580004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb586000700 1 -- 172.21.15.52:0/2887264439 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580021070 con 0x7fb5881138f0 2024-04-26T02:07:15.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb586000700 1 -- 172.21.15.52:0/2887264439 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb580007e30 con 0x7fb5881138f0 2024-04-26T02:07:15.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb586000700 1 -- 172.21.15.52:0/2887264439 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580004620 con 0x7fb5881138f0 2024-04-26T02:07:15.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.668+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58819ff80 con 0x7fb5881138f0 2024-04-26T02:07:15.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.669+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5881ac2a0 con 0x7fb5881138f0 2024-04-26T02:07:15.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.673+0000 7fb586000700 1 -- 172.21.15.52:0/2887264439 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb580007980 con 0x7fb5881138f0 2024-04-26T02:07:15.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.674+0000 7fb586000700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 0x7fb5700a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.674+0000 7fb59ac9cb80 1 Processor -- start 2024-04-26T02:07:15.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.674+0000 7fb586000700 1 -- 172.21.15.52:0/2887264439 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fb58000f020 con 0x7fb5881138f0 2024-04-26T02:07:15.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb58da00700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 0x7fb5700a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 -- start start 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb2cb0 0x56105abb3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56105ab05f00 con 0x56105abb2cb0 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56105aaac7a0 con 0x56105abbe440 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.675+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56105ab01ec0 con 0x56105abb35d0 2024-04-26T02:07:15.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58d000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45004/0 (socket says 172.21.15.52:45004) 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58d000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60582/0 (socket says 172.21.15.52:60582) 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 -- 172.21.15.52:0/3408833792 learned_addr learned my addr 172.21.15.52:0/3408833792 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58da00700 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb2cb0 0x56105abb3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58da00700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 0x7fb5700a6270 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009680 tx=0x7fb57c009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 -- 172.21.15.52:0/3408833792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 msgr2=0x56105abbdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 -- 172.21.15.52:0/3408833792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb2cb0 msgr2=0x56105abb3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb2cb0 0x56105abb3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 -- 172.21.15.52:0/3408833792 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56105aaa26f0 con 0x56105abbe440 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58d000700 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb58e400700 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5800a6240 tx=0x7fb5800040b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad869a0c09bf6aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.676+0000 7fb57aa00700 1 -- 172.21.15.52:0/3408833792 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580007c70 con 0x56105abbe440 2024-04-26T02:07:15.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.677+0000 7fb57aa00700 1 -- 172.21.15.52:0/3408833792 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb58009e290 con 0x56105abbe440 2024-04-26T02:07:15.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.677+0000 7fb57aa00700 1 -- 172.21.15.52:0/3408833792 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5800be070 con 0x56105abbe440 2024-04-26T02:07:15.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.677+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3408833792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 msgr2=0x56105abc0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.677+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5800a6240 tx=0x7fb5800040b0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3408833792 shutdown_connections 2024-04-26T02:07:15.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105abc0830 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb2cb0 0x56105abb3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3408833792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb35d0 0x56105abbdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3408833792 >> 172.21.15.52:0/3408833792 conn(0x56105aae1fc0 msgr2=0x56105abaf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3408833792 shutdown_connections 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3408833792 wait complete. 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.678+0000 7fb59ac9cb80 1 -- start start 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 Processor -- start 2024-04-26T02:07:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abbb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56105aaac7a0 con 0x56105abb35d0 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x56105aabf8d0 con 0x56105abb2cb0 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x56105abc0d70 con 0x56105abbe440 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb59ac9cb80 10 client.0 _ll_register_callbacks cb 0x56105aa9e790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb58da00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb58d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb58da00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45008/0 (socket says 172.21.15.52:45008) 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.679+0000 7fb58da00700 1 -- 172.21.15.52:0/3058531309 learned_addr learned my addr 172.21.15.52:0/3058531309 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58e400700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abbb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 -- 172.21.15.52:0/3058531309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 msgr2=0x56105abbb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abbb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 -- 172.21.15.52:0/3058531309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 msgr2=0x56105abbaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 -- 172.21.15.52:0/3058531309 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56105aa53940 con 0x56105abb2cb0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58e400700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abbb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb58da00700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb57c011fd0 tx=0x7fb57c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da14a4f81a664985 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb56f400700 10 client.0 ms_handle_connect on v2:172.21.15.183:3300/0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.680+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c004020 con 0x56105abb2cb0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb58009e290 con 0x56105abb2cb0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mon.2 v2:172.21.15.183:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fb57c0041c0 con 0x56105abb2cb0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fb57c0043c0 con 0x56105abb2cb0 2024-04-26T02:07:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb59ac9cb80 10 client.27548 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x56105aa9cb60 con 0x56105abb2cb0 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.681+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mon.2 v2:172.21.15.183:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb58009e290 con 0x56105abb2cb0 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mon.2 v2:172.21.15.183:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7fb57c003920 con 0x56105abb2cb0 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 10 client.27548 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 10 client.27548 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 20 client.27548 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105abcda00 0x56105abcfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 4 client.27548 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T02:07:15.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56105ab136f0 con 0x56105abcda00 2024-04-26T02:07:15.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.682+0000 7fb58d000700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105abcda00 0x56105abcfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.683+0000 7fb58d000700 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105abcda00 0x56105abcfdf0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef2e8ac0f478a644 server_cookie=3483a05c891c6967 in_seq=0 out_seq=0 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.683+0000 7fb56f400700 10 client.27548 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.683+0000 7fb56f400700 1 -- 172.21.15.52:0/3058531309 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56105ab136f0 con 0x56105abcda00 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.683+0000 7fb56f400700 10 client.27548 handle_command_reply: tid=0 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.683+0000 7fb59ac9cb80 1 client.27548 shutdown 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 msgr2=0x56105abbdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb57c011fd0 tx=0x7fb57c0047c0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 shutdown_connections 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105abcda00 0x56105abcfdf0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abbb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abbdbf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/3058531309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb56f400700 0 client.27548 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 >> 172.21.15.52:0/3058531309 conn(0x56105aae1fc0 msgr2=0x56105abaedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 shutdown_connections 2024-04-26T02:07:15.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/3058531309 wait complete. 2024-04-26T02:07:15.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.684+0000 7fb59ac9cb80 20 client.27548 trim_cache size 0 max 16384 2024-04-26T02:07:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.706+0000 7fb59ac9cb80 1 Processor -- start 2024-04-26T02:07:15.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.706+0000 7fb59ac9cb80 1 -- start start 2024-04-26T02:07:15.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abc9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b6150 con 0x56105abb35d0 2024-04-26T02:07:15.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b39e0 con 0x56105abb2cb0 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b1850 con 0x56105abbe440 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60606/0 (socket says 172.21.15.52:60606) 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50886/0 (socket says 172.21.15.52:50886) 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 -- 172.21.15.52:0/1914061798 learned_addr learned my addr 172.21.15.52:0/1914061798 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58da00700 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abc9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 -- 172.21.15.52:0/1914061798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 msgr2=0x56105abc9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abc9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 -- 172.21.15.52:0/1914061798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 msgr2=0x56105abbce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.707+0000 7fb58e400700 1 -- 172.21.15.52:0/1914061798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56105abb94a0 con 0x56105abbe440 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.708+0000 7fb58e400700 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb58009d890 tx=0x7fb58009da10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aebaeffa2b1ece server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.708+0000 7fb57a000700 1 -- 172.21.15.52:0/1914061798 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580007c70 con 0x56105abbe440 2024-04-26T02:07:15.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.708+0000 7fb57a000700 1 -- 172.21.15.52:0/1914061798 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb58009db90 con 0x56105abbe440 2024-04-26T02:07:15.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.708+0000 7fb57a000700 1 -- 172.21.15.52:0/1914061798 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb580003c00 con 0x56105abbe440 2024-04-26T02:07:15.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/1914061798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 msgr2=0x56105abd56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb58009d890 tx=0x7fb58009da10 comp rx=0 tx=0).stop 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/1914061798 shutdown_connections 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abbe440 0x56105abd56c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abb2cb0 0x56105abc9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/1914061798 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abbce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/1914061798 >> 172.21.15.52:0/1914061798 conn(0x56105aae1fc0 msgr2=0x56105abbc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/1914061798 shutdown_connections 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/1914061798 wait complete. 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.709+0000 7fb59ac9cb80 1 -- start start 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 Processor -- start 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abc3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105ac33f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b1ea0 con 0x56105abb35d0 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b3310 con 0x56105abbe440 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb5881b2570 con 0x56105abb2cb0 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb59ac9cb80 10 client.0 _ll_register_callbacks cb 0x56105aa9e790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb58da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb58d000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abc3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb58da00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60618/0 (socket says 172.21.15.52:60618) 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.710+0000 7fb58da00700 1 -- 172.21.15.52:0/2931574877 learned_addr learned my addr 172.21.15.52:0/2931574877 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:15.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58e400700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105ac33f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 -- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 msgr2=0x56105ac33f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105ac33f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 -- 172.21.15.52:0/2931574877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 msgr2=0x56105abc3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abc3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 -- 172.21.15.52:0/2931574877 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56105ac34480 con 0x56105abb2cb0 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb58da00700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb57c003d10 tx=0x7fb57c009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7766b63a132c53b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb56f400700 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-04-26T02:07:15.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c003ac0 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56105ab0d790 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.711+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fb57c01c6d0 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.712+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7fb57c040050 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.712+0000 7fb59ac9cb80 10 client.27571 fetch_fsmap learned FSMap version 73 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.712+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x56105ac35660 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.712+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56105ab0d790 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.712+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 73) ==== 8338+0+0 (secure 0 0 0) 0x7fb57c037070 con 0x56105abb2cb0 2024-04-26T02:07:15.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 10 client.27571 fetch_fsmap finished waiting for FSMap version 73 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 10 client.27571 resolve_mds: validated gid 24524 aka daemon mds.b 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 20 client.27571 populate_metadata read hostname 'smithi052' 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105ac35da0 0x56105ac381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 4 client.27571 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 --> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x56105a9862c0 con 0x56105ac35da0 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb58d000700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105ac35da0 0x56105ac381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb58d000700 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105ac35da0 0x56105ac381c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88f666e92de55262 server_cookie=cdd6549aad42e7c9 in_seq=0 out_seq=0 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.713+0000 7fb56f400700 10 client.27571 ms_handle_connect on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb56f400700 1 -- 172.21.15.52:0/2931574877 <== mds.0 v2:172.21.15.183:6838/1410734884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x56105a9862c0 con 0x56105ac35da0 2024-04-26T02:07:15.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb56f400700 10 client.27571 handle_command_reply: tid=0 2024-04-26T02:07:15.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 client.27571 shutdown 2024-04-26T02:07:15.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 msgr2=0x56105abc2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb57c003d10 tx=0x7fb57c009470 comp rx=0 tx=0).stop 2024-04-26T02:07:15.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 shutdown_connections 2024-04-26T02:07:15.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:6838/1410734884,v1:172.21.15.183:6839/1410734884] conn(0x56105ac35da0 0x56105ac381c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x56105abbe440 0x56105ac33f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56105abb2cb0 0x56105abc2fe0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 --2- 172.21.15.52:0/2931574877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56105abb35d0 0x56105abc3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb56f400700 0 client.27571 ms_handle_reset on v2:172.21.15.183:6838/1410734884 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 >> 172.21.15.52:0/2931574877 conn(0x56105aae1fc0 msgr2=0x56105abbcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 shutdown_connections 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.714+0000 7fb59ac9cb80 1 -- 172.21.15.52:0/2931574877 wait complete. 2024-04-26T02:07:15.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb59ac9cb80 20 client.27571 trim_cache size 0 max 16384 2024-04-26T02:07:15.722 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:15.722 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 73, 2024-04-26T02:07:15.722 INFO:teuthology.orchestra.run.smithi052.stdout: "leader": 24524, 2024-04-26T02:07:15.722 INFO:teuthology.orchestra.run.smithi052.stdout: "set_version": 50, 2024-04-26T02:07:15.722 INFO:teuthology.orchestra.run.smithi052.stdout: "sets": {} 2024-04-26T02:07:15.723 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:15.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 msgr2=0x7fb5700a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 0x7fb5700a6270 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009680 tx=0x7fb57c009260 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 msgr2=0x7fb58819f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb58000b070 tx=0x7fb580004c80 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 shutdown_connections 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb5700a3db0 0x7fb5700a6270 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5881138f0 0x7fb58819f590 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb588076f70 0x7fb5881a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 --2- 172.21.15.52:0/2887264439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb588077890 0x7fb5881a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.715+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 >> 172.21.15.52:0/2887264439 conn(0x7fb588100740 msgr2=0x7fb588102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.716+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 shutdown_connections 2024-04-26T02:07:15.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:15.716+0000 7fb590400700 1 -- 172.21.15.52:0/2887264439 wait complete. 2024-04-26T02:07:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:15 smithi191 ceph-mon[32745]: pgmap v707: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 3.7 MiB/s wr, 147 op/s 2024-04-26T02:07:15.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:15 smithi183 ceph-mon[32752]: pgmap v707: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 3.7 MiB/s wr, 147 op/s 2024-04-26T02:07:15.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:15 smithi052 ceph-mon[27851]: pgmap v707: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 3.7 MiB/s wr, 147 op/s 2024-04-26T02:07:16.572 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 419, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_afa1933b0fdbb6c99c947d1eda34d661d23cd327/qa/tasks/quiescer.py", line 378, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error quiescing set 'f112abc': 110 (ETIMEDOUT) 2024-04-26T02:07:16.577 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-26T02:07:16.590 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T02:07:16.632 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-26T02:07:16.641 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-26T02:07:16.642 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T02:07:16.642 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T02:07:16.642 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-26T02:07:16.765 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-26T02:07:16.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 -- 172.21.15.52:0/3102078591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4188104200 msgr2=0x7f41881045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 --2- 172.21.15.52:0/3102078591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4188104200 0x7f41881045e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f417c009900 tx=0x7f417c02a3f0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 -- 172.21.15.52:0/3102078591 shutdown_connections 2024-04-26T02:07:17.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 --2- 172.21.15.52:0/3102078591 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41881092d0 0x7f4188110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 --2- 172.21.15.52:0/3102078591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4188104bb0 0x7f4188108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 --2- 172.21.15.52:0/3102078591 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4188104200 0x7f41881045e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 -- 172.21.15.52:0/3102078591 >> 172.21.15.52:0/3102078591 conn(0x7f41880ffa60 msgr2=0x7f4188101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 -- 172.21.15.52:0/3102078591 shutdown_connections 2024-04-26T02:07:17.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.060+0000 7f418ee00640 1 -- 172.21.15.52:0/3102078591 wait complete. 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 Processor -- start 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 -- start start 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 0x7f41881a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f418819a660 0x7f418819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.061+0000 7f418ee00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4188114280 con 0x7f4188104200 2024-04-26T02:07:17.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418ee00640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4188114100 con 0x7f41881092d0 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4188114400 con 0x7f418819a660 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 0x7f41881a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418d000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45044/0 (socket says 172.21.15.52:45044) 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418d000640 1 -- 172.21.15.52:0/4209183349 learned_addr learned my addr 172.21.15.52:0/4209183349 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418e400640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f418819a660 0x7f418819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 -- 172.21.15.52:0/4209183349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f418819a660 msgr2=0x7f418819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f418819a660 0x7f418819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 -- 172.21.15.52:0/4209183349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 msgr2=0x7f41881a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418da00640 1 -- 172.21.15.52:0/4209183349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418819b180 con 0x7f4188104200 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.062+0000 7f418d000640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f418da00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 0x7f41881a0460 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f417c02a9e0 tx=0x7f417c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c03f070 con 0x7f4188104200 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f417c031b50 con 0x7f4188104200 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c035030 con 0x7f4188104200 2024-04-26T02:07:17.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f418819b410 con 0x7f4188104200 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.063+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41881a7560 con 0x7f4188104200 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.064+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4188050530 con 0x7f4188104200 2024-04-26T02:07:17.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.067+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f417c042070 con 0x7f4188104200 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.068+0000 7f4186000640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 0x7f41700a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.068+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f417c0f0ff0 con 0x7f4188104200 2024-04-26T02:07:17.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.068+0000 7f418d000640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 0x7f41700a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.069+0000 7f418d000640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 0x7f41700a6220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41880fff80 tx=0x7f4174005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:17.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.071+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f417c0bd670 con 0x7f4188104200 2024-04-26T02:07:17.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.304+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f4188104c20 con 0x7f41700a3d60 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.328+0000 7f4186000640 1 -- 172.21.15.52:0/4209183349 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f4188104c20 con 0x7f41700a3d60 2024-04-26T02:07:17.327 INFO:teuthology.orchestra.run.smithi052.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T01:45:00", "first": "2024-04-26T01:46:00", "last": "2024-04-26T02:07:00", "last_pruned": "2024-04-26T02:07:03", "created_count": 22, "pruned_count": 15, "active": true} 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 msgr2=0x7f41700a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 0x7f41700a6220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41880fff80 tx=0x7f4174005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:17.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 msgr2=0x7f41881a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 0x7f41881a0460 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f417c02a9e0 tx=0x7f417c003b20 comp rx=0 tx=0).stop 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 shutdown_connections 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f41700a3d60 0x7f41700a6220 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f418819a660 0x7f418819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f41881092d0 0x7f41881a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 --2- 172.21.15.52:0/4209183349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4188104200 0x7f41881a0460 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 >> 172.21.15.52:0/4209183349 conn(0x7f41880ffa60 msgr2=0x7f4188100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:17.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 shutdown_connections 2024-04-26T02:07:17.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.331+0000 7f418ee00640 1 -- 172.21.15.52:0/4209183349 wait complete. 2024-04-26T02:07:17.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.552+0000 7f396b200640 1 -- 172.21.15.52:0/3949401403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1092c0 msgr2=0x7f396c110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.552+0000 7f396b200640 1 --2- 172.21.15.52:0/3949401403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1092c0 0x7f396c110e90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3964009ca0 tx=0x7f396402a8c0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 -- 172.21.15.52:0/3949401403 shutdown_connections 2024-04-26T02:07:17.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 --2- 172.21.15.52:0/3949401403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1092c0 0x7f396c110e90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 --2- 172.21.15.52:0/3949401403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c104ba0 0x7f396c108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 --2- 172.21.15.52:0/3949401403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1041f0 0x7f396c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 -- 172.21.15.52:0/3949401403 >> 172.21.15.52:0/3949401403 conn(0x7f396c0ffa30 msgr2=0x7f396c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 -- 172.21.15.52:0/3949401403 shutdown_connections 2024-04-26T02:07:17.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.553+0000 7f396b200640 1 -- 172.21.15.52:0/3949401403 wait complete. 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 Processor -- start 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 -- start start 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 0x7f396c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 0x7f396c19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114280 con 0x7f396c1092c0 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114100 con 0x7f396c19f010 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.554+0000 7f396b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114400 con 0x7f396c1041f0 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 0x7f396c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50916/0 (socket says 172.21.15.52:50916) 2024-04-26T02:07:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 -- 172.21.15.52:0/2971643662 learned_addr learned my addr 172.21.15.52:0/2971643662 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f396a800640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 0x7f396c19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 -- 172.21.15.52:0/2971643662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 msgr2=0x7f396c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 0x7f396c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 -- 172.21.15.52:0/2971643662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 msgr2=0x7f396c19f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 0x7f396c19f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 -- 172.21.15.52:0/2971643662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c19fb00 con 0x7f396c1092c0 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969e00640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 0x7f396c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f396a800640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 0x7f396c19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.555+0000 7f3969400640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9f0 tx=0x7f395800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.556+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395800ccc0 con 0x7f396c1092c0 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.556+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39580054d0 con 0x7f396c1092c0 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.556+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958010430 con 0x7f396c1092c0 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.556+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c19fdf0 con 0x7f396c1092c0 2024-04-26T02:07:17.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.556+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1a7590 con 0x7f396c1092c0 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.557+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c050530 con 0x7f396c1092c0 2024-04-26T02:07:17.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.560+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f39580046c0 con 0x7f396c1092c0 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.561+0000 7f3962a00640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 0x7f39500a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:17.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.562+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f39580d0ad0 con 0x7f396c1092c0 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.562+0000 7f3969e00640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 0x7f39500a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:17.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.562+0000 7f3969e00640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 0x7f39500a6250 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f395c005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:17.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.566+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f395809d110 con 0x7f396c1092c0 2024-04-26T02:07:17.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.797+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 --> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f396c104ba0 con 0x7f39500a3d90 2024-04-26T02:07:17.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.821+0000 7f3962a00640 1 -- 172.21.15.52:0/2971643662 <== mgr.25039 v2:172.21.15.52:6800/1185318772 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f396c104ba0 con 0x7f39500a3d90 2024-04-26T02:07:17.820 INFO:teuthology.orchestra.run.smithi052.stdout:/ 1m 6m3h 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.823+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 msgr2=0x7f39500a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.823+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 0x7f39500a6250 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f395c005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.823+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 msgr2=0x7f396c19ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.823+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9f0 tx=0x7f395800eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 shutdown_connections 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f39500a3d90 0x7f39500a6250 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c19f010 0x7f396c19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f396c1092c0 0x7f396c19ace0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 --2- 172.21.15.52:0/2971643662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f396c1041f0 0x7f396c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 >> 172.21.15.52:0/2971643662 conn(0x7f396c0ffa30 msgr2=0x7f396c101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 shutdown_connections 2024-04-26T02:07:17.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:17.824+0000 7f396b200640 1 -- 172.21.15.52:0/2971643662 wait complete. 2024-04-26T02:07:17.844 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-26T02:07:17.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T02:07:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:17 smithi191 ceph-mon[32745]: pgmap v708: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 5.1 MiB/s wr, 155 op/s 2024-04-26T02:07:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:17 smithi183 ceph-mon[32752]: pgmap v708: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 5.1 MiB/s wr, 155 op/s 2024-04-26T02:07:17.933 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:17 smithi052 ceph-mon[27851]: pgmap v708: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 5.1 MiB/s wr, 155 op/s 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.085+0000 7f57a2000640 1 -- 172.21.15.52:0/2346460244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c109460 msgr2=0x7f579c111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.085+0000 7f57a2000640 1 --2- 172.21.15.52:0/2346460244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c109460 0x7f579c111030 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5794009ca0 tx=0x7f579402a8c0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 -- 172.21.15.52:0/2346460244 shutdown_connections 2024-04-26T02:07:18.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 --2- 172.21.15.52:0/2346460244 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c109460 0x7f579c111030 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 --2- 172.21.15.52:0/2346460244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104d40 0x7f579c108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 --2- 172.21.15.52:0/2346460244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c104390 0x7f579c104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 -- 172.21.15.52:0/2346460244 >> 172.21.15.52:0/2346460244 conn(0x7f579c0ffc10 msgr2=0x7f579c102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 -- 172.21.15.52:0/2346460244 shutdown_connections 2024-04-26T02:07:18.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.086+0000 7f57a2000640 1 -- 172.21.15.52:0/2346460244 wait complete. 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 Processor -- start 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 -- start start 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104390 0x7f579c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c109460 0x7f579c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f579c114400 con 0x7f579c19a780 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f579c114280 con 0x7f579c109460 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.087+0000 7f57a2000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f579c114580 con 0x7f579c104390 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50944/0 (socket says 172.21.15.52:50944) 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 -- 172.21.15.52:0/633642564 learned_addr learned my addr 172.21.15.52:0/633642564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:18.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579aa00640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c109460 0x7f579c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579b400640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104390 0x7f579c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 -- 172.21.15.52:0/633642564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104390 msgr2=0x7f579c1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104390 0x7f579c1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 -- 172.21.15.52:0/633642564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c109460 msgr2=0x7f579c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c109460 0x7f579c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.088+0000 7f579be00640 1 -- 172.21.15.52:0/633642564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579c19b380 con 0x7f579c19a780 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f579be00640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f579402f540 tx=0x7f57940316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:18.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5794003b70 con 0x7f579c19a780 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5794003d10 con 0x7f579c19a780 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579402f990 con 0x7f579c19a780 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579c19b610 con 0x7f579c19a780 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.089+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f579c1a7710 con 0x7f579c19a780 2024-04-26T02:07:18.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.090+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f579c105860 con 0x7f579c19a780 2024-04-26T02:07:18.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.092+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5794004b40 con 0x7f579c19a780 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.094+0000 7f5793e00640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 0x7f57800a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.094+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f57940f1bd0 con 0x7f579c19a780 2024-04-26T02:07:18.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.094+0000 7f579b400640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 0x7f57800a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.095+0000 7f579b400640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 0x7f57800a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f578c009bd0 tx=0x7f578c005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:18.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.096+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57940be320 con 0x7f579c19a780 2024-04-26T02:07:18.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.318+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f579c104d40 con 0x7f579c19a780 2024-04-26T02:07:18.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.322+0000 7f5793e00640 1 -- 172.21.15.52:0/633642564 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+23138 (secure 0 0 0) 0x7f57940bdc20 con 0x7f579c19a780 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:07:18.322 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":132,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T02:07:13.517831+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":38,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/1613611936":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/2463815619":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/243168308":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/2167000831":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":38,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:07:18.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 msgr2=0x7f57800a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 0x7f57800a6260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f578c009bd0 tx=0x7f578c005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 msgr2=0x7f579c19ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f579402f540 tx=0x7f57940316f0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 shutdown_connections 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f57800a3da0 0x7f57800a6260 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f579c19a780 0x7f579c19ac00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f579c109460 0x7f579c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 --2- 172.21.15.52:0/633642564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f579c104390 0x7f579c1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.325+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 >> 172.21.15.52:0/633642564 conn(0x7f579c0ffc10 msgr2=0x7f579c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.326+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 shutdown_connections 2024-04-26T02:07:18.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.326+0000 7f57a2000640 1 -- 172.21.15.52:0/633642564 wait complete. 2024-04-26T02:07:18.347 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T01:35:33.675964+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T01:40:57.761799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T01:41:02.168712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 38, 'snap_epoch': 131, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T02:07:18.347 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.554+0000 7fb3a7400640 1 -- 172.21.15.52:0/3432880270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 msgr2=0x7fb3a0111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.554+0000 7fb3a7400640 1 --2- 172.21.15.52:0/3432880270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a0111040 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb398009ca0 tx=0x7fb39802a8c0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 -- 172.21.15.52:0/3432880270 shutdown_connections 2024-04-26T02:07:18.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 --2- 172.21.15.52:0/3432880270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a0111040 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 --2- 172.21.15.52:0/3432880270 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a0104d50 0x7fb3a0108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 --2- 172.21.15.52:0/3432880270 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a01043a0 0x7fb3a0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 -- 172.21.15.52:0/3432880270 >> 172.21.15.52:0/3432880270 conn(0x7fb3a00ffbc0 msgr2=0x7fb3a0101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 -- 172.21.15.52:0/3432880270 shutdown_connections 2024-04-26T02:07:18.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.555+0000 7fb3a7400640 1 -- 172.21.15.52:0/3432880270 wait complete. 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 Processor -- start 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 -- start start 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a01043a0 0x7fb3a01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a0114410 con 0x7fb3a01043a0 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a0114290 con 0x7fb3a019a750 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.556+0000 7fb3a7400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a0114590 con 0x7fb3a0109470 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a4a00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a01043a0 0x7fb3a01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb39fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:45104/0 (socket says 172.21.15.52:45104) 2024-04-26T02:07:18.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 -- 172.21.15.52:0/2488212243 learned_addr learned my addr 172.21.15.52:0/2488212243 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 -- 172.21.15.52:0/2488212243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 msgr2=0x7fb3a01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 -- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a01043a0 msgr2=0x7fb3a01a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a01043a0 0x7fb3a01a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 -- 172.21.15.52:0/2488212243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a019b350 con 0x7fb3a019a750 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.557+0000 7fb3a5400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb39802f540 tx=0x7fb3980316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.558+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398003b70 con 0x7fb3a019a750 2024-04-26T02:07:18.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.558+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb398003d10 con 0x7fb3a019a750 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.558+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39802f990 con 0x7fb3a019a750 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.558+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a019b5e0 con 0x7fb3a019a750 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.558+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a01a7730 con 0x7fb3a019a750 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.559+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a0105870 con 0x7fb3a019a750 2024-04-26T02:07:18.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.561+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb39802fb30 con 0x7fb3a019a750 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.563+0000 7fb39d600640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 0x7fb3880a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.563+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fb3980f1a80 con 0x7fb3a019a750 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.563+0000 7fb3a4a00640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 0x7fb3880a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:18.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.564+0000 7fb3a4a00640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 0x7fb3880a6260 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3940095f0 tx=0x7fb394005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:18.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.567+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3980f3e20 con 0x7fb3a019a750 2024-04-26T02:07:18.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.780+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3a0104d50 con 0x7fb3a019a750 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.781+0000 7fb39d600640 1 -- 172.21.15.52:0/2488212243 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) ==== 93+0+10 (secure 0 0 0) 0x7fb3980be1d0 con 0x7fb3a019a750 2024-04-26T02:07:18.780 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 msgr2=0x7fb3880a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 0x7fb3880a6260 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3940095f0 tx=0x7fb394005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 msgr2=0x7fb3a019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb39802f540 tx=0x7fb3980316f0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 shutdown_connections 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fb3880a3da0 0x7fb3880a6260 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb3a019a750 0x7fb3a019abd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3a0109470 0x7fb3a01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 --2- 172.21.15.52:0/2488212243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb3a01043a0 0x7fb3a01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 >> 172.21.15.52:0/2488212243 conn(0x7fb3a00ffbc0 msgr2=0x7fb3a0101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 shutdown_connections 2024-04-26T02:07:18.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:18.784+0000 7fb3a7400640 1 -- 172.21.15.52:0/2488212243 wait complete. 2024-04-26T02:07:18.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T02:07:18.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:18 smithi183 ceph-mon[32752]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:18.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:18 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/633642564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:18.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:18 smithi052 ceph-mon[27851]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:18.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:18 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/633642564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:19.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 -- 172.21.15.52:0/1348378202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc107140 msgr2=0x7f23cc10ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 --2- 172.21.15.52:0/1348378202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc107140 0x7f23cc10ed10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009900 tx=0x7f23c402a410 comp rx=0 tx=0).stop 2024-04-26T02:07:19.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 -- 172.21.15.52:0/1348378202 shutdown_connections 2024-04-26T02:07:19.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 --2- 172.21.15.52:0/1348378202 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc107140 0x7f23cc10ed10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 --2- 172.21.15.52:0/1348378202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc102a20 0x7f23cc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 --2- 172.21.15.52:0/1348378202 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc102070 0x7f23cc102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 -- 172.21.15.52:0/1348378202 >> 172.21.15.52:0/1348378202 conn(0x7f23cc0fd8f0 msgr2=0x7f23cc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:19.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 -- 172.21.15.52:0/1348378202 shutdown_connections 2024-04-26T02:07:19.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.018+0000 7f23cb200640 1 -- 172.21.15.52:0/1348378202 wait complete. 2024-04-26T02:07:19.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.019+0000 7f23cb200640 1 Processor -- start 2024-04-26T02:07:19.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.019+0000 7f23cb200640 1 -- start start 2024-04-26T02:07:19.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc107140 0x7f23cc198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc19cec0 0x7f23cc19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc112100 con 0x7f23cc102070 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc111f80 con 0x7f23cc107140 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23cb200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc112280 con 0x7f23cc19cec0 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50974/0 (socket says 172.21.15.52:50974) 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 -- 172.21.15.52:0/4214483258 learned_addr learned my addr 172.21.15.52:0/4214483258 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9400640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc107140 0x7f23cc198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23ca800640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc19cec0 0x7f23cc19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 -- 172.21.15.52:0/4214483258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc19cec0 msgr2=0x7f23cc19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc19cec0 0x7f23cc19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 -- 172.21.15.52:0/4214483258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc107140 msgr2=0x7f23cc198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc107140 0x7f23cc198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.020+0000 7f23c9e00640 1 -- 172.21.15.52:0/4214483258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23cc19d9b0 con 0x7f23cc102070 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23c9e00640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00e9f0 tx=0x7f23bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc00cd80 con 0x7f23cc102070 2024-04-26T02:07:19.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23cc19dca0 con 0x7f23cc102070 2024-04-26T02:07:19.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23bc0108e0 con 0x7f23cc102070 2024-04-26T02:07:19.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc018980 con 0x7f23cc102070 2024-04-26T02:07:19.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.021+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23cc1a5490 con 0x7f23cc102070 2024-04-26T02:07:19.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.022+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23cc0690e0 con 0x7f23cc102070 2024-04-26T02:07:19.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.024+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f23bc010430 con 0x7f23cc102070 2024-04-26T02:07:19.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.026+0000 7f23c2a00640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 0x7f23ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.026+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f23bc0d0b40 con 0x7f23cc102070 2024-04-26T02:07:19.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.026+0000 7f23c9400640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 0x7f23ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.027+0000 7f23c9400640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 0x7f23ac0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23cc100ff0 tx=0x7f23b4009500 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.036+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23bc09cb30 con 0x7f23cc102070 2024-04-26T02:07:19.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:18 smithi191 ceph-mon[32745]: from='client.17355 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:19.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:18 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/633642564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:07:19.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.246+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f23cc10a3d0 con 0x7f23cc102070 2024-04-26T02:07:19.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.247+0000 7f23c2a00640 1 -- 172.21.15.52:0/4214483258 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) ==== 104+0+11 (secure 0 0 0) 0x7f23bc0a0e30 con 0x7f23cc102070 2024-04-26T02:07:19.246 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2024-04-26T02:07:19.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 msgr2=0x7f23ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 0x7f23ac0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23cc100ff0 tx=0x7f23b4009500 comp rx=0 tx=0).stop 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 msgr2=0x7f23cc198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00e9f0 tx=0x7f23bc00eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 shutdown_connections 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f23ac0a3d20 0x7f23ac0a61e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23cc19cec0 0x7f23cc19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f23cc107140 0x7f23cc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 --2- 172.21.15.52:0/4214483258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23cc102070 0x7f23cc198600 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.250+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 >> 172.21.15.52:0/4214483258 conn(0x7f23cc0fd8f0 msgr2=0x7f23cc0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:19.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.251+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 shutdown_connections 2024-04-26T02:07:19.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.251+0000 7f23cb200640 1 -- 172.21.15.52:0/4214483258 wait complete. 2024-04-26T02:07:19.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T02:07:19.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 -- 172.21.15.52:0/4149762606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd0108e90 msgr2=0x7f9dd01152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 --2- 172.21.15.52:0/4149762606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd0108e90 0x7f9dd01152b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009ca0 tx=0x7f9dc802a950 comp rx=0 tx=0).stop 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 -- 172.21.15.52:0/4149762606 shutdown_connections 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 --2- 172.21.15.52:0/4149762606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd0108e90 0x7f9dd01152b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 --2- 172.21.15.52:0/4149762606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd01084f0 0x7f9dd0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 --2- 172.21.15.52:0/4149762606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 0x7f9dd010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 -- 172.21.15.52:0/4149762606 >> 172.21.15.52:0/4149762606 conn(0x7f9dd0077d70 msgr2=0x7f9dd0106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:19.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 -- 172.21.15.52:0/4149762606 shutdown_connections 2024-04-26T02:07:19.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.488+0000 7f9dd5000640 1 -- 172.21.15.52:0/4149762606 wait complete. 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.489+0000 7f9dd5000640 1 Processor -- start 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 -- start start 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd0108e90 0x7f9dd01a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 0x7f9dd019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd0118690 con 0x7f9dd010d3c0 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd0118510 con 0x7f9dd0108e90 2024-04-26T02:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dd5000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd0118810 con 0x7f9dd01084f0 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dcf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dcf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:60716/0 (socket says 172.21.15.52:60716) 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dcf400640 1 -- 172.21.15.52:0/3495828890 learned_addr learned my addr 172.21.15.52:0/3495828890 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dc7e00640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd0108e90 0x7f9dd01a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.490+0000 7f9dcfe00640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 0x7f9dd019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 -- 172.21.15.52:0/3495828890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd0108e90 msgr2=0x7f9dd01a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd0108e90 0x7f9dd01a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 -- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 msgr2=0x7f9dd019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 0x7f9dd019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 -- 172.21.15.52:0/3495828890 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd019f190 con 0x7f9dd01084f0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dcf400640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000e9f0 tx=0x7f9dc000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000cd80 con 0x7f9dd01084f0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dc00108e0 con 0x7f9dd01084f0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd019f480 con 0x7f9dd01084f0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.491+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0018980 con 0x7f9dd01084f0 2024-04-26T02:07:19.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.492+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd01ab6d0 con 0x7f9dd01084f0 2024-04-26T02:07:19.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.493+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd00690e0 con 0x7f9dd01084f0 2024-04-26T02:07:19.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.495+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9dc0010430 con 0x7f9dd01084f0 2024-04-26T02:07:19.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.496+0000 7f9dccc00640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 0x7f9db40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.496+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f9dc00d0ed0 con 0x7f9dd01084f0 2024-04-26T02:07:19.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.497+0000 7f9dc7e00640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 0x7f9db40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.497+0000 7f9dc7e00640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 0x7f9db40a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0107fd0 tx=0x7f9db8005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.499+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dc009cec0 con 0x7f9dd01084f0 2024-04-26T02:07:19.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.711+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9dd0110960 con 0x7f9dd01084f0 2024-04-26T02:07:19.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.712+0000 7f9dccc00640 1 -- 172.21.15.52:0/3495828890 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) ==== 100+0+11 (secure 0 0 0) 0x7f9dc00a11c0 con 0x7f9dd01084f0 2024-04-26T02:07:19.711 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 64 2024-04-26T02:07:19.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.714+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 msgr2=0x7f9db40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.714+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 0x7f9db40a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0107fd0 tx=0x7f9db8005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:19.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.714+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 msgr2=0x7f9dd01a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.714+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000e9f0 tx=0x7f9dc000eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 shutdown_connections 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f9db40a3da0 0x7f9db40a6260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd010d3c0 0x7f9dd019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9dd0108e90 0x7f9dd01a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 --2- 172.21.15.52:0/3495828890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dd01084f0 0x7f9dd01a4810 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 >> 172.21.15.52:0/3495828890 conn(0x7f9dd0077d70 msgr2=0x7f9dd00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 shutdown_connections 2024-04-26T02:07:19.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.715+0000 7f9dd5000640 1 -- 172.21.15.52:0/3495828890 wait complete. 2024-04-26T02:07:19.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-26T02:07:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:19 smithi183 ceph-mon[32752]: from='client.17361 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:19.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:19 smithi183 ceph-mon[32752]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:19.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:19 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2488212243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:19.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:19 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/4214483258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:19.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 -- 172.21.15.52:0/2380766313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 msgr2=0x7fdd60115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 --2- 172.21.15.52:0/2380766313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd60115200 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009ca0 tx=0x7fdd5802a940 comp rx=0 tx=0).stop 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 -- 172.21.15.52:0/2380766313 shutdown_connections 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 --2- 172.21.15.52:0/2380766313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd60115200 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 --2- 172.21.15.52:0/2380766313 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd60108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 --2- 172.21.15.52:0/2380766313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 0x7fdd6010e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 -- 172.21.15.52:0/2380766313 >> 172.21.15.52:0/2380766313 conn(0x7fdd60077ad0 msgr2=0x7fdd60079ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 -- 172.21.15.52:0/2380766313 shutdown_connections 2024-04-26T02:07:19.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.951+0000 7fdd65200640 1 -- 172.21.15.52:0/2380766313 wait complete. 2024-04-26T02:07:19.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.952+0000 7fdd65200640 1 Processor -- start 2024-04-26T02:07:19.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 -- start start 2024-04-26T02:07:19.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd601a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 0x7fdd6019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdd60118640 con 0x7fdd60108d90 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd601184c0 con 0x7fdd6010e460 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd65200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdd601187c0 con 0x7fdd601083f0 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd5ea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd5f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd601a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd5ea00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51016/0 (socket says 172.21.15.52:51016) 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd5ea00640 1 -- 172.21.15.52:0/1061363041 learned_addr learned my addr 172.21.15.52:0/1061363041 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.953+0000 7fdd5fe00640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 0x7fdd6019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 -- 172.21.15.52:0/1061363041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 msgr2=0x7fdd6019e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 0x7fdd6019e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 -- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 msgr2=0x7fdd601a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 -- 172.21.15.52:0/1061363041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6019f080 con 0x7fdd601083f0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5ea00640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd5f400640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd601a4700 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000e9f0 tx=0x7fdd5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5000ccc0 con 0x7fdd601083f0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd500108e0 con 0x7fdd601083f0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50018880 con 0x7fdd601083f0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.954+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6019f370 con 0x7fdd601083f0 2024-04-26T02:07:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.955+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd601ab5c0 con 0x7fdd601083f0 2024-04-26T02:07:19.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.956+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd600690e0 con 0x7fdd601083f0 2024-04-26T02:07:19.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.958+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdd50010430 con 0x7fdd601083f0 2024-04-26T02:07:19.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.959+0000 7fdd57e00640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 0x7fdd440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:07:19.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.959+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7fdd500d0ff0 con 0x7fdd601083f0 2024-04-26T02:07:19.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.960+0000 7fdd5ea00640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 0x7fdd440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:07:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.960+0000 7fdd5ea00640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 0x7fdd440a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdd6010da60 tx=0x7fdd4c005f90 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:07:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:19.962+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd5009d020 con 0x7fdd601083f0 2024-04-26T02:07:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:19 smithi052 ceph-mon[27851]: from='client.17361 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:19 smithi052 ceph-mon[27851]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:19 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2488212243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:19.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:19 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/4214483258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:20.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:19 smithi191 ceph-mon[32745]: from='client.17361 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T02:07:20.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:19 smithi191 ceph-mon[32745]: pgmap v709: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:20.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:19 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2488212243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T02:07:20.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:19 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/4214483258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T02:07:20.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.223+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fdd601102c0 con 0x7fdd601083f0 2024-04-26T02:07:20.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.234+0000 7fdd57e00640 1 -- 172.21.15.52:0/1061363041 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746063 (secure 0 0 0) 0x7fdd500a1320 con 0x7fdd601083f0 2024-04-26T02:07:20.245 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:07:20.245 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 27, 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "active_gid": 25039, 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "active_name": "x", 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "active_addrs": { 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.246 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6800", 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1185318772 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1185318772 2024-04-26T02:07:20.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "active_addr": "172.21.15.52:6801/1185318772", 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "active_change": "2024-04-26T01:44:59.819415+0000", 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "active_mgr_features": 4540701547738038271, 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "available": true, 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "standbys": [ 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.248 INFO:teuthology.orchestra.run.smithi052.stdout: "gid": 25025, 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "y", 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_features": 4540701547738038271, 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.249 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.250 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.251 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.252 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.253 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.254 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.255 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.257 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.258 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.259 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.260 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.261 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.263 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.264 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.265 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-04-26T02:07:20.266 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:20.267 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.268 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.269 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.270 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-04-26T02:07:20.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:20.272 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.273 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-04-26T02:07:20.275 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:20.276 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.277 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:20.279 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.280 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.281 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:20.282 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-04-26T02:07:20.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:20.284 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-04-26T02:07:20.285 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-04-26T02:07:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:20.287 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.288 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.290 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.291 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.292 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.293 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.295 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.296 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.298 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.299 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.300 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:20.302 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.303 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.304 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.305 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.306 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:20.307 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.309 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-04-26T02:07:20.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.311 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.312 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-04-26T02:07:20.313 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:20.314 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.315 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-04-26T02:07:20.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-04-26T02:07:20.317 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-04-26T02:07:20.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:20.319 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.320 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-04-26T02:07:20.321 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.322 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.325 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:20.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.327 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:20.328 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-04-26T02:07:20.329 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.330 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.331 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.333 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:20.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:20.335 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.336 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-04-26T02:07:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.338 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.339 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.340 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.341 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:20.342 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.343 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-04-26T02:07:20.345 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:20.346 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.347 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.348 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.350 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.351 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.352 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-04-26T02:07:20.353 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.354 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.355 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:20.356 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.358 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:20.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:20.360 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.361 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.365 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.366 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.367 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.368 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.369 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.370 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.371 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.372 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.373 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.374 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.375 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.376 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.377 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.378 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.379 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.380 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.381 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.382 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-04-26T02:07:20.383 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.385 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.386 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.387 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.388 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-04-26T02:07:20.389 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.390 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.391 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-04-26T02:07:20.392 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:20.393 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.394 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.396 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.397 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.398 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.399 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.400 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.401 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.402 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:20.403 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:20.404 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.405 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.407 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.408 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.409 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.410 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.411 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.412 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.413 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-04-26T02:07:20.415 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:20.416 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.417 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:20.419 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.420 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.421 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:20.422 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.423 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.424 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:20.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.426 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.427 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.429 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.430 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.433 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.434 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.435 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.436 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.437 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.438 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.439 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.440 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.441 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.442 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.443 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.444 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.445 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.446 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.448 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.449 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.450 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.451 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.452 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.453 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-04-26T02:07:20.454 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:20.455 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.456 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.458 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.459 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.460 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.461 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.462 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:20.463 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.464 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.465 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:20.466 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.467 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.468 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:20.469 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.470 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.471 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:20.472 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.473 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.474 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.475 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.476 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.477 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.478 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.479 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.480 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.481 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.482 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.483 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.485 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:20.486 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.488 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.489 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.490 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.491 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.493 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-04-26T02:07:20.494 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.495 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.496 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.498 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.499 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.500 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.501 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.502 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.503 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.504 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.505 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.507 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:20.508 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.509 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.510 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.512 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.513 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.515 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.516 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.518 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.519 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.520 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.521 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.522 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.523 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.525 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.528 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.530 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-04-26T02:07:20.531 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.532 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.533 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.534 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.536 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.537 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.553 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.554 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:20.555 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.556 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.557 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.558 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.559 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.560 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.561 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.562 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.563 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.566 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.567 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.568 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-04-26T02:07:20.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:20.570 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.571 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:20.573 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.576 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.577 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.578 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.579 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.580 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.582 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.584 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.585 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.586 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.588 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.589 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.590 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.591 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.592 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.593 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.594 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.595 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.596 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.597 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-04-26T02:07:20.598 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:20.599 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.601 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.602 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.603 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.604 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.605 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.606 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.607 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.608 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-04-26T02:07:20.609 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.611 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.613 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.614 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.615 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.617 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-04-26T02:07:20.618 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:20.619 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-04-26T02:07:20.620 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.621 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.622 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.623 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.624 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.625 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.627 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.628 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.630 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.631 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.632 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.633 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.634 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.635 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.636 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.637 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.638 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.639 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.640 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.641 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.642 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.643 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.645 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.646 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.647 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.649 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.650 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.651 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.652 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.653 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.654 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.655 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.656 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.657 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.658 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.659 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.660 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.662 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:20.663 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-04-26T02:07:20.664 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.665 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.666 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.667 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.668 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.669 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.670 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.671 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.672 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.673 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-04-26T02:07:20.674 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.675 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.676 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.677 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.679 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.680 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.681 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.682 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.683 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.684 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.686 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.687 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.689 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.690 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.691 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.692 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.693 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:20.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.696 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.697 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.698 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.699 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.700 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.701 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.702 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.703 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.704 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.705 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.707 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.708 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:20.709 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.710 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.711 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:20.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:20.713 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.714 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-04-26T02:07:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-04-26T02:07:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.735 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.736 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.737 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.738 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.740 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.741 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.742 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.743 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.744 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.745 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.747 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.749 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.750 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.751 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.752 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.753 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.755 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.756 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.757 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.758 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.759 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.760 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.761 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.762 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.764 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.765 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.766 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.767 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.768 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.769 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.771 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.773 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.774 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.775 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.777 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.778 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.782 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.783 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.784 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.785 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.786 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.787 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.788 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.789 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.790 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.791 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.792 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.793 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-04-26T02:07:20.794 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.795 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.796 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.797 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.798 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.799 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.800 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.801 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.802 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-04-26T02:07:20.803 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.804 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.805 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-04-26T02:07:20.806 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.807 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.810 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.811 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.812 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.813 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.814 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.815 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.816 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.818 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.820 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.822 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.823 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.824 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.825 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.826 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.827 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.828 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.831 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.832 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.834 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.835 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.836 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.837 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-04-26T02:07:20.839 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.840 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.841 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.842 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.843 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.844 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.845 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.846 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.847 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.848 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.850 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:20.851 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.852 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-04-26T02:07:20.853 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.854 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.855 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.856 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.857 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.858 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.860 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.861 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.863 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.864 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-04-26T02:07:20.866 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-04-26T02:07:20.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.869 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.870 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.871 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.872 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.873 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.874 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.875 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.876 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.877 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-04-26T02:07:20.878 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.879 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.880 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.881 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.882 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.883 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.884 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.886 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.887 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.888 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.889 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.890 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-04-26T02:07:20.891 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.892 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.893 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.894 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.896 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.897 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.898 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.899 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.900 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.901 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.903 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.904 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.905 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.906 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.907 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.908 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.909 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.910 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.911 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.912 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.913 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.914 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.915 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.916 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.917 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.918 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.919 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-04-26T02:07:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-04-26T02:07:20.922 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.923 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.924 INFO:teuthology.orchestra.run.smithi052.stdout: "gid": 25036, 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "z", 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_features": 4540701547738038271, 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:20.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:20.926 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.927 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.928 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.929 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.931 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.932 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.933 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-04-26T02:07:20.934 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.935 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-04-26T02:07:20.936 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-04-26T02:07:20.937 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.938 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.939 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-04-26T02:07:20.940 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-04-26T02:07:20.941 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.942 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-04-26T02:07:20.943 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.944 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.945 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.946 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.947 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.948 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.949 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.950 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:20.952 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:20.953 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:20.955 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.956 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.958 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:20.959 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.960 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:20.964 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.965 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.966 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.967 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.968 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.969 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-04-26T02:07:20.970 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.971 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.972 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.973 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.974 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.976 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-04-26T02:07:20.977 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:20.978 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.979 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.980 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.981 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.983 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.986 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:20.987 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-04-26T02:07:20.988 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.989 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.990 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:20.991 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-04-26T02:07:20.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:20.993 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.994 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.996 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:20.997 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:20.998 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:20.999 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-04-26T02:07:21.000 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:21.001 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.002 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.004 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.005 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.006 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.007 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.008 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.009 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.010 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.011 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.012 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.013 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.014 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.015 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.016 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.017 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.018 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-04-26T02:07:21.019 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:21.020 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.021 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.023 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-04-26T02:07:21.024 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:21.025 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.026 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.028 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:21.029 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:21.030 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.031 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.032 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.033 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.034 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.035 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.036 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.038 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.039 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.040 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:21.042 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.043 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.044 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.045 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.046 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.047 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.048 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.049 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.050 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-04-26T02:07:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-04-26T02:07:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.067 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.068 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.069 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.070 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.071 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.072 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:21.073 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.074 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:21.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.076 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.077 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.079 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.080 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.082 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.083 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.086 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-04-26T02:07:21.087 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.088 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.089 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-04-26T02:07:21.090 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.091 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:21.092 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.093 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.094 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.095 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.096 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.098 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.099 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:21.100 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.101 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.102 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.103 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.104 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:21.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.106 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.107 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.108 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.109 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.110 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.111 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.112 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.113 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.114 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.115 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:21.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.117 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.118 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.119 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.120 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.121 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:21.122 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.123 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.124 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:21.125 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.126 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.127 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:21.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.129 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.130 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.131 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.132 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.135 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.136 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.137 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.138 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.139 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.140 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.141 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:21.142 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.143 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:21.144 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:21.145 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.146 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.147 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:21.148 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.150 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.152 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.153 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:21.154 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.155 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.156 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:21.157 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.158 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.159 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.160 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.162 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-04-26T02:07:21.163 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.164 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.165 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-04-26T02:07:21.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.167 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.168 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.170 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.172 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.173 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.174 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.175 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.176 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:21.178 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.180 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.181 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.182 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.183 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.184 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.185 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-04-26T02:07:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:21.188 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.189 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.190 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.191 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.192 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.194 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.195 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.196 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.197 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.198 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.199 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.200 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.201 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.202 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.203 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.204 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.206 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-04-26T02:07:21.207 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.208 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-04-26T02:07:21.210 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:21.211 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.212 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.213 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.214 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.215 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.216 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.217 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.218 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.219 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.220 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.221 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.222 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.223 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.224 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-04-26T02:07:21.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.226 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.227 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.228 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.229 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-04-26T02:07:21.230 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.231 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.233 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.235 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.236 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.237 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.238 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.239 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.240 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.241 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:21.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.243 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.244 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-04-26T02:07:21.245 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:21.246 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.247 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.248 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.250 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.252 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.253 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.254 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.255 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.256 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-04-26T02:07:21.257 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.258 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.259 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.260 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.261 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.262 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.263 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.265 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.266 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.267 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-04-26T02:07:21.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.270 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.271 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.272 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.273 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.274 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.275 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.278 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.279 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.280 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.280 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:21.280 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:21.281 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.281 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.281 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.283 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.284 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.285 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.286 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.287 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.288 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.289 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-04-26T02:07:21.290 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:21.291 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.292 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.293 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-04-26T02:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.295 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.296 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.297 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:21.298 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.299 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.300 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.301 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.302 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.303 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.305 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.306 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.307 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.309 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.310 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.311 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.312 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.313 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.314 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.315 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.316 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.318 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.319 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.320 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.321 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.322 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.323 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.324 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.325 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.327 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.328 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.329 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.330 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.331 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.332 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.333 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.334 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.337 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.338 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:21.339 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.340 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.341 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.342 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.343 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.344 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.345 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.346 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.347 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.348 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.349 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.350 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.351 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.352 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.353 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.354 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.355 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.356 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.357 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.359 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.360 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.361 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.362 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.364 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.365 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.366 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.367 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.368 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-04-26T02:07:21.370 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.372 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.373 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.374 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.375 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.376 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.377 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.378 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.379 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.380 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.381 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.382 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.383 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.384 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.385 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.386 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.387 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-04-26T02:07:21.388 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.389 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.390 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:21.391 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-04-26T02:07:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:21.406 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.407 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.408 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.409 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.410 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.411 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.412 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.413 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:21.414 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.415 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.416 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.417 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.418 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.420 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.421 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.422 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.423 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.424 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.425 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.426 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.427 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.428 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.431 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.432 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.433 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.434 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.436 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.437 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.438 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.439 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.440 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.441 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.442 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.443 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.444 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.445 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.446 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.447 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.448 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.449 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.450 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.451 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.467 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.468 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.469 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.470 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.471 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.472 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.473 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.474 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.475 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-04-26T02:07:21.476 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.477 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.478 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.479 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.480 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.482 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.485 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-04-26T02:07:21.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.487 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.489 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.490 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.491 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-04-26T02:07:21.492 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-04-26T02:07:21.493 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.494 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.495 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.497 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:21.499 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.500 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.501 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.503 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.504 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.505 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.506 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.507 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.508 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.509 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.510 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.512 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.513 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.514 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.515 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.516 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.517 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.518 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.519 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-04-26T02:07:21.520 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.521 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.522 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.523 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.524 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.525 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.526 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.527 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.528 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.529 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:21.530 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.531 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.532 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.533 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.534 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.536 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.537 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.538 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.539 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.540 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.541 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.542 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.543 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.544 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.545 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-04-26T02:07:21.546 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:21.547 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.548 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.550 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-04-26T02:07:21.551 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.552 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.553 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.555 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.556 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.557 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-04-26T02:07:21.558 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.559 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.560 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.561 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.563 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.564 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.565 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.566 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.567 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.568 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.569 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.570 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.571 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.573 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.574 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.575 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.576 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.577 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.578 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.579 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.581 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.582 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.583 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.586 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.587 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.588 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.589 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.590 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.591 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.592 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.593 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-04-26T02:07:21.594 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.596 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:21.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.598 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.599 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.600 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.601 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.602 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-04-26T02:07:21.603 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.605 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.606 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.607 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.608 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.609 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.610 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.611 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.612 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.613 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.614 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.615 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.616 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.617 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-04-26T02:07:21.618 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:21.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "modules": [ 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "dashboard", 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "iostat", 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "nfs", 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "restful", 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_schedule" 2024-04-26T02:07:21.621 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.622 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.623 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.624 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.625 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.626 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.627 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.628 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.629 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.630 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.631 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-04-26T02:07:21.632 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.633 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.634 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.635 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.636 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.637 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.638 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-04-26T02:07:21.639 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-04-26T02:07:21.640 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.641 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.643 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-04-26T02:07:21.644 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.646 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.647 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.648 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.649 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:21.650 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:21.651 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-04-26T02:07:21.652 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.653 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.654 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.656 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.657 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-04-26T02:07:21.658 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T02:07:21.659 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.660 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.661 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.662 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.663 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.664 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.665 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.666 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.667 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.668 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.669 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.671 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.673 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-04-26T02:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-04-26T02:07:21.689 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.690 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-04-26T02:07:21.691 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-04-26T02:07:21.692 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.693 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.694 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-04-26T02:07:21.695 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.696 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.697 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.699 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-04-26T02:07:21.700 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-04-26T02:07:21.701 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.702 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.703 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T02:07:21.704 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.705 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.706 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.707 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.708 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.709 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-04-26T02:07:21.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-04-26T02:07:21.711 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T02:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-04-26T02:07:21.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.714 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.715 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.716 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.717 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-04-26T02:07:21.718 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.719 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-04-26T02:07:21.721 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.722 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.723 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.724 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T02:07:21.725 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.726 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.727 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-04-26T02:07:21.728 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.729 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.730 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-04-26T02:07:21.731 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-04-26T02:07:21.732 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.733 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.734 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.735 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.737 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-04-26T02:07:21.738 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.740 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.741 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.742 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.743 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.744 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.745 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.746 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.747 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.748 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.749 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.750 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.751 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.752 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.754 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T02:07:21.755 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T02:07:21.756 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.758 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.759 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-04-26T02:07:21.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-04-26T02:07:21.761 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.762 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-04-26T02:07:21.763 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-04-26T02:07:21.764 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T02:07:21.765 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.766 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.768 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-04-26T02:07:21.769 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T02:07:21.770 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.771 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.773 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-04-26T02:07:21.774 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.775 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.776 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.777 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-04-26T02:07:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.780 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.783 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-04-26T02:07:21.784 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.785 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.786 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.789 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.790 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.791 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.792 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.793 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.794 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.795 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.796 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.797 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.799 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.801 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.802 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.803 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T02:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.805 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.806 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.808 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T02:07:21.809 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.810 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.811 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.813 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T02:07:21.814 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.815 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.816 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.818 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T02:07:21.819 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.820 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.821 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.823 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T02:07:21.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.825 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.826 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.827 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.828 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T02:07:21.829 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.830 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T02:07:21.831 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.832 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.833 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.835 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T02:07:21.836 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.837 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.838 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.840 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T02:07:21.841 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.843 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.844 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.845 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.846 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.847 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.848 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.849 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T02:07:21.850 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.851 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.852 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.853 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.854 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.855 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.856 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.857 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.858 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.859 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T02:07:21.860 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.861 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.862 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.863 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-04-26T02:07:21.864 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T02:07:21.865 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.866 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.867 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.868 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.869 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.870 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.872 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T02:07:21.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.874 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-04-26T02:07:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-04-26T02:07:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.907 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:21.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.909 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:21.910 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.912 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.914 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-04-26T02:07:21.915 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.916 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.917 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.919 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.920 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.921 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.923 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.924 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.925 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.926 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.927 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.928 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-04-26T02:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.930 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.931 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.932 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.934 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.935 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.937 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.938 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.939 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.940 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.941 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.942 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.944 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-04-26T02:07:21.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.946 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.948 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.949 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-04-26T02:07:21.951 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T02:07:21.952 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.953 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.954 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.955 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.956 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.957 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.958 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.959 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.960 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.961 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.962 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.963 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.964 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.965 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-04-26T02:07:21.966 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.967 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.968 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.971 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-04-26T02:07:21.972 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.973 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.975 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.976 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.977 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.978 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.979 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.982 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.983 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.984 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-04-26T02:07:21.985 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.986 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.987 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.988 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.990 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.992 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-04-26T02:07:21.994 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T02:07:21.995 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:21.996 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:21.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:21.998 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:21.999 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.000 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.001 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.002 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.003 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.004 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.006 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.007 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.008 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.009 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.010 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.011 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.012 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.013 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.015 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.016 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.017 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.018 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.019 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-04-26T02:07:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.032 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.033 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.034 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.035 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.036 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.037 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.038 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.039 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.040 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.042 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.043 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.044 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.045 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.046 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.048 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.049 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.050 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.051 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:22.052 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.053 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.055 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.056 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.059 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.060 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.061 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.063 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-04-26T02:07:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-04-26T02:07:22.065 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.066 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.067 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.069 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.070 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.071 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.072 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.073 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.076 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.077 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.078 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.095 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.098 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.099 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.100 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.101 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.102 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.103 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.104 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.105 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.106 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.108 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.109 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.111 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.113 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.117 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.118 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.119 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.120 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.121 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.122 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.123 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.125 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.127 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:22.128 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.130 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.132 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-04-26T02:07:22.133 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.134 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.135 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-04-26T02:07:22.136 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-04-26T02:07:22.137 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.138 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-04-26T02:07:22.139 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.140 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.141 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.143 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.144 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.145 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.147 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-04-26T02:07:22.148 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.150 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.152 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-04-26T02:07:22.154 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.155 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.156 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.158 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.159 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.160 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.161 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.162 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.163 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.164 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.165 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.166 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.168 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.170 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.171 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.173 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.174 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.175 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.176 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.179 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.180 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.181 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.182 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.184 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.185 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.186 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.188 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.190 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.191 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.192 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.193 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.194 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.195 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.196 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.197 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-04-26T02:07:22.198 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.199 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.201 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.202 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.203 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.204 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.205 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.207 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.208 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.209 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.210 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-04-26T02:07:22.212 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-04-26T02:07:22.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.214 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.215 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.216 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.217 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-04-26T02:07:22.219 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T02:07:22.220 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.221 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.222 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.223 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.224 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.226 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.227 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.228 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.230 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.232 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.233 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.235 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.236 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.237 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.238 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.239 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.240 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.242 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.244 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.246 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.247 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.248 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.249 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.250 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.251 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:22.252 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.253 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.254 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.255 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.256 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.257 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.258 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.259 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.260 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.261 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.262 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.263 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.264 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.265 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-04-26T02:07:22.266 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.267 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.268 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.269 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.270 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.271 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.272 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.273 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-04-26T02:07:22.274 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.275 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.276 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.277 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-04-26T02:07:22.278 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-04-26T02:07:22.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.280 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.281 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.282 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.283 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.284 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.285 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.286 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.287 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.289 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.290 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.291 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.293 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.294 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.295 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.296 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.299 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.300 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.301 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.302 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.303 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.304 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.306 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.308 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.309 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.311 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.313 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.314 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.315 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-04-26T02:07:22.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.319 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.320 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.321 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.322 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-04-26T02:07:22.324 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.325 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.326 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.327 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-04-26T02:07:22.328 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.329 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.330 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.331 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.332 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.333 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.334 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-04-26T02:07:22.335 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-04-26T02:07:22.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.337 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.338 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-04-26T02:07:22.339 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-04-26T02:07:22.339 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.339 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.339 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.339 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.340 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-04-26T02:07:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-04-26T02:07:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.343 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T02:07:22.344 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-04-26T02:07:22.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: "services": { 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: "dashboard": "https://172.21.15.52:8443/" 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: "always_on_modules": { 2024-04-26T02:07:22.346 INFO:teuthology.orchestra.run.smithi052.stdout: "octopus": [ 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-04-26T02:07:22.347 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pacific": [ 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-04-26T02:07:22.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "quincy": [ 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-04-26T02:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-04-26T02:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "reef": [ 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-04-26T02:07:22.351 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "squid": [ 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-04-26T02:07:22.352 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "last_failure_osd_epoch": 95, 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "active_clients": [ 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.354 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1182785025 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "libcephsqlite", 2024-04-26T02:07:22.355 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1228668697 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.356 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1084559622 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 4060031959 2024-04-26T02:07:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.359 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1221261053 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-04-26T02:07:22.360 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1451631031 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-04-26T02:07:22.361 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:20 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3495828890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:20 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1061363041' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:21 smithi191 ceph-mon[32745]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:20 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3495828890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:20 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1061363041' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:22.363 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:21 smithi183 ceph-mon[32752]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:22.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.247+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 msgr2=0x7fdd440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:22.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.247+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 0x7fdd440a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdd6010da60 tx=0x7fdd4c005f90 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 msgr2=0x7fdd601a4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd601a4700 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000e9f0 tx=0x7fdd5000eeb0 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 shutdown_connections 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fdd440a3d20 0x7fdd440a61e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd6010e460 0x7fdd6019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdd60108d90 0x7fdd601a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 --2- 172.21.15.52:0/1061363041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd601083f0 0x7fdd601a4700 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:07:22.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 >> 172.21.15.52:0/1061363041 conn(0x7fdd60077ad0 msgr2=0x7fdd60079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:07:22.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 shutdown_connections 2024-04-26T02:07:22.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:07:20.248+0000 7fdd65200640 1 -- 172.21.15.52:0/1061363041 wait complete. 2024-04-26T02:07:22.372 DEBUG:tasks.check_counter:Getting stats from a 2024-04-26T02:07:22.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-26T02:07:22.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:20 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3495828890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T02:07:22.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:20 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1061363041' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T02:07:22.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:21 smithi052 ceph-mon[27851]: pgmap v710: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 5.9 MiB/s wr, 195 op/s 2024-04-26T02:07:22.886 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:23.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:23 smithi052 ceph-mon[27851]: pgmap v711: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-04-26T02:07:24.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:23 smithi191 ceph-mon[32745]: pgmap v711: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-04-26T02:07:24.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:23 smithi183 ceph-mon[32752]: pgmap v711: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-04-26T02:07:25.261 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:25.261 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 278143, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 253259, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 88547331, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 331857465, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 14, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 37.057719885, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 16.489758437, 2024-04-26T02:07:25.262 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 22.613919950, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 2.465323597, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 253257, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 22.072352795, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000087153 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 476282, 2024-04-26T02:07:25.263 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.160694615, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000337 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 458150, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 591582, 2024-04-26T02:07:25.264 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 151377916, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 501537993, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 17, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 90.233570437, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 58.492516912, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 39.432194379, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 1.151150446, 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.265 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 591580, 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 53.466899178, 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000090379 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 922591, 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.366769318, 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000397 2024-04-26T02:07:25.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 5008, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 166672 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 21795, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 22005, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 9955918, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 238411024, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 7, 2024-04-26T02:07:25.267 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 4.707141833, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 1.748175074, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.839630789, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 1.885541358, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 22004, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 2.909114954, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000132208 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.268 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 2058288, 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 847296 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "cct": { 2024-04-26T02:07:25.269 INFO:teuthology.orchestra.run.smithi052.stdout: "total_workers": 1, 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "unhealthy_workers": 0 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "finisher-MDSRank": { 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "queue_len": 0, 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "complete_latency": { 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 57801, 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 135.792775685, 2024-04-26T02:07:25.270 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.002349315 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "queue_len": 0, 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "complete_latency": { 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 38312, 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 14.220007156, 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000371163 2024-04-26T02:07:25.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "mds": { 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "request": 209288, 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "reply": 180548, 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "reply_latency": { 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 180548, 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 1789.281843306, 2024-04-26T02:07:25.272 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.009910283 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "slow_reply": 0, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "forward": 28738, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_fetch_complete": 3, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_commit": 115, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_split": 9, 2024-04-26T02:07:25.273 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_merge": 3, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes": 3060, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_top": 1277, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_bottom": 1767, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_pin_tail": 16, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_pinned": 1226, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_expired": 629, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "inodes_with_caps": 251, 2024-04-26T02:07:25.274 INFO:teuthology.orchestra.run.smithi052.stdout: "caps": 251, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "subtrees": 21, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse": 335814, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_hit": 172126, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_forward": 28725, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_discover": 254, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "traverse_lock": 2494, 2024-04-26T02:07:25.275 INFO:teuthology.orchestra.run.smithi052.stdout: "load_cent": 204925, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "q": 0, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "exported": 15, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "exported_inodes": 378, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "imported": 16, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "imported_inodes": 832, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "root_rfiles": 671, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "root_rbytes": 211546248, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "root_rsnaps": 7, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:25.277 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_inode_file_caps": 11, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_cap_op_revoke": 28823, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_cap_op_grant": 111595, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_client_caps": 64495, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_client_caps_dirty": 18705, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_client_cap_release": 8816, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "process_request_cap_release": 140616 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_cache": { 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_update": 786, 2024-04-26T02:07:25.279 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_update_receipt": 1166, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_try_discover": 660, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_send_discover": 1131, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "dir_handle_discover": 902, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "num_strays": 0, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "strays_created": 25152, 2024-04-26T02:07:25.280 INFO:teuthology.orchestra.run.smithi052.stdout: "strays_enqueued": 25152, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "strays_migrated": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_started": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "recovery_completed": 0, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_quiesce_path": 13, 2024-04-26T02:07:25.281 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_quiesce_inode": 1501, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_exportdir": 15, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_flush": 0, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_fragmentdir": 12, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "ireq_inodestats": 0 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.282 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_log": { 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evlrg": 0, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evadd": 136887, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evex": 50769, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evtrm": 49161, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "ev": 87726, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evexg": 0, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "evexd": 1608, 2024-04-26T02:07:25.283 INFO:teuthology.orchestra.run.smithi052.stdout: "segadd": 199, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "segex": 71, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "segtrm": 69, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "seg": 130, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "segmjr": 8, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "segexg": 0, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "segexd": 2, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "expos": 139695473, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "wrpos": 625436287, 2024-04-26T02:07:25.284 INFO:teuthology.orchestra.run.smithi052.stdout: "rdpos": 0, 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "jlat": { 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 32199, 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 221.066432433, 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.006865630 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "replayed": 0 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.285 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_mem": { 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "ino": 2385, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "ino+": 28041, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "ino-": 25656, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "dir": 263, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "dir+": 823, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "dir-": 560, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "dn": 3038, 2024-04-26T02:07:25.286 INFO:teuthology.orchestra.run.smithi052.stdout: "dn+": 50679, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "dn-": 47641, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "cap": 251, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "cap+": 29166, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "cap-": 28915, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "rss": 86152, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "heap": 223608 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_server": { 2024-04-26T02:07:25.287 INFO:teuthology.orchestra.run.smithi052.stdout: "dispatch_client_request": 306917, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "dispatch_server_request": 604, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_client_request": 209288, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_client_session": 2812, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "handle_peer_request": 17165, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "req_create_latency": { 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 25085, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 607.023228077, 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.024198653 2024-04-26T02:07:25.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "req_getattr_latency": { 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 3723, 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 43.027930596, 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.011557327 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.289 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "req_link_latency": { 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.290 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookup_latency": { 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 51189, 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 225.906828345, 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.004413190 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookupino_latency": { 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookupname_latency": { 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.292 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.293 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "req_lssnap_latency": { 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "req_mkdir_latency": { 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.295 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "req_mknod_latency": { 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "req_mksnap_latency": { 2024-04-26T02:07:25.296 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "req_open_latency": { 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 1, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.002428578, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.002428578 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.297 INFO:teuthology.orchestra.run.smithi052.stdout: "req_readdir_latency": { 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 69789, 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 424.141482976, 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.006077483 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "req_rename_latency": { 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 4953, 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 74.768806129, 2024-04-26T02:07:25.298 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.015095660 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.299 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "req_rmdir_latency": { 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.300 INFO:teuthology.orchestra.run.smithi052.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "req_setattr_latency": { 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 72, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 34.071410587, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.473214035 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.302 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 584, 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 2.046189702, 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.003503749 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "req_setlayout_latency": { 2024-04-26T02:07:25.303 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "req_setxattr_latency": { 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.304 INFO:teuthology.orchestra.run.smithi052.stdout: "req_symlink_latency": { 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "req_unlink_latency": { 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 25152, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 378.293538316, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.015040296 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.305 INFO:teuthology.orchestra.run.smithi052.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_sessions": { 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "session_count": 1, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "session_add": 3, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "session_remove": 2, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "sessions_open": 1, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "sessions_stale": 0, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "total_load": 35696, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "average_load": 35696, 2024-04-26T02:07:25.307 INFO:teuthology.orchestra.run.smithi052.stdout: "avg_session_uptime": 1241, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "mempool": { 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:25.308 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:25.309 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_items": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:25.311 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_bytes": 665642, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_items": 1146, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_bytes": 264, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_items": 3, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bytes": 0, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_items": 0, 2024-04-26T02:07:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_items": 168, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:25.313 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_items": 0, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_bytes": 8461384, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_items": 67083, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_items": 0, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_items": 0 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.314 INFO:teuthology.orchestra.run.smithi052.stdout: "objecter": { 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_active": 0, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_laggy": 0, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_send": 60983, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_send_bytes": 627855004, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_resend": 33, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_reply": 60983, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "op_latency": { 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 60983, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 414.870536791, 2024-04-26T02:07:25.315 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.006803052 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "op_inflight": 0, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "oplen_avg": { 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 60983, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 62421 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "op": 60983, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "op_r": 989, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "op_w": 59994, 2024-04-26T02:07:25.316 INFO:teuthology.orchestra.run.smithi052.stdout: "op_rmw": 0, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "op_pg": 0, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_stat": 29, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_create": 24, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_read": 986, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_write": 33185, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_writefull": 535, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_writesame": 0, 2024-04-26T02:07:25.317 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_append": 0, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_zero": 0, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_truncate": 9, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_delete": 25361, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_mapext": 0, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_getxattr": 1, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_setxattr": 48, 2024-04-26T02:07:25.318 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_call": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_watch": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_notify": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_pgls": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "osdop_other": 0, 2024-04-26T02:07:25.319 INFO:teuthology.orchestra.run.smithi052.stdout: "linger_active": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "linger_send": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "linger_resend": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "linger_ping": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "poolop_active": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "poolop_send": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "poolop_resend": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "poolstat_active": 0, 2024-04-26T02:07:25.320 INFO:teuthology.orchestra.run.smithi052.stdout: "poolstat_send": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "poolstat_resend": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "statfs_active": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "statfs_send": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "statfs_resend": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "command_active": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "command_send": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "command_resend": 0, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "map_epoch": 132, 2024-04-26T02:07:25.321 INFO:teuthology.orchestra.run.smithi052.stdout: "map_full": 0, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "map_inc": 40, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_sessions": 12, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_session_open": 12, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_session_close": 0, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_laggy": 0, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_wr": 1664, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_rd": 6, 2024-04-26T02:07:25.322 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_del": 573 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "oft": { 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_total_objs": 1, 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_total_updates": 0, 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "omap_total_removes": 0 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.323 INFO:teuthology.orchestra.run.smithi052.stdout: "osd-slow-ops": { 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "slow_ops_count": 1739 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "purge_queue": { 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executing": 0, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executed_ops": 25152, 2024-04-26T02:07:25.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_executed": 25152, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 104857600, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 758084, 2024-04-26T02:07:25.325 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 181621666, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 758084, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 758084, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 181621666, 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:25.326 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 104857600, 2024-04-26T02:07:25.327 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 0, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 0, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 60983, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 630400380, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 60983, 2024-04-26T02:07:25.328 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 630400380, 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 1024, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 60983, 2024-04-26T02:07:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 60983, 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 60983, 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 60983, 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 3758096384, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 25152, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 2545424, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 25152, 2024-04-26T02:07:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 986, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 2545424, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-write_buf_throttle-0x5629cacb2500": { 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 17038, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 3758096384, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 136887, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 621241983, 2024-04-26T02:07:25.334 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 136887, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 32199, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 621224945, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:25.335 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:25.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.336 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:25.336 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:25.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:25 smithi052 ceph-mon[27851]: pgmap v712: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-04-26T02:07:26.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:25 smithi191 ceph-mon[32745]: pgmap v712: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-04-26T02:07:26.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:25 smithi183 ceph-mon[32752]: pgmap v712: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds.exported=15 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds.imported=16 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=786 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=1166 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=7 2024-04-26T02:07:26.234 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:26.235 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:26.235 DEBUG:tasks.check_counter:Getting stats from d 2024-04-26T02:07:26.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-26T02:07:26.725 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:27.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:27 smithi052 ceph-mon[27851]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-04-26T02:07:28.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:27 smithi191 ceph-mon[32745]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-04-26T02:07:28.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:27 smithi183 ceph-mon[32752]: pgmap v713: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 0, 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 2, 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 0, 2024-04-26T02:07:29.088 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 0, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.003609988, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.000656443, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:29.089 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.090 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 6, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 332, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 3312, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 214608, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 5, 2024-04-26T02:07:29.091 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.061387687, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.047851056, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.004787239, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 330, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.048003708, 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000145465 2024-04-26T02:07:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 214608 2024-04-26T02:07:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 444, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 432, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 1792112, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 267632, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.163075237, 2024-04-26T02:07:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.064977164, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.093954395, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.018738813, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 431, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.039932601, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000092651 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 1792112, 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 267632 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "cct": { 2024-04-26T02:07:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "total_workers": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "unhealthy_workers": 0 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "mempool": { 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_items": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_bytes": 150776, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_items": 32, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bytes": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_items": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_items": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_items": 0, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_items": 2, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_items": 0, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_items": 0 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 104857600, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 446, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 1729379, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 446, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 446, 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 1729379, 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:29.105 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:29.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:29 smithi052 ceph-mon[27851]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 945 KiB/s wr, 66 op/s 2024-04-26T02:07:29.935 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-26T02:07:29.935 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-26T02:07:29.935 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-26T02:07:29.935 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-26T02:07:29.936 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-26T02:07:29.936 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-26T02:07:29.936 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-26T02:07:29.936 DEBUG:tasks.check_counter:Getting stats from g 2024-04-26T02:07:29.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-26T02:07:30.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:29 smithi191 ceph-mon[32745]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 945 KiB/s wr, 66 op/s 2024-04-26T02:07:30.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:29 smithi183 ceph-mon[32752]: pgmap v714: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 945 KiB/s wr, 66 op/s 2024-04-26T02:07:30.444 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:31.890 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:31 smithi052 ceph-mon[27851]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 196 KiB/s wr, 29 op/s 2024-04-26T02:07:32.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:31 smithi191 ceph-mon[32745]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 196 KiB/s wr, 29 op/s 2024-04-26T02:07:32.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:31 smithi183 ceph-mon[32752]: pgmap v715: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 196 KiB/s wr, 29 op/s 2024-04-26T02:07:32.791 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:32.791 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:32.791 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 0, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 2, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 0, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 0, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.004443367, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T02:07:32.792 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.001079569, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:32.793 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 7, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 330, 2024-04-26T02:07:32.794 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 3472, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 292560, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.063126810, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.046687191, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.007656061, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:32.795 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 328, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.055576287, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000169439 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.796 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 292560 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 439, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 427, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 1790128, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 264736, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:32.797 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.175588126, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.065213916, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.119043893, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.019181894, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 426, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.040787245, 2024-04-26T02:07:32.798 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000095744 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 1790128, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 264736 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.799 INFO:teuthology.orchestra.run.smithi052.stdout: "cct": { 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "total_workers": 0, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "unhealthy_workers": 0 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "mempool": { 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:32.800 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:32.801 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:32.802 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_items": 0, 2024-04-26T02:07:32.803 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_bytes": 149745, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_items": 31, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bytes": 0, 2024-04-26T02:07:32.804 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_items": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_items": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:32.805 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_items": 0, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_items": 2, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_items": 0, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_items": 0 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:32.806 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 104857600, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 442, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 1728119, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 442, 2024-04-26T02:07:32.807 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 442, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 1728119, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:32.808 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:32.809 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:33.767 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-26T02:07:33.768 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-26T02:07:33.768 DEBUG:tasks.check_counter:Getting stats from j 2024-04-26T02:07:33.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-26T02:07:33.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:33 smithi052 ceph-mon[27851]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 206 KiB/s wr, 33 op/s 2024-04-26T02:07:34.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:33 smithi191 ceph-mon[32745]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 206 KiB/s wr, 33 op/s 2024-04-26T02:07:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:33 smithi183 ceph-mon[32752]: pgmap v716: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 206 KiB/s wr, 33 op/s 2024-04-26T02:07:34.265 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:07:35.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:34 smithi191 ceph-mon[32745]: pgmap v717: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T02:07:35.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:34 smithi183 ceph-mon[32752]: pgmap v717: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T02:07:35.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:34 smithi052 ceph-mon[27851]: pgmap v717: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 9 op/s 2024-04-26T02:07:36.421 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-04-26T02:07:36.421 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:36.421 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 0, 2024-04-26T02:07:36.421 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 2, 2024-04-26T02:07:36.421 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 0, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 0, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.003719197, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.000680376, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:36.422 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.424 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:36.424 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:36.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.424 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 451, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 437, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 1795920, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 270000, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:36.425 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.167333300, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.064163882, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.102522457, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.020029416, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 437, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.046490807, 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000106386 2024-04-26T02:07:36.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 1795920, 2024-04-26T02:07:36.427 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 270000 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_messages": 2, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages": 333, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_bytes": 320, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_bytes": 214688, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_created_connections": 5, 2024-04-26T02:07:36.428 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_total_time": 0.060935887, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_send_time": 0.047926785, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_recv_time": 0.004498467, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 330, 2024-04-26T02:07:36.429 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.046966151, 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000142321 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.433 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T02:07:36.433 INFO:teuthology.orchestra.run.smithi052.stdout: "msgr_send_encrypted_bytes": 214688 2024-04-26T02:07:36.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "cct": { 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "total_workers": 0, 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "unhealthy_workers": 0 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "mempool": { 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:36.434 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:36.435 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:36.436 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:36.437 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_items": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:36.438 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_bytes": 148714, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:36.439 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_bytes": 0, 2024-04-26T02:07:36.440 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_items": 0, 2024-04-26T02:07:36.440 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:36.440 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:36.440 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:36.440 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_items": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "pgmap_items": 0, 2024-04-26T02:07:36.441 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_co_items": 2, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_1_items": 0, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "unittest_2_items": 0 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:36.442 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 0, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "max": 104857600, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "get_started": 0, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "get": 449, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "get_sum": 1729769, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "get_or_fail_success": 449, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "take": 0, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "take_sum": 0, 2024-04-26T02:07:36.443 INFO:teuthology.orchestra.run.smithi052.stdout: "put": 449, 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: "put_sum": 1729769, 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: "wait": { 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: "avgcount": 0, 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: "sum": 0.000000000, 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: "avgtime": 0.000000000 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-04-26T02:07:36.444 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-26T02:07:37.459 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-26T02:07:37.460 DEBUG:tasks.check_counter:Getting stats from b 2024-04-26T02:07:37.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-26T02:07:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:37 smithi052 ceph-mon[27851]: pgmap v718: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 51 KiB/s wr, 13 op/s 2024-04-26T02:07:37.776 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:37.899 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:37 smithi183 ceph-mon[32752]: pgmap v718: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 51 KiB/s wr, 13 op/s 2024-04-26T02:07:37.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:37 smithi191 ceph-mon[32745]: pgmap v718: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 51 KiB/s wr, 13 op/s 2024-04-26T02:07:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:39 smithi183 ceph-mon[32752]: pgmap v719: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:39.667 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-26T02:07:39.667 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 261411, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 286181, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 77652970, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 348764228, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 242, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 14.348441339, 2024-04-26T02:07:39.668 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 6.025622015, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 8.767445681, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 1.101495899, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 286184, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 18.721291846, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000065416 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 467053, 2024-04-26T02:07:39.669 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.065296491, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000139 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 2112272, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 286656 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 369440, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 487526, 2024-04-26T02:07:39.670 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 122637481, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 543530122, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 23, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 19.240057540, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 8.398714969, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 12.290854435, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 1.208167538, 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:39.671 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 487523, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 27.251739543, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000055898 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 655393, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.105426668, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000160 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.672 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 107072, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 861328 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 252001, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 283252, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 75242372, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 322346497, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 20, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 8, 2024-04-26T02:07:39.673 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 13.801924886, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 5.944933807, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 7.785730024, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 1.164822727, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 283247, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 24.239466181, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000085577 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.674 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 463112, 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.060893442, 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000131 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 697904, 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 230176 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.675 INFO:teuthology.orchestra.run.smithi183.stdout: "cct": { 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "total_workers": 1, 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "unhealthy_workers": 0 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "finisher-MDSRank": { 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "queue_len": 0, 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "complete_latency": { 2024-04-26T02:07:39.676 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 92046, 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 110.790858446, 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.001203646 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "queue_len": 0, 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "complete_latency": { 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 45664, 2024-04-26T02:07:39.677 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 7.145697615, 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000156484 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "mds": { 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "request": 245471, 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "reply": 214792, 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "reply_latency": { 2024-04-26T02:07:39.678 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 214792, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 1703.615532244, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.007931466 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "slow_reply": 0, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "forward": 30679, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_fetch_complete": 16, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_commit": 3969, 2024-04-26T02:07:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_split": 15, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_merge": 11, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes": 3414, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_top": 2136, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_bottom": 1268, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_pin_tail": 10, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_pinned": 364, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_expired": 2076, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_with_caps": 253, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "caps": 254, 2024-04-26T02:07:39.680 INFO:teuthology.orchestra.run.smithi183.stdout: "subtrees": 21, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse": 368250, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_hit": 224855, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_forward": 30639, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_discover": 119, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_dir_fetch": 8, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_lock": 2674, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "load_cent": 37668, 2024-04-26T02:07:39.681 INFO:teuthology.orchestra.run.smithi183.stdout: "q": 0, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "exported": 25, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "exported_inodes": 1479, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "imported": 28, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "imported_inodes": 921, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:39.682 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rfiles": 672, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rbytes": 217532579, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rsnaps": 7, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:39.683 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_inode_file_caps": 47, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_revoke": 26532, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_grant": 94242, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_caps": 69689, 2024-04-26T02:07:39.684 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_caps_dirty": 23551, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_cap_release": 9517, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "process_request_cap_release": 152923 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache": { 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_update": 294, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_update_receipt": 206, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_try_discover": 122, 2024-04-26T02:07:39.685 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_send_discover": 244, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_handle_discover": 1820, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays": 0, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_created": 25170, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_enqueued": 25170, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:39.686 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_migrated": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_started": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_completed": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_quiesce_path": 13, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_quiesce_inode": 1488, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:39.687 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_exportdir": 26, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_flush": 0, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_fragmentdir": 26, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_inodestats": 0 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log": { 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "evlrg": 0, 2024-04-26T02:07:39.688 INFO:teuthology.orchestra.run.smithi183.stdout: "evadd": 142802, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "evex": 142651, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "evtrm": 142651, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "ev": 151, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "evexg": 0, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "evexd": 667, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "segadd": 285, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "segex": 284, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "segtrm": 284, 2024-04-26T02:07:39.689 INFO:teuthology.orchestra.run.smithi183.stdout: "seg": 1, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "segmjr": 1, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "segexg": 0, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "segexd": 2, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "expos": 639470755, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "wrpos": 640345337, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "rdpos": 0, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "jlat": { 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 49554, 2024-04-26T02:07:39.690 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 334.739431908, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.006755043 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "replayed": 0 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_mem": { 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "ino": 2629, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "ino+": 29236, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "ino-": 26607, 2024-04-26T02:07:39.691 INFO:teuthology.orchestra.run.smithi183.stdout: "dir": 255, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "dir+": 472, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "dir-": 217, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "dn": 3414, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "dn+": 57905, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "dn-": 54491, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "cap": 254, 2024-04-26T02:07:39.692 INFO:teuthology.orchestra.run.smithi183.stdout: "cap+": 29295, 2024-04-26T02:07:39.693 INFO:teuthology.orchestra.run.smithi183.stdout: "cap-": 29041, 2024-04-26T02:07:39.693 INFO:teuthology.orchestra.run.smithi183.stdout: "rss": 97296, 2024-04-26T02:07:39.693 INFO:teuthology.orchestra.run.smithi183.stdout: "heap": 231800 2024-04-26T02:07:39.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.694 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_server": { 2024-04-26T02:07:39.694 INFO:teuthology.orchestra.run.smithi183.stdout: "dispatch_client_request": 349545, 2024-04-26T02:07:39.694 INFO:teuthology.orchestra.run.smithi183.stdout: "dispatch_server_request": 23078, 2024-04-26T02:07:39.694 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_request": 245471, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_session": 4000, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_peer_request": 46312, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "req_create_latency": { 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 25984, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 307.725597560, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.011842887 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getattr_latency": { 2024-04-26T02:07:39.695 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 23372, 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 253.107073470, 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.010829499 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "req_link_latency": { 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookup_latency": { 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 53236, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 180.486051451, 2024-04-26T02:07:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.003390300 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.698 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupino_latency": { 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 2, 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000420924, 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000210462 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupname_latency": { 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.699 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lssnap_latency": { 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 22, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.010712002, 2024-04-26T02:07:39.701 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000486909 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mkdir_latency": { 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 119, 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 7.371955474, 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.061949205 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mknod_latency": { 2024-04-26T02:07:39.702 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mksnap_latency": { 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 22, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 19.075491765, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.867067807 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.703 INFO:teuthology.orchestra.run.smithi183.stdout: "req_open_latency": { 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 12, 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.030647151, 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.002553929 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "req_readdir_latency": { 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 80208, 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 345.075260196, 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.004302254 2024-04-26T02:07:39.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rename_latency": { 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 5612, 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 192.781206070, 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.034351604 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.705 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmdir_latency": { 2024-04-26T02:07:39.706 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 1, 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.002700627, 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.002700627 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 15, 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 9.320516877, 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.621367791 2024-04-26T02:07:39.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setattr_latency": { 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 174, 2024-04-26T02:07:39.708 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 164.350134580, 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.944541003 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:39.709 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 836, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 2.187585007, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.002616728 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setlayout_latency": { 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.710 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setxattr_latency": { 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 7, 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.021348035, 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.003049719 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "req_symlink_latency": { 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "req_unlink_latency": { 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 25170, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 222.068831055, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.008822758 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:39.712 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_sessions": { 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "session_count": 2, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "session_add": 5, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "session_remove": 3, 2024-04-26T02:07:39.713 INFO:teuthology.orchestra.run.smithi183.stdout: "sessions_open": 2, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "sessions_stale": 0, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "total_load": 38104, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "average_load": 19052, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "avg_session_uptime": 1326, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "mempool": { 2024-04-26T02:07:39.714 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_bytes": 720, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_items": 720, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:39.715 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:39.716 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:39.717 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_items": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:39.718 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_bytes": 406487, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_items": 1472, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_bytes": 88, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_items": 1, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bytes": 0, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_items": 0, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:39.719 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_items": 168, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_items": 0, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_bytes": 9246123, 2024-04-26T02:07:39.720 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_items": 74808, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_items": 0, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_items": 0 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter": { 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "op_active": 0, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "op_laggy": 0, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "op_send": 96304, 2024-04-26T02:07:39.721 INFO:teuthology.orchestra.run.smithi183.stdout: "op_send_bytes": 669632233, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "op_resend": 32, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "op_reply": 96304, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "op_latency": { 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 96304, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 1793.364367962, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.018621909 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.722 INFO:teuthology.orchestra.run.smithi183.stdout: "op_inflight": 0, 2024-04-26T02:07:39.723 INFO:teuthology.orchestra.run.smithi183.stdout: "oplen_avg": { 2024-04-26T02:07:39.725 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 96304, 2024-04-26T02:07:39.725 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 133065 2024-04-26T02:07:39.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "op": 96304, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "op_r": 879, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "op_w": 95425, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "op_rmw": 0, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "op_pg": 0, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_stat": 3707, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_create": 13092, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_read": 863, 2024-04-26T02:07:39.726 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_write": 50417, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_writefull": 884, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_writesame": 0, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_append": 0, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_zero": 0, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_truncate": 15, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_delete": 26059, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_mapext": 0, 2024-04-26T02:07:39.727 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_getxattr": 11, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_setxattr": 26184, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_call": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_watch": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_notify": 0, 2024-04-26T02:07:39.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_pgls": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_other": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_active": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_send": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_resend": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_ping": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_active": 0, 2024-04-26T02:07:39.729 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_send": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_resend": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_active": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_send": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_resend": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_active": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_send": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_resend": 0, 2024-04-26T02:07:39.730 INFO:teuthology.orchestra.run.smithi183.stdout: "command_active": 0, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "command_send": 0, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "command_resend": 0, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "map_epoch": 132, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "map_full": 0, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "map_inc": 40, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_sessions": 12, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_session_open": 12, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_session_close": 0, 2024-04-26T02:07:39.731 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_laggy": 0, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_wr": 9218, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_rd": 32, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_del": 2583 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "oft": { 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_objs": 1, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_updates": 0, 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_removes": 0 2024-04-26T02:07:39.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "osd-slow-ops": { 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "slow_ops_count": 2360 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "purge_queue": { 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing": 0, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:39.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executed_ops": 25170, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executed": 25170, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:39.734 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 882848, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 195955028, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 882848, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 882848, 2024-04-26T02:07:39.735 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 195955028, 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:39.736 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 96304, 2024-04-26T02:07:39.737 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 672178891, 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 96304, 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 672178891, 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 1024, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:39.739 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 96304, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 96304, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 96304, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 96304, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.740 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.741 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.741 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:39.741 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 3758096384, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 25170, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 2546914, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 25170, 2024-04-26T02:07:39.742 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 863, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 2546914, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-write_buf_throttle-0x55b692f14500": { 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 3758096384, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 142802, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 636151033, 2024-04-26T02:07:39.744 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 142802, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 49554, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 636151033, 2024-04-26T02:07:39.745 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:39.746 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:39.746 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:39.746 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:39.746 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:39.747 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-26T02:07:39.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:39 smithi191 ceph-mon[32745]: pgmap v719: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:39.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:39 smithi052 ceph-mon[27851]: pgmap v719: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:40.559 INFO:tasks.check_counter:Daemon mds.b mds.exported=25 2024-04-26T02:07:40.559 INFO:tasks.check_counter:Daemon mds.b mds.imported=28 2024-04-26T02:07:40.559 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=294 2024-04-26T02:07:40.560 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=206 2024-04-26T02:07:40.560 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-26T02:07:40.560 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=22 2024-04-26T02:07:40.560 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=15 2024-04-26T02:07:40.560 DEBUG:tasks.check_counter:Getting stats from e 2024-04-26T02:07:40.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-26T02:07:40.809 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:41.676 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:41 smithi183 ceph-mon[32752]: pgmap v720: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:41.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:41 smithi191 ceph-mon[32745]: pgmap v720: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:41.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:41 smithi052 ceph-mon[27851]: pgmap v720: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-26T02:07:42.541 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 447, 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 435, 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 1792928, 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 269552, 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:42.542 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.072370141, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.024688349, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.044365508, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.009840583, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 434, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.018937029, 2024-04-26T02:07:42.543 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000043633 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 1792928, 2024-04-26T02:07:42.544 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 269552 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 335, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 640, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 371472, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 7, 2024-04-26T02:07:42.545 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.020005870, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.014123061, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.002188037, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 331, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.035891226, 2024-04-26T02:07:42.546 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000108432 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 640, 2024-04-26T02:07:42.547 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 371472 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:42.548 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.000572367, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.000041016, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.000514988, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 3, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000746899, 2024-04-26T02:07:42.549 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000248966 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:42.550 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "cct": { 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "total_workers": 0, 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "unhealthy_workers": 0 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "mempool": { 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:42.551 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:42.552 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:42.553 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:42.554 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_items": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:42.555 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_bytes": 148692, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bytes": 0, 2024-04-26T02:07:42.556 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_items": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_items": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:42.557 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_items": 0, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_items": 2, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_items": 0, 2024-04-26T02:07:42.558 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_items": 0 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 451, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 1729823, 2024-04-26T02:07:42.559 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 451, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 451, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 1729823, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:42.560 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:42.561 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:42.561 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:42.561 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-26T02:07:43.296 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-26T02:07:43.297 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-26T02:07:43.297 DEBUG:tasks.check_counter:Getting stats from h 2024-04-26T02:07:43.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-26T02:07:43.539 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:43.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:43 smithi183 ceph-mon[32752]: pgmap v721: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 12 op/s 2024-04-26T02:07:43.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:43 smithi191 ceph-mon[32745]: pgmap v721: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 12 op/s 2024-04-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:43 smithi052 ceph-mon[27851]: pgmap v721: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 KiB/s wr, 12 op/s 2024-04-26T02:07:45.180 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-26T02:07:45.180 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:45.180 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 235, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 235, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 29586684, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 81150, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 2, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.035872094, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.007304105, 2024-04-26T02:07:45.181 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.022539260, 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.005454168, 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 234, 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.009107754, 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000038922 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:45.182 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 164512, 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 19136 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:45.183 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 147, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 167, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 28191927, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 169261, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.025771280, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.003953103, 2024-04-26T02:07:45.184 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.017630691, 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.003221589, 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 166, 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.009749713, 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000058733 2024-04-26T02:07:45.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 124048 2024-04-26T02:07:45.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 116, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 118, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 21039600, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 36904, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 5, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.021510844, 2024-04-26T02:07:45.187 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.002021546, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.015527637, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.003011893, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 116, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.009066196, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000078156 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "cct": { 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "total_workers": 1, 2024-04-26T02:07:45.189 INFO:teuthology.orchestra.run.smithi183.stdout: "unhealthy_workers": 0 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "finisher-MDSRank": { 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "queue_len": 0, 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "complete_latency": { 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 705, 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.042741912, 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000060626 2024-04-26T02:07:45.190 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "queue_len": 0, 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "complete_latency": { 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "mds": { 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "request": 0, 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "reply": 0, 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "reply_latency": { 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "slow_reply": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "forward": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_commit": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_split": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_merge": 0, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes": 3672, 2024-04-26T02:07:45.193 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_top": 2438, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_bottom": 1234, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_pinned": 189, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_expired": 0, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "caps": 0, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "subtrees": 20, 2024-04-26T02:07:45.194 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_hit": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_forward": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_discover": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "traverse_lock": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "load_cent": 0, 2024-04-26T02:07:45.195 INFO:teuthology.orchestra.run.smithi183.stdout: "q": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "exported": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "exported_inodes": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "imported": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "imported_inodes": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:45.196 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rfiles": 672, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rbytes": 217532579, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "root_rsnaps": 7, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:45.197 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:45.198 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_caps": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "process_request_cap_release": 0 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache": { 2024-04-26T02:07:45.199 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_update": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_try_discover": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_send_discover": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_created": 0, 2024-04-26T02:07:45.200 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_enqueued": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "strays_migrated": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_started": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_completed": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:45.201 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_flush": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ireq_inodestats": 0 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.202 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log": { 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evlrg": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evadd": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evex": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evtrm": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "ev": 14816, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evexg": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "evexd": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "segadd": 0, 2024-04-26T02:07:45.203 INFO:teuthology.orchestra.run.smithi183.stdout: "segex": 0, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "segtrm": 0, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "seg": 2, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "segmjr": 6, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "segexg": 0, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "segexd": 0, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "expos": 639470755, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "wrpos": 640594670, 2024-04-26T02:07:45.204 INFO:teuthology.orchestra.run.smithi183.stdout: "rdpos": 640584613, 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "jlat": { 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "replayed": 14816 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_mem": { 2024-04-26T02:07:45.205 INFO:teuthology.orchestra.run.smithi183.stdout: "ino": 548, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "ino+": 3425, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "ino-": 2877, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "dir": 127, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "dir+": 128, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "dir-": 1, 2024-04-26T02:07:45.206 INFO:teuthology.orchestra.run.smithi183.stdout: "dn": 3672, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "dn+": 3672, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "dn-": 0, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "cap": 0, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "cap+": 0, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "cap-": 0, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "rss": 56692, 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: "heap": 215392 2024-04-26T02:07:45.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_server": { 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_request": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_client_session": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "handle_peer_request": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "req_create_latency": { 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.208 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getattr_latency": { 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "req_link_latency": { 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.210 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookup_latency": { 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.211 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupino_latency": { 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupname_latency": { 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.212 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.213 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: "req_lssnap_latency": { 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mkdir_latency": { 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.215 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mknod_latency": { 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "req_mksnap_latency": { 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.216 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "req_open_latency": { 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "req_readdir_latency": { 2024-04-26T02:07:45.217 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rename_latency": { 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.219 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmdir_latency": { 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.220 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setattr_latency": { 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.221 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:45.222 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setlayout_latency": { 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.223 INFO:teuthology.orchestra.run.smithi183.stdout: "req_setxattr_latency": { 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "req_symlink_latency": { 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "req_unlink_latency": { 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:45.225 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_sessions": { 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "session_count": 2, 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "session_add": 2, 2024-04-26T02:07:45.226 INFO:teuthology.orchestra.run.smithi183.stdout: "session_remove": 0, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "sessions_open": 0, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "sessions_stale": 0, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "total_load": 0, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "average_load": 0, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "avg_session_uptime": 2983, 2024-04-26T02:07:45.227 INFO:teuthology.orchestra.run.smithi183.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "mempool": { 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:45.228 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:45.229 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:45.230 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:45.231 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_items": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_bytes": 248120, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_items": 988, 2024-04-26T02:07:45.232 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bytes": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_items": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:45.233 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_items": 168, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_items": 0, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_bytes": 4402679, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_items": 35098, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_items": 0, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_items": 0 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter": { 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "op_active": 0, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "op_laggy": 0, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "op_send": 453, 2024-04-26T02:07:45.235 INFO:teuthology.orchestra.run.smithi183.stdout: "op_send_bytes": 26, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "op_resend": 4, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "op_reply": 453, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "op_latency": { 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 453, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.767980314, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.001695320 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "op_inflight": 0, 2024-04-26T02:07:45.236 INFO:teuthology.orchestra.run.smithi183.stdout: "oplen_avg": { 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 453, 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 454 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "op": 453, 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "op_r": 453, 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "op_w": 0, 2024-04-26T02:07:45.237 INFO:teuthology.orchestra.run.smithi183.stdout: "op_rmw": 0, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "op_pg": 0, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_stat": 263, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_create": 0, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_read": 187, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_write": 0, 2024-04-26T02:07:45.238 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_writefull": 0, 2024-04-26T02:07:45.242 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_writesame": 0, 2024-04-26T02:07:45.242 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_append": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_zero": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_truncate": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_delete": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_mapext": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_getxattr": 2, 2024-04-26T02:07:45.243 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_call": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_watch": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_notify": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_pgls": 0, 2024-04-26T02:07:45.244 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "osdop_other": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_active": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_send": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_resend": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "linger_ping": 0, 2024-04-26T02:07:45.245 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_active": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_send": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "poolop_resend": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_active": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_send": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "poolstat_resend": 0, 2024-04-26T02:07:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_active": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_send": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "statfs_resend": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "command_active": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "command_send": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "command_resend": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "map_epoch": 132, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "map_full": 0, 2024-04-26T02:07:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: "map_inc": 3, 2024-04-26T02:07:45.248 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_sessions": 8, 2024-04-26T02:07:45.248 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_session_open": 8, 2024-04-26T02:07:45.248 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_session_close": 0, 2024-04-26T02:07:45.248 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_laggy": 0, 2024-04-26T02:07:45.248 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_wr": 0, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_rd": 2, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_del": 0 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "oft": { 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_objs": 0, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_updates": 0, 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: "omap_total_removes": 0 2024-04-26T02:07:45.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "osd-slow-ops": { 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "slow_ops_count": 0 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "purge_queue": { 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing": 0, 2024-04-26T02:07:45.250 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_executed": 0, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:45.251 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 494, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 78768734, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 494, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 494, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 78768734, 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:45.252 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:45.253 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 453, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 78520517, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 453, 2024-04-26T02:07:45.254 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 78520517, 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:45.255 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 1024, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 453, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 453, 2024-04-26T02:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 453, 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 453, 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 3758096384, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 0, 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:45.259 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-write_buf_throttle-0x560ea245c640": { 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 3758096384, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 0, 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:45.261 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:45.262 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-26T02:07:45.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:45 smithi191 ceph-mon[32745]: pgmap v722: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 38 KiB/s wr, 8 op/s 2024-04-26T02:07:45.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:45 smithi183 ceph-mon[32752]: pgmap v722: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 38 KiB/s wr, 8 op/s 2024-04-26T02:07:45.961 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-26T02:07:45.961 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-26T02:07:45.961 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-26T02:07:45.961 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-26T02:07:45.961 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-04-26T02:07:45.962 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:45.962 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:45.962 DEBUG:tasks.check_counter:Getting stats from k 2024-04-26T02:07:45.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-26T02:07:45.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:45 smithi052 ceph-mon[27851]: pgmap v722: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 38 KiB/s wr, 8 op/s 2024-04-26T02:07:46.202 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T02:07:47.550 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:47 smithi183 ceph-mon[32752]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-26T02:07:47.551 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:47 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:47.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:47 smithi052 ceph-mon[27851]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-26T02:07:47.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:47 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:47.820 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-04-26T02:07:47.820 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:47.820 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 451, 2024-04-26T02:07:47.820 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 439, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 1794016, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 272112, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.073955648, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.024908137, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.045943556, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.010045075, 2024-04-26T02:07:47.821 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 438, 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.018584886, 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000042431 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:47.822 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 1794016, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 272112 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 2, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 336, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 320, 2024-04-26T02:07:47.823 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 215216, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 5, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.019144370, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.013802953, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.001545391, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 333, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.034843355, 2024-04-26T02:07:47.824 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000104634 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 215216 2024-04-26T02:07:47.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_messages": 4, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages": 4, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_bytes": 2992, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_bytes": 448, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:47.826 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_total_time": 0.000575127, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_send_time": 0.000042415, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_recv_time": 0.000506181, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 3, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000684487, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000228162 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.827 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "cct": { 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "total_workers": 0, 2024-04-26T02:07:47.828 INFO:teuthology.orchestra.run.smithi183.stdout: "unhealthy_workers": 0 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "mempool": { 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:47.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:47.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:47.831 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:47.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_items": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_bytes": 148692, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:47.833 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bytes": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_items": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:47.834 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_items": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "pgmap_items": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_co_items": 2, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_1_items": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:47.835 INFO:teuthology.orchestra.run.smithi183.stdout: "unittest_2_items": 0 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 0, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "max": 104857600, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "get_started": 0, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "get": 453, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "get_sum": 1730289, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:47.836 INFO:teuthology.orchestra.run.smithi183.stdout: "get_or_fail_success": 453, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "take": 0, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "take_sum": 0, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "put": 453, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "put_sum": 1730289, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "wait": { 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "avgcount": 0, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "sum": 0.000000000, 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: "avgtime": 0.000000000 2024-04-26T02:07:47.837 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:47.838 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-04-26T02:07:47.838 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-04-26T02:07:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:47 smithi191 ceph-mon[32745]: pgmap v723: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-26T02:07:47.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:47 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:48.542 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-26T02:07:48.543 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-26T02:07:48.543 DEBUG:tasks.check_counter:Getting stats from c 2024-04-26T02:07:48.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-26T02:07:48.823 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:49.513 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:49 smithi191 ceph-mon[32745]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:07:49.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:49 smithi183 ceph-mon[32752]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:07:49.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:49 smithi052 ceph-mon[27851]: pgmap v724: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T02:07:50.622 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-26T02:07:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 3934, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 3936, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 228550321, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 1251019, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 8, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.443219055, 2024-04-26T02:07:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.104605153, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.196731709, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.113485452, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3934, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.129054653, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000032804 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1711, 2024-04-26T02:07:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1648, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 203708300, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 648910, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.287817431, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.051293443, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.198433035, 2024-04-26T02:07:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.039613467, 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1647, 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.058542268, 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000035544 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 2049248, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 277424 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 884, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1213, 2024-04-26T02:07:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 193184121, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 1099926, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 10, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.178387240, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.029047702, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.120320679, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.020003093, 2024-04-26T02:07:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1210, 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.055434655, 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000045813 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 821040 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 9783, 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.264530589, 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000027039 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 0, 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 0, 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 28263, 2024-04-26T02:07:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 7077, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 21186, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 18153, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 0, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 18, 2024-04-26T02:07:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-04-26T02:07:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 0 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 0, 2024-04-26T02:07:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 136909, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-04-26T02:07:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 0, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 130, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 13, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 139695473, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 625871337, 2024-04-26T02:07:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 625863996, 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 136909 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 2710, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 28199, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 25489, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 122, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 163, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 41, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 28263, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 28658, 2024-04-26T02:07:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 395, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 0, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 0, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 89056, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 215416 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-04-26T02:07:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-04-26T02:07:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-04-26T02:07:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-04-26T02:07:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 3, 2024-04-26T02:07:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 2, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 0, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 0, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 2982, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 355107, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 3166, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-04-26T02:07:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 26782855, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 180787, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-04-26T02:07:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 5979, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 21, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 3, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 5979, 2024-04-26T02:07:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 5979, 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 7.570873876, 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001266244 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-04-26T02:07:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 5979, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 5980 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 5979, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 5979, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 0, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 3516, 2024-04-26T02:07:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 2461, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-04-26T02:07:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 1, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-04-26T02:07:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-04-26T02:07:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-04-26T02:07:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 132, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 40, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 11, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 1, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 0, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 2, 2024-04-26T02:07:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 0 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 0, 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-04-26T02:07:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 0, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 6525, 2024-04-26T02:07:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 624824916, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 6525, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 6525, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 624824916, 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 5979, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 621736570, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 5979, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 621736570, 2024-04-26T02:07:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-04-26T02:07:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 5979, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 5979, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 5979, 2024-04-26T02:07:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 5979, 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x555acc9586e0": { 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:50.700 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:51.489 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:51.490 DEBUG:tasks.check_counter:Getting stats from f 2024-04-26T02:07:51.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-26T02:07:51.739 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:51.778 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:51 smithi191 ceph-mon[32745]: pgmap v725: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 8 op/s 2024-04-26T02:07:51.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:51 smithi183 ceph-mon[32752]: pgmap v725: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 8 op/s 2024-04-26T02:07:51.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:51 smithi052 ceph-mon[27851]: pgmap v725: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 8 op/s 2024-04-26T02:07:52.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:52 smithi191 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:52.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:52 smithi183 ceph-mon[32752]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:52.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:52 smithi052 ceph-mon[27851]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T02:07:53.518 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-26T02:07:53.518 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1509, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1445, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 192693679, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 584944, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 8, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.247156768, 2024-04-26T02:07:53.519 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.047369107, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.169323903, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.033576837, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1444, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.050730173, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000035131 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.520 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 2054144, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 278704 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1562, 2024-04-26T02:07:53.521 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1894, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 189155076, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 1240165, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 9, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.232253199, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.050817126, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.121133108, 2024-04-26T02:07:53.522 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.045772277, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1891, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.081971792, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000043348 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.523 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 744080 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 3668, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 3668, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 315653455, 2024-04-26T02:07:53.524 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 1165658, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.482812941, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.092696168, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.250471254, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.111424177, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3667, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.124054427, 2024-04-26T02:07:53.525 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000033829 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T02:07:53.526 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:53.527 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 9929, 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.273070843, 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000027502 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.528 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 0, 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 0, 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.529 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 0, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 0, 2024-04-26T02:07:53.530 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 27104, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 5498, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 21606, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 19250, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-04-26T02:07:53.531 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 7, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-04-26T02:07:53.532 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 0, 2024-04-26T02:07:53.533 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:53.534 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 0, 2024-04-26T02:07:53.535 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 0 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-04-26T02:07:53.536 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:53.537 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:53.538 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 0, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 134770, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-04-26T02:07:53.539 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 144, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 12, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-04-26T02:07:53.540 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 142045360, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 697902993, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 697902419, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 134770 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.541 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 1881, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 27438, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 25557, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 83, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 159, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 76, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 27104, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 27729, 2024-04-26T02:07:53.542 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 625, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 0, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 0, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 86272, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 215416 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 0, 2024-04-26T02:07:53.543 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 0, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 0, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 0, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 0, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-04-26T02:07:53.544 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.545 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.546 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.549 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.550 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-04-26T02:07:53.550 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.550 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.550 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.551 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.552 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-04-26T02:07:53.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.559 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.560 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.562 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 2, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 5, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 3, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 2, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-04-26T02:07:53.563 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 0, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 0, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 6216, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:53.564 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:53.565 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:53.566 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:53.567 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 323010, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 2280, 2024-04-26T02:07:53.568 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:53.569 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 23958688, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 157711, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:53.570 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 6187, 2024-04-26T02:07:53.571 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 21, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 3, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 6187, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 6187, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 8.380860353, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001354591 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-04-26T02:07:53.572 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 6187, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 6188 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 6187, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 6187, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 0, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-04-26T02:07:53.573 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 3778, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 2407, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-04-26T02:07:53.574 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 1, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:53.575 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-04-26T02:07:53.576 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-04-26T02:07:53.577 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 132, 2024-04-26T02:07:53.578 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 40, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 11, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 1, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 0, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 2, 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 0 2024-04-26T02:07:53.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 0, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-04-26T02:07:53.580 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 0, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 0, 2024-04-26T02:07:53.581 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 0, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 6735, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 696865378, 2024-04-26T02:07:53.582 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 6735, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 6735, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 696865378, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.583 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:53.584 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 6187, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 693737826, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 6187, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 693737826, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:53.586 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 6187, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 6187, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 6187, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 6187, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.588 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.589 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x559b8d91e6e0": { 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:53.590 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:53.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:53.592 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-26T02:07:53.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:53 smithi191 ceph-mon[32745]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 13 op/s 2024-04-26T02:07:53.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:53 smithi183 ceph-mon[32752]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 13 op/s 2024-04-26T02:07:53.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:53 smithi052 ceph-mon[27851]: pgmap v726: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 13 op/s 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:54.283 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:54.283 DEBUG:tasks.check_counter:Getting stats from i 2024-04-26T02:07:54.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-26T02:07:54.531 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:55.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:55 smithi191 ceph-mon[32745]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 KiB/s wr, 8 op/s 2024-04-26T02:07:55.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:55 smithi183 ceph-mon[32752]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 KiB/s wr, 8 op/s 2024-04-26T02:07:55.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:55 smithi052 ceph-mon[27851]: pgmap v727: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 KiB/s wr, 8 op/s 2024-04-26T02:07:56.236 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-26T02:07:56.236 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 169827, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 175267, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 53581263, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 403293549, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 17, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 5, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 8.917904138, 2024-04-26T02:07:56.237 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 3.713288049, 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 5.025974442, 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.921488030, 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 175263, 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 4.712027286, 2024-04-26T02:07:56.238 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000026885 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 280586, 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.043672816, 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000155 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 4416, 2024-04-26T02:07:56.239 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 5088 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 312216, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 425772, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 101592180, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 557809633, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 14, 2024-04-26T02:07:56.240 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 16.754546961, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 7.453826647, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 10.869076666, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.711211521, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 425769, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 11.658407611, 2024-04-26T02:07:56.241 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000027382 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 589626, 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.105390567, 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000178 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-04-26T02:07:56.242 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 824976 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 266057, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 233562, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 89119637, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 193023602, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 10, 2024-04-26T02:07:56.243 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 12.419877863, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 5.126330807, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 8.449405887, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.562969490, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 233564, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 5.765953742, 2024-04-26T02:07:56.244 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000024686 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 461401, 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.065340859, 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000141 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 2074368, 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 310944 2024-04-26T02:07:56.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:56.246 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 63093, 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 93.019395031, 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001474321 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 44750, 2024-04-26T02:07:56.247 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 6.814898878, 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000152288 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 214248, 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 186576, 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-04-26T02:07:56.248 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 186576, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1265.446662226, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.006782472 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 27672, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 8, 2024-04-26T02:07:56.249 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 126, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 10, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 7, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 2888, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 1299, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 1572, 2024-04-26T02:07:56.250 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 17, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 1032, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 769, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 181, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 181, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 13, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 333412, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 182088, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 27654, 2024-04-26T02:07:56.251 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 199, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 2499, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 1162, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 15, 2024-04-26T02:07:56.252 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 602, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 11, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 706, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 672, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 217550115, 2024-04-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 7, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-04-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 16, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 14136, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 91701, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 57129, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 22393, 2024-04-26T02:07:56.255 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 9863, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 141319 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 370, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 1850, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 990, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 1829, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 482, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-04-26T02:07:56.256 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 25014, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 25014, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-04-26T02:07:56.257 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 13, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 1510, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 20, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 17, 2024-04-26T02:07:56.258 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 134771, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 60654, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 49161, 2024-04-26T02:07:56.259 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 85610, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 11493, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 212, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 84, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 68, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 144, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 7, 2024-04-26T02:07:56.260 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 16, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 142045360, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 697903567, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 0, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 34394, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 205.458905233, 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.005973684 2024-04-26T02:07:56.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 0 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 1982, 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 27657, 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 25675, 2024-04-26T02:07:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 252, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 1211, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 959, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 2888, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 50096, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 47208, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 181, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 29108, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 28927, 2024-04-26T02:07:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 86860, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 223608 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 303426, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 258, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 214248, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 3828, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 7210, 2024-04-26T02:07:56.264 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 24939, 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 463.634707877, 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.018590749 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 7153, 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 72.502995651, 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.010136026 2024-04-26T02:07:56.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.266 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 52311, 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 124.522442919, 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002380425 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.267 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.270 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-04-26T02:07:56.270 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000416413, 2024-04-26T02:07:56.273 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000416413 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 70799, 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 296.475270246, 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.004187562 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 5284, 2024-04-26T02:07:56.274 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 69.811456033, 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.013211857 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.275 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.276 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.277 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 67, 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.775551421, 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.011575394 2024-04-26T02:07:56.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1008, 2024-04-26T02:07:56.279 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.843276774, 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001828647 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.281 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 25014, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 235.880544892, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.009429941 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-04-26T02:07:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-04-26T02:07:56.283 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 2, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 5, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 3, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 2, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 36840, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 18420, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 1292, 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-04-26T02:07:56.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:56.285 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:56.286 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-04-26T02:07:56.288 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 322863, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 910, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 88, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 1, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-04-26T02:07:56.289 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8040, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 168, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:56.290 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 7380121, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 59163, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-04-26T02:07:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 62732, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 700351966, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 33, 2024-04-26T02:07:56.292 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 62732, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 62732, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 432.615769860, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.006896253 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 62732, 2024-04-26T02:07:56.293 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 64117 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 62732, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 852, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 61880, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 28, 2024-04-26T02:07:56.294 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 21, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 844, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 35238, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 501, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 10, 2024-04-26T02:07:56.295 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 25265, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 42, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-04-26T02:07:56.296 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-04-26T02:07:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-04-26T02:07:56.298 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 132, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 40, 2024-04-26T02:07:56.299 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 12, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 1580, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 16, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 572 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.300 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 1, 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 1713 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-04-26T02:07:56.301 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 10, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 10, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 25014, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 25014, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.302 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 748096, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 176930493, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 748096, 2024-04-26T02:07:56.303 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 748096, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 176930493, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.304 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:56.305 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 62732, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 702883092, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 62732, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 702883092, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.306 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.307 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 62732, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 62732, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 62732, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 62732, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.308 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 25014, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 2531254, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 25014, 2024-04-26T02:07:56.309 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 844, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 2531254, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.310 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x557dc119e500": { 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 574, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 134771, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 693709263, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 134771, 2024-04-26T02:07:56.311 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 34394, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 693708689, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.312 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:56.313 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-26T02:07:56.948 INFO:tasks.check_counter:Daemon mds.i mds.exported=15 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds.imported=11 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=370 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1850 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-26T02:07:56.949 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T02:07:56.949 DEBUG:tasks.check_counter:Getting stats from l 2024-04-26T02:07:56.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-26T02:07:57.185 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T02:07:57.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:57 smithi191 ceph-mon[32745]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 11 op/s 2024-04-26T02:07:57.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:57 smithi183 ceph-mon[32752]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 11 op/s 2024-04-26T02:07:57.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:57 smithi052 ceph-mon[27851]: pgmap v728: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 11 op/s 2024-04-26T02:07:58.901 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 0, 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 2, 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 0, 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 0, 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 4, 2024-04-26T02:07:58.902 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 0, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.001016676, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.000218651, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:58.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.904 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 3, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 340, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 480, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 294048, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 6, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 3, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.019774191, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.014321231, 2024-04-26T02:07:58.905 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.001647005, 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 336, 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.035989896, 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000107112 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.906 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 294048 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T02:07:58.907 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 459, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 445, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 1798096, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 275120, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 2, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 1, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.076226711, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.025690899, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.047889358, 2024-04-26T02:07:58.908 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.010271592, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 445, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.020279709, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000045572 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:58.909 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 1798096, 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 275120 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 0, 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-04-26T02:07:58.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-04-26T02:07:58.911 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T02:07:58.912 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T02:07:58.913 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-04-26T02:07:58.914 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 148692, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 30, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-04-26T02:07:58.915 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T02:07:58.916 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 928, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 2, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-04-26T02:07:58.917 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 458, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 1730836, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-04-26T02:07:58.918 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 458, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 458, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 1730836, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-04-26T02:07:58.919 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:58.920 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-04-26T02:07:58.920 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-04-26T02:07:59.649 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-26T02:07:59.649 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-26T02:07:59.650 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-26T02:07:59.650 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-26T02:07:59.650 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-26T02:07:59.650 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-26T02:07:59.650 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-26T02:07:59.651 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T02:07:59.663 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T02:07:59.663 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T02:07:59.663 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:07:59.729 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T02:07:59.729 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:07:59.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-26T02:07:59.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:07:59 smithi052 ceph-mon[27851]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 8 op/s 2024-04-26T02:07:59.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:07:59 smithi191 ceph-mon[32745]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 8 op/s 2024-04-26T02:07:59.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:07:59 smithi183 ceph-mon[32752]: pgmap v729: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 8 op/s 2024-04-26T02:08:00.271 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:01.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:01 smithi052 ceph-mon[27851]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-26T02:08:01.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:01 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:08:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:01 smithi191 ceph-mon[32745]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-26T02:08:01.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:01 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:08:01.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:01 smithi183 ceph-mon[32752]: pgmap v730: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-26T02:08:01.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:01 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T02:08:02.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 -- 172.21.15.52:0/2875055867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 msgr2=0x7f5ae810f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:02.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 --2- 172.21.15.52:0/2875055867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae810f840 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009f90 tx=0x7f5ae000b330 comp rx=0 tx=0).stop 2024-04-26T02:08:02.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 -- 172.21.15.52:0/2875055867 shutdown_connections 2024-04-26T02:08:02.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 --2- 172.21.15.52:0/2875055867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae810f840 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 --2- 172.21.15.52:0/2875055867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 --2- 172.21.15.52:0/2875055867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 -- 172.21.15.52:0/2875055867 >> 172.21.15.52:0/2875055867 conn(0x7f5ae8078780 msgr2=0x7f5ae8078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:02.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.557+0000 7f5aedc00700 1 -- 172.21.15.52:0/2875055867 shutdown_connections 2024-04-26T02:08:02.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.558+0000 7f5aedc00700 1 -- 172.21.15.52:0/2875055867 wait complete. 2024-04-26T02:08:02.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 Processor -- start 2024-04-26T02:08:02.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 -- start start 2024-04-26T02:08:02.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae8101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:02.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae8102110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8112ba0 con 0x7f5ae806afb0 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8112a20 con 0x7f5ae806b970 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.559+0000 7f5aedc00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8112d20 con 0x7f5ae8108a20 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7e00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae8101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae8102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5aed200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:02.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5aed200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:45016/0 (socket says 172.21.15.52:45016) 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5aed200700 1 -- 172.21.15.52:0/289440186 learned_addr learned my addr 172.21.15.52:0/289440186 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 -- 172.21.15.52:0/289440186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 msgr2=0x7f5ae8102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 -- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 msgr2=0x7f5ae8101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae8101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7400700 1 -- 172.21.15.52:0/289440186 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae81a74c0 con 0x7f5ae806b970 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5aed200700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:02.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.560+0000 7f5ae7e00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5ae7400700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae8102110 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400eb90 tx=0x7f5ad400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400cc90 con 0x7f5ae806b970 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ad400ce30 con 0x7f5ae806b970 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4010430 con 0x7f5ae806b970 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae81a77b0 con 0x7f5ae806b970 2024-04-26T02:08:02.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.561+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae81a7ca0 con 0x7f5ae806b970 2024-04-26T02:08:02.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.562+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae8069730 con 0x7f5ae806b970 2024-04-26T02:08:02.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.565+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5ad4003ee0 con 0x7f5ae806b970 2024-04-26T02:08:02.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.567+0000 7f5ae4c00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 0x7f5acc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:02.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.567+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 8521+0+0 (secure 0 0 0) 0x7f5ad40d0b90 con 0x7f5ae806b970 2024-04-26T02:08:02.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.567+0000 7f5ae7e00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 0x7f5acc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:02.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.568+0000 7f5ae7e00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 0x7f5acc0a62f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009fd0 tx=0x7f5ad8009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:02.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.571+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ad409d1d0 con 0x7f5ae806b970 2024-04-26T02:08:02.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:02 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:08:02.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:02 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:08:02.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:02 smithi052 ceph-mon[27851]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:08:02.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.826+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5ae802df80 con 0x7f5ae806b970 2024-04-26T02:08:02.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.827+0000 7f5ae4c00700 1 -- 172.21.15.52:0/289440186 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v132) ==== 81+0+939 (secure 0 0 0) 0x7f5ad409cad0 con 0x7f5ae806b970 2024-04-26T02:08:02.826 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/2415060450 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/1613611936 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/3772941101 2024-04-27T01:32:45.509758+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/4041331855 2024-04-27T01:32:45.509758+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/3772941101 2024-04-27T01:32:45.509758+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2463815619 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/243168308 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/292261907 2024-04-27T01:32:45.509758+0000 2024-04-26T02:08:02.827 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/2415060450 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.828 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/496114326 2024-04-27T01:32:09.875117+0000 2024-04-26T02:08:02.828 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/459130865 2024-04-27T01:32:09.875117+0000 2024-04-26T02:08:02.828 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/496114326 2024-04-27T01:32:09.875117+0000 2024-04-26T02:08:02.828 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3936152513 2024-04-27T01:32:09.875117+0000 2024-04-26T02:08:02.829 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/483957502 2024-04-27T01:32:09.875117+0000 2024-04-26T02:08:02.829 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2167000831 2024-04-27T01:44:59.818840+0000 2024-04-26T02:08:02.829 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/320501252 2024-04-27T01:32:45.509758+0000 2024-04-26T02:08:02.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 msgr2=0x7f5acc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:02.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 0x7f5acc0a62f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009fd0 tx=0x7f5ad8009260 comp rx=0 tx=0).stop 2024-04-26T02:08:02.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 msgr2=0x7f5ae8102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae8102110 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400eb90 tx=0x7f5ad400eea0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 shutdown_connections 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f5acc0a3e30 0x7f5acc0a62f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ae8108a20 0x7f5ae8102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ae806afb0 0x7f5ae8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 --2- 172.21.15.52:0/289440186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ae806b970 0x7f5ae8102110 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.830+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 >> 172.21.15.52:0/289440186 conn(0x7f5ae8078780 msgr2=0x7f5ae80ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:02.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.831+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 shutdown_connections 2024-04-26T02:08:02.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:02.831+0000 7f5aedc00700 1 -- 172.21.15.52:0/289440186 wait complete. 2024-04-26T02:08:02.833 INFO:teuthology.orchestra.run.smithi052.stderr:listed 16 entries 2024-04-26T02:08:02.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:02 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:08:02.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:02 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:08:02.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:02 smithi191 ceph-mon[32745]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:08:02.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:02 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T02:08:02.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:02 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T02:08:02.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:02 smithi183 ceph-mon[32752]: from='mgr.25039 ' entity='mgr.x' 2024-04-26T02:08:03.715 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T02:08:03.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:03.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T02:08:03.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:03 smithi052 ceph-mon[27851]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 123 KiB/s wr, 29 op/s 2024-04-26T02:08:03.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:03 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/289440186' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:03.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:03 smithi191 ceph-mon[32745]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 123 KiB/s wr, 29 op/s 2024-04-26T02:08:03.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:03 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/289440186' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:03.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:03 smithi183 ceph-mon[32752]: pgmap v731: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 123 KiB/s wr, 29 op/s 2024-04-26T02:08:03.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:03 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/289440186' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T02:08:04.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T02:08:04.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.179 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T02:08:04.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.281 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-26T02:08:04.397 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T02:08:04.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.398 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T02:08:04.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-04-26T02:08:04.578 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T02:08:04.578 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-04-26T02:08:04.578 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T02:08:04.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/821a2a7d6b485d916ec0b07aecd10c051d46794e68699c304fcb73a01159d6a3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/821a2a7d6b485d916ec0b07aecd10c051d46794e68699c304fcb73a01159d6a3/diff,workdir=/var/lib/containers/storage/overlay/821a2a7d6b485d916ec0b07aecd10c051d46794e68699c304fcb73a01159d6a3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4a813a6c1c3eba2222d93a53c357f3a80aa5859b20ed590efb71e89a0aa1b194/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/4a813a6c1c3eba2222d93a53c357f3a80aa5859b20ed590efb71e89a0aa1b194/diff,workdir=/var/lib/containers/storage/overlay/4a813a6c1c3eba2222d93a53c357f3a80aa5859b20ed590efb71e89a0aa1b194/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/a543f4eba42187bc057ba9affa32ffcf7b757ac992881e0fbf1544aecdb48ab0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/a543f4eba42187bc057ba9affa32ffcf7b757ac992881e0fbf1544aecdb48ab0/diff,workdir=/var/lib/containers/storage/overlay/a543f4eba42187bc057ba9affa32ffcf7b757ac992881e0fbf1544aecdb48ab0/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c04896c988a4da35bdcd5710e737eb2e6bd3e696010dde8100eb27640ffe2cc3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/c04896c988a4da35bdcd5710e737eb2e6bd3e696010dde8100eb27640ffe2cc3/diff,workdir=/var/lib/containers/storage/overlay/c04896c988a4da35bdcd5710e737eb2e6bd3e696010dde8100eb27640ffe2cc3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0c461cdab2c64de64aef97d8a973204f66776eecde763a437c4c2bdacf5de0fa/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/0c461cdab2c64de64aef97d8a973204f66776eecde763a437c4c2bdacf5de0fa/diff,workdir=/var/lib/containers/storage/overlay/0c461cdab2c64de64aef97d8a973204f66776eecde763a437c4c2bdacf5de0fa/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9b2212b905513c39e01a495f79db687c0cacdcf9deb7d59b193b5f1926204b2e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/9b2212b905513c39e01a495f79db687c0cacdcf9deb7d59b193b5f1926204b2e/diff,workdir=/var/lib/containers/storage/overlay/9b2212b905513c39e01a495f79db687c0cacdcf9deb7d59b193b5f1926204b2e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bad1dc0d207b52013ca4b09f2044a95feac88523f514bb1a9980fea1ae6aa258/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/bad1dc0d207b52013ca4b09f2044a95feac88523f514bb1a9980fea1ae6aa258/diff,workdir=/var/lib/containers/storage/overlay/bad1dc0d207b52013ca4b09f2044a95feac88523f514bb1a9980fea1ae6aa258/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/89957e5af4b429e7b9beebae167dd128a488af43d4122356a7344407ddb57dba/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/89957e5af4b429e7b9beebae167dd128a488af43d4122356a7344407ddb57dba/diff,workdir=/var/lib/containers/storage/overlay/89957e5af4b429e7b9beebae167dd128a488af43d4122356a7344407ddb57dba/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a470eb315df2ad10eca675b2f6c34229f28698fdf0cb82799f8e2aa66fd3192e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/a470eb315df2ad10eca675b2f6c34229f28698fdf0cb82799f8e2aa66fd3192e/diff,workdir=/var/lib/containers/storage/overlay/a470eb315df2ad10eca675b2f6c34229f28698fdf0cb82799f8e2aa66fd3192e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/065636f900a5056b69b344040426d24d318d7029ca8719a42c2fa81e24765b68/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/2XKIMXPC3PS4WYNPJ3FEU2PN4B:/var/lib/containers/storage/overlay/l/VXRM5A67JC76NKNSISEWCPSVTI,upperdir=/var/lib/containers/storage/overlay/065636f900a5056b69b344040426d24d318d7029ca8719a42c2fa81e24765b68/diff,workdir=/var/lib/containers/storage/overlay/065636f900a5056b69b344040426d24d318d7029ca8719a42c2fa81e24765b68/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 2024-04-26T02:08:04.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.617 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-04-26T02:08:04.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T02:08:04.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-04-26T02:08:04.900 INFO:teuthology.orchestra.run.smithi052.stderr:Cannot find device "ceph-brx" 2024-04-26T02:08:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:08:04.905 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T02:08:04.916 INFO:tasks.cephadm:Teardown begin 2024-04-26T02:08:04.916 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T02:08:04.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:08:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:05.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:05 smithi052 ceph-mon[27851]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2024-04-26T02:08:05.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:05 smithi191 ceph-mon[32745]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2024-04-26T02:08:05.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:05 smithi183 ceph-mon[32752]: pgmap v732: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2024-04-26T02:08:07.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:07 smithi052 ceph-mon[27851]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 250 KiB/s wr, 30 op/s 2024-04-26T02:08:07.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:07 smithi052 ceph-mon[27851]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 -- 172.21.15.52:0/372425219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff764098730 msgr2=0x7ff764098bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 --2- 172.21.15.52:0/372425219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff764098730 0x7ff764098bb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0694d0 tx=0x7ff76c051850 comp rx=0 tx=0).stop 2024-04-26T02:08:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 -- 172.21.15.52:0/372425219 shutdown_connections 2024-04-26T02:08:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 --2- 172.21.15.52:0/372425219 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff764098730 0x7ff764098bb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 --2- 172.21.15.52:0/372425219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff76409ad30 0x7ff764004d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 --2- 172.21.15.52:0/372425219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.751+0000 7ff76b200700 1 -- 172.21.15.52:0/372425219 >> 172.21.15.52:0/372425219 conn(0x7ff76400a7b0 msgr2=0x7ff76400abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.752+0000 7ff76b200700 1 -- 172.21.15.52:0/372425219 shutdown_connections 2024-04-26T02:08:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.752+0000 7ff76b200700 1 -- 172.21.15.52:0/372425219 wait complete. 2024-04-26T02:08:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.756+0000 7ff76b200700 1 Processor -- start 2024-04-26T02:08:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.756+0000 7ff76b200700 1 -- start start 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.757+0000 7ff76b200700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.757+0000 7ff76b200700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff764098730 0x7ff7640a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.757+0000 7ff76b200700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 0x7ff7640a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff76b200700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff7640ac3d0 con 0x7ff764098730 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff76b200700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff7640ac250 con 0x7ff76409ad30 2024-04-26T02:08:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff76b200700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7640ac550 con 0x7ff764005250 2024-04-26T02:08:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff769e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff769e00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640a9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.52:44282/0 (socket says 172.21.15.52:44282) 2024-04-26T02:08:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff769e00700 1 -- 172.21.15.52:0/2281527168 learned_addr learned my addr 172.21.15.52:0/2281527168 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff769400700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff764098730 0x7ff7640a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.758+0000 7ff76a800700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 0x7ff7640a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.759+0000 7ff76a800700 1 -- 172.21.15.52:0/2281527168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 msgr2=0x7ff7640a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.759+0000 7ff76a800700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640a9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.759+0000 7ff76a800700 1 -- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff764098730 msgr2=0x7ff7640a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.759+0000 7ff76a800700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff764098730 0x7ff7640a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.759+0000 7ff76a800700 1 -- 172.21.15.52:0/2281527168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7640a2ad0 con 0x7ff76409ad30 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.760+0000 7ff76a800700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 0x7ff7640a21d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff76c051580 tx=0x7ff76c06b340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.761+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c08e020 con 0x7ff76409ad30 2024-04-26T02:08:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.761+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff76c06c1e0 con 0x7ff76409ad30 2024-04-26T02:08:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.761+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c08a430 con 0x7ff76409ad30 2024-04-26T02:08:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.761+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7640a2d60 con 0x7ff76409ad30 2024-04-26T02:08:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.761+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7640a6e00 con 0x7ff76409ad30 2024-04-26T02:08:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.765+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff764095270 con 0x7ff76409ad30 2024-04-26T02:08:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.766+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff76c091020 con 0x7ff76409ad30 2024-04-26T02:08:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.770+0000 7ff762a00700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 0x7ff7500a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:07.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.770+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8565+0+0 (secure 0 0 0) 0x7ff76c13eef0 con 0x7ff76409ad30 2024-04-26T02:08:07.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.770+0000 7ff769e00700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 0x7ff7500a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:07.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.771+0000 7ff769e00700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 0x7ff7500a62f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009ca0 tx=0x7ff75c009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:07.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:07.774+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff76c098020 con 0x7ff76409ad30 2024-04-26T02:08:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:07 smithi191 ceph-mon[32745]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 250 KiB/s wr, 30 op/s 2024-04-26T02:08:07.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:07 smithi191 ceph-mon[32745]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:07 smithi183 ceph-mon[32752]: pgmap v733: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 250 KiB/s wr, 30 op/s 2024-04-26T02:08:07.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:07 smithi183 ceph-mon[32752]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T02:08:08.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.285+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7640a7090 con 0x7ff76409ad30 2024-04-26T02:08:08.286 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:08:08.286 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":75,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:07:50.988688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27443":{"gid":27443,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1603134393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1603134393},{"type":"v1","addr":"172.21.15.183:6837","nonce":1603134393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T02:08:08.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.286+0000 7ff762a00700 1 -- 172.21.15.52:0/2281527168 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+11885 (secure 0 0 0) 0x7ff76c10b5f0 con 0x7ff76409ad30 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.289+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 msgr2=0x7ff7500a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.289+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 0x7ff7500a62f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009ca0 tx=0x7ff75c009260 comp rx=0 tx=0).stop 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 msgr2=0x7ff7640a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 0x7ff7640a21d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff76c051580 tx=0x7ff76c06b340 comp rx=0 tx=0).stop 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 shutdown_connections 2024-04-26T02:08:08.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7ff7500a3e30 0x7ff7500a62f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff76409ad30 0x7ff7640a21d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff764005250 0x7ff7640a9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 --2- 172.21.15.52:0/2281527168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff764098730 0x7ff7640a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.290+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 >> 172.21.15.52:0/2281527168 conn(0x7ff76400a7b0 msgr2=0x7ff7640972f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.291+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 shutdown_connections 2024-04-26T02:08:08.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:08.291+0000 7ff76b200700 1 -- 172.21.15.52:0/2281527168 wait complete. 2024-04-26T02:08:08.291 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 75 2024-04-26T02:08:08.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:08 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2281527168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:08.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:08 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2281527168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:08.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:08 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2281527168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:09.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-26T02:08:09.799 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:09.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:09 smithi191 ceph-mon[32745]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:09.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:09 smithi183 ceph-mon[32752]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:09.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:09 smithi052 ceph-mon[27851]: pgmap v735: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:11.858 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:11 smithi052 ceph-mon[27851]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:11.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:11 smithi191 ceph-mon[32745]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:11.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:11 smithi183 ceph-mon[32752]: pgmap v736: 97 pgs: 97 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 293 KiB/s wr, 34 op/s 2024-04-26T02:08:12.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 -- 172.21.15.52:0/1578391111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8104ec0 msgr2=0x7f29f81052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 --2- 172.21.15.52:0/1578391111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8104ec0 0x7f29f81052a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0099c0 tx=0x7f29ec009cd0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 -- 172.21.15.52:0/1578391111 shutdown_connections 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 --2- 172.21.15.52:0/1578391111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8109fa0 0x7f29f8111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 --2- 172.21.15.52:0/1578391111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8104ec0 0x7f29f81052a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 --2- 172.21.15.52:0/1578391111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8105870 0x7f29f81098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 -- 172.21.15.52:0/1578391111 >> 172.21.15.52:0/1578391111 conn(0x7f29f8100750 msgr2=0x7f29f8102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:12.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 -- 172.21.15.52:0/1578391111 shutdown_connections 2024-04-26T02:08:12.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.095+0000 7f29ffe00700 1 -- 172.21.15.52:0/1578391111 wait complete. 2024-04-26T02:08:12.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.096+0000 7f29ffe00700 1 Processor -- start 2024-04-26T02:08:12.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 -- start start 2024-04-26T02:08:12.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8104ec0 0x7f29f819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:12.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:12.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8109fa0 0x7f29f819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f29f8114ea0 con 0x7f29f8105870 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f29f8114d20 con 0x7f29f8104ec0 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29ffe00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29f8115020 con 0x7f29f8109fa0 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.097+0000 7f29fca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35028/0 (socket says 172.21.15.52:35028) 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 -- 172.21.15.52:0/3147611478 learned_addr learned my addr 172.21.15.52:0/3147611478 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fd400700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8104ec0 0x7f29f819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:12.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fde00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8109fa0 0x7f29f819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:12.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 -- 172.21.15.52:0/3147611478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8109fa0 msgr2=0x7f29f819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8109fa0 0x7f29f819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 -- 172.21.15.52:0/3147611478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8104ec0 msgr2=0x7f29f819b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8104ec0 0x7f29f819b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 -- 172.21.15.52:0/3147611478 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f81a0410 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.098+0000 7f29fca00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f29e400e9f0 tx=0x7f29e400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.099+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4004a30 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.099+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e400cc70 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.099+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f81a0700 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.099+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4005530 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.099+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f81a7f80 con 0x7f29f8105870 2024-04-26T02:08:12.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.100+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f8069730 con 0x7f29f8105870 2024-04-26T02:08:12.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.102+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f29e4004bd0 con 0x7f29f8105870 2024-04-26T02:08:12.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.103+0000 7f29f6000700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 0x7f29e00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:12.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.103+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8565+0+0 (secure 0 0 0) 0x7f29e40d0c00 con 0x7f29f8105870 2024-04-26T02:08:12.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.104+0000 7f29fd400700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 0x7f29e00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:12.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.104+0000 7f29fd400700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 0x7f29e00a62f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f29ec000c00 tx=0x7f29ec005fb0 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:12.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.108+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29e400ce20 con 0x7f29f8105870 2024-04-26T02:08:12.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.386+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29f819c760 con 0x7f29f8105870 2024-04-26T02:08:12.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.388+0000 7f29f6000700 1 -- 172.21.15.52:0/3147611478 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+11885 (secure 0 0 0) 0x7f29e409d210 con 0x7f29f8105870 2024-04-26T02:08:12.388 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:08:12.388 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14361,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/1730690276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1730690276},{"type":"v1","addr":"172.21.15.52:6835","nonce":1730690276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/2681513710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":2681513710},{"type":"v1","addr":"172.21.15.52:6837","nonce":2681513710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14397,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/3379340628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3379340628},{"type":"v1","addr":"172.21.15.52:6839","nonce":3379340628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/4135915621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":4135915621},{"type":"v1","addr":"172.21.15.191:6833","nonce":4135915621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/2084061199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":2084061199},{"type":"v1","addr":"172.21.15.183:6833","nonce":2084061199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/2708532468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2708532468},{"type":"v1","addr":"172.21.15.183:6835","nonce":2708532468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":75,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T01:41:06.424614+0000","modified":"2024-04-26T02:07:50.988688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24524,"mds_1":24512,"mds_2":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24472":{"gid":24472,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/234906181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":234906181},{"type":"v1","addr":"172.21.15.191:6835","nonce":234906181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6837/2929119914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2929119914},{"type":"v1","addr":"172.21.15.191:6837","nonce":2929119914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.191:6839/1544004799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1544004799},{"type":"v1","addr":"172.21.15.191:6839","nonce":1544004799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.52:6841/1834450967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":1834450967},{"type":"v1","addr":"172.21.15.52:6841","nonce":1834450967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.183:6839/1410734884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":1410734884},{"type":"v1","addr":"172.21.15.183:6839","nonce":1410734884}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27443":{"gid":27443,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.183:6837/1603134393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":1603134393},{"type":"v1","addr":"172.21.15.183:6837","nonce":1603134393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24512,24524,24496]},"id":1}]} 2024-04-26T02:08:12.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 msgr2=0x7f29e00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 0x7f29e00a62f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f29ec000c00 tx=0x7f29ec005fb0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 msgr2=0x7f29f819b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f29e400e9f0 tx=0x7f29e400ed00 comp rx=0 tx=0).stop 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 shutdown_connections 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f29e00a3e30 0x7f29e00a62f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29f8109fa0 0x7f29f819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f29f8104ec0 0x7f29f819b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 --2- 172.21.15.52:0/3147611478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29f8105870 0x7f29f819b990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:12.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.391+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 >> 172.21.15.52:0/3147611478 conn(0x7f29f8100750 msgr2=0x7f29f8101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:12.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.392+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 shutdown_connections 2024-04-26T02:08:12.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:12.392+0000 7f29ffe00700 1 -- 172.21.15.52:0/3147611478 wait complete. 2024-04-26T02:08:12.393 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 75 2024-04-26T02:08:12.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:12 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3147611478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:12.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:12 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3147611478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:12.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:12 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3147611478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T02:08:13.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T02:08:13.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:13 smithi052 ceph-mon[27851]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-04-26T02:08:13.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:13 smithi052 ceph-mon[27851]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:13 smithi191 ceph-mon[32745]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-04-26T02:08:13.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:13 smithi191 ceph-mon[32745]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:13.906 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:13.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:13 smithi183 ceph-mon[32752]: pgmap v737: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-04-26T02:08:13.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:13 smithi183 ceph-mon[32752]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T02:08:15.835 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:15 smithi052 ceph-mon[27851]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-26T02:08:15.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:15 smithi191 ceph-mon[32745]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-26T02:08:15.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:15 smithi183 ceph-mon[32752]: pgmap v739: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-04-26T02:08:16.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 -- 172.21.15.52:0/2115188733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 msgr2=0x7f87c0113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 --2- 172.21.15.52:0/2115188733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c0113380 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009990 tx=0x7f87ac009ca0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 -- 172.21.15.52:0/2115188733 shutdown_connections 2024-04-26T02:08:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 --2- 172.21.15.52:0/2115188733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 0x7f87c0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 --2- 172.21.15.52:0/2115188733 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c0077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 --2- 172.21.15.52:0/2115188733 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c0113380 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 -- 172.21.15.52:0/2115188733 >> 172.21.15.52:0/2115188733 conn(0x7f87c01006f0 msgr2=0x7f87c0102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:16.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 -- 172.21.15.52:0/2115188733 shutdown_connections 2024-04-26T02:08:16.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.451+0000 7f87c7600700 1 -- 172.21.15.52:0/2115188733 wait complete. 2024-04-26T02:08:16.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.452+0000 7f87c7600700 1 Processor -- start 2024-04-26T02:08:16.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.452+0000 7f87c7600700 1 -- start start 2024-04-26T02:08:16.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c01a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 0x7f87c019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f87c01192a0 con 0x7f87c0077890 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f87c0119120 con 0x7f87c0076f70 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c7600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f87c0119420 con 0x7f87c01138c0 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c4c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c4c00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33740/0 (socket says 172.21.15.52:33740) 2024-04-26T02:08:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c4c00700 1 -- 172.21.15.52:0/1696749523 learned_addr learned my addr 172.21.15.52:0/1696749523 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87bfe00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c01a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.453+0000 7f87c5600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 0x7f87c019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 -- 172.21.15.52:0/1696749523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 msgr2=0x7f87c019f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 0x7f87c019f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 -- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 msgr2=0x7f87c01a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c01a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 -- 172.21.15.52:0/1696749523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c019fd80 con 0x7f87c0076f70 2024-04-26T02:08:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c4c00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87b400e9a0 tx=0x7f87b400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400ca10 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b400cbb0 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.454+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c01a0070 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.455+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4018680 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.455+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c01ac2e0 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.456+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c0069730 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.459+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f87b4010430 con 0x7f87c0076f70 2024-04-26T02:08:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.460+0000 7f87bd600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 0x7f87a80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:16.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.460+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 8521+0+0 (secure 0 0 0) 0x7f87b40d0d10 con 0x7f87c0076f70 2024-04-26T02:08:16.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.461+0000 7f87bfe00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 0x7f87a80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:16.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.461+0000 7f87bfe00700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 0x7f87a80a6270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87c0079be0 tx=0x7f87ac00b260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:16.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.462+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87b409d350 con 0x7f87c0076f70 2024-04-26T02:08:16.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.698+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87c002df80 con 0x7f87c0076f70 2024-04-26T02:08:16.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.700+0000 7f87bd600700 1 -- 172.21.15.52:0/1696749523 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+23138 (secure 0 0 0) 0x7f87b409cc50 con 0x7f87c0076f70 2024-04-26T02:08:16.700 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-04-26T02:08:16.701 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":134,"fsid":"8b150f36-036c-11ef-bc93-c7b262605968","created":"2024-04-26T01:31:27.358609+0000","modified":"2024-04-26T02:08:13.542429+0000","last_up_change":"2024-04-26T01:39:46.810474+0000","last_in_change":"2024-04-26T01:39:25.865504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:35:33.675964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T01:40:57.761799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T01:41:02.168712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":40,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db649844-3866-493f-aa45-bbdcc973f556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6803","nonce":4212634049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6805","nonce":4212634049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6809","nonce":4212634049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":4212634049},{"type":"v1","addr":"172.21.15.52:6807","nonce":4212634049}]},"public_addr":"172.21.15.52:6803/4212634049","cluster_addr":"172.21.15.52:6805/4212634049","heartbeat_back_addr":"172.21.15.52:6809/4212634049","heartbeat_front_addr":"172.21.15.52:6807/4212634049","state":["exists","up"]},{"osd":1,"uuid":"285c6829-a343-4ccb-9883-8a1e24fb28ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6801","nonce":3104160515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6803","nonce":3104160515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6807","nonce":3104160515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":3104160515},{"type":"v1","addr":"172.21.15.183:6805","nonce":3104160515}]},"public_addr":"172.21.15.183:6801/3104160515","cluster_addr":"172.21.15.183:6803/3104160515","heartbeat_back_addr":"172.21.15.183:6807/3104160515","heartbeat_front_addr":"172.21.15.183:6805/3104160515","state":["exists","up"]},{"osd":2,"uuid":"bc51fafd-4a20-45b8-9b1f-d22980ab4b86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6801","nonce":932162512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6803","nonce":932162512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6807","nonce":932162512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":932162512},{"type":"v1","addr":"172.21.15.191:6805","nonce":932162512}]},"public_addr":"172.21.15.191:6801/932162512","cluster_addr":"172.21.15.191:6803/932162512","heartbeat_back_addr":"172.21.15.191:6807/932162512","heartbeat_front_addr":"172.21.15.191:6805/932162512","state":["exists","up"]},{"osd":3,"uuid":"376c8ee8-7497-4f7d-b3fe-d61a6cc2b56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6811","nonce":1589015031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6813","nonce":1589015031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6817","nonce":1589015031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":1589015031},{"type":"v1","addr":"172.21.15.52:6815","nonce":1589015031}]},"public_addr":"172.21.15.52:6811/1589015031","cluster_addr":"172.21.15.52:6813/1589015031","heartbeat_back_addr":"172.21.15.52:6817/1589015031","heartbeat_front_addr":"172.21.15.52:6815/1589015031","state":["exists","up"]},{"osd":4,"uuid":"014075df-27cd-4f11-8542-ecc8ac24a820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6809","nonce":2309146877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6811","nonce":2309146877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6815","nonce":2309146877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2309146877},{"type":"v1","addr":"172.21.15.183:6813","nonce":2309146877}]},"public_addr":"172.21.15.183:6809/2309146877","cluster_addr":"172.21.15.183:6811/2309146877","heartbeat_back_addr":"172.21.15.183:6815/2309146877","heartbeat_front_addr":"172.21.15.183:6813/2309146877","state":["exists","up"]},{"osd":5,"uuid":"59ea2f70-c81c-4121-890e-1216b8b142f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6809","nonce":3556847761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6811","nonce":3556847761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6815","nonce":3556847761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3556847761},{"type":"v1","addr":"172.21.15.191:6813","nonce":3556847761}]},"public_addr":"172.21.15.191:6809/3556847761","cluster_addr":"172.21.15.191:6811/3556847761","heartbeat_back_addr":"172.21.15.191:6815/3556847761","heartbeat_front_addr":"172.21.15.191:6813/3556847761","state":["exists","up"]},{"osd":6,"uuid":"b37abf58-d1b9-4290-bcc1-c1163dfc256f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6819","nonce":915546379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6821","nonce":915546379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6825","nonce":915546379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":915546379},{"type":"v1","addr":"172.21.15.52:6823","nonce":915546379}]},"public_addr":"172.21.15.52:6819/915546379","cluster_addr":"172.21.15.52:6821/915546379","heartbeat_back_addr":"172.21.15.52:6825/915546379","heartbeat_front_addr":"172.21.15.52:6823/915546379","state":["exists","up"]},{"osd":7,"uuid":"569c973e-98a6-4957-bb2e-3eb411ea369c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6817","nonce":1224783051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6819","nonce":1224783051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6823","nonce":1224783051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1224783051},{"type":"v1","addr":"172.21.15.183:6821","nonce":1224783051}]},"public_addr":"172.21.15.183:6817/1224783051","cluster_addr":"172.21.15.183:6819/1224783051","heartbeat_back_addr":"172.21.15.183:6823/1224783051","heartbeat_front_addr":"172.21.15.183:6821/1224783051","state":["exists","up"]},{"osd":8,"uuid":"85ed3a78-d945-4ea1-a9a6-1391924890fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6817","nonce":3371568470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6819","nonce":3371568470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6823","nonce":3371568470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":3371568470},{"type":"v1","addr":"172.21.15.191:6821","nonce":3371568470}]},"public_addr":"172.21.15.191:6817/3371568470","cluster_addr":"172.21.15.191:6819/3371568470","heartbeat_back_addr":"172.21.15.191:6823/3371568470","heartbeat_front_addr":"172.21.15.191:6821/3371568470","state":["exists","up"]},{"osd":9,"uuid":"dd9bae00-f288-4771-a5c7-da5fb4c4f7d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6827","nonce":405746213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6829","nonce":405746213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6833","nonce":405746213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":405746213},{"type":"v1","addr":"172.21.15.52:6831","nonce":405746213}]},"public_addr":"172.21.15.52:6827/405746213","cluster_addr":"172.21.15.52:6829/405746213","heartbeat_back_addr":"172.21.15.52:6833/405746213","heartbeat_front_addr":"172.21.15.52:6831/405746213","state":["exists","up"]},{"osd":10,"uuid":"4d3f07a6-f55c-443d-9e9d-16036d02a9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6825","nonce":2192270029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6827","nonce":2192270029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6831","nonce":2192270029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":2192270029},{"type":"v1","addr":"172.21.15.183:6829","nonce":2192270029}]},"public_addr":"172.21.15.183:6825/2192270029","cluster_addr":"172.21.15.183:6827/2192270029","heartbeat_back_addr":"172.21.15.183:6831/2192270029","heartbeat_front_addr":"172.21.15.183:6829/2192270029","state":["exists","up"]},{"osd":11,"uuid":"6c795e48-0fbd-4ff6-b778-f57b2a07a4fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6825","nonce":1249163110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6827","nonce":1249163110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6831","nonce":1249163110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1249163110},{"type":"v1","addr":"172.21.15.191:6829","nonce":1249163110}]},"public_addr":"172.21.15.191:6825/1249163110","cluster_addr":"172.21.15.191:6827/1249163110","heartbeat_back_addr":"172.21.15.191:6831/1249163110","heartbeat_front_addr":"172.21.15.191:6829/1249163110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:34:39.295648+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:04.739600+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:35:29.303323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:01.768286+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:27.447641+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:36:52.583410+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:26.336512+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:37:52.142794+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:17.714960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:38:51.780569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:18.682411+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:39:43.774713+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/1613611936":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/4041331855":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/3772941101":"2024-04-27T01:32:45.509758+0000","172.21.15.52:0/2463815619":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/243168308":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/292261907":"2024-04-27T01:32:45.509758+0000","172.21.15.52:6800/2415060450":"2024-04-27T01:44:59.818840+0000","172.21.15.52:6801/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/459130865":"2024-04-27T01:32:09.875117+0000","172.21.15.52:6800/496114326":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/3936152513":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/483957502":"2024-04-27T01:32:09.875117+0000","172.21.15.52:0/2167000831":"2024-04-27T01:44:59.818840+0000","172.21.15.52:0/320501252":"2024-04-27T01:32:45.509758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":40,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T02:08:16.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.703+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 msgr2=0x7f87a80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:16.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.703+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 0x7f87a80a6270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87c0079be0 tx=0x7f87ac00b260 comp rx=0 tx=0).stop 2024-04-26T02:08:16.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.703+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 msgr2=0x7f87c01a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:16.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.703+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87b400e9a0 tx=0x7f87b400ecb0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 shutdown_connections 2024-04-26T02:08:16.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f87a80a3db0 0x7f87a80a6270 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f87c01138c0 0x7f87c019f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87c0076f70 0x7f87c01a5410 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 --2- 172.21.15.52:0/1696749523 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87c0077890 0x7f87c01a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:16.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 >> 172.21.15.52:0/1696749523 conn(0x7f87c01006f0 msgr2=0x7f87c0102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:16.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 shutdown_connections 2024-04-26T02:08:16.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:16.704+0000 7f87c7600700 1 -- 172.21.15.52:0/1696749523 wait complete. 2024-04-26T02:08:17.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-26T02:08:17.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:17 smithi052 ceph-mon[27851]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-26T02:08:17.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:17 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1696749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:17.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:17 smithi191 ceph-mon[32745]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-26T02:08:17.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:17 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1696749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:17.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:17 smithi183 ceph-mon[32752]: pgmap v740: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-04-26T02:08:17.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:17 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1696749523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T02:08:17.989 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:19.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:19 smithi052 ceph-mon[27851]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:19.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:19 smithi191 ceph-mon[32745]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:19.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:19 smithi183 ceph-mon[32752]: pgmap v741: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:20.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.388+0000 7efce9000700 1 -- 172.21.15.52:0/652628021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 msgr2=0x7efce4115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:20.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.388+0000 7efce9000700 1 --2- 172.21.15.52:0/652628021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce4115f80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009fc0 tx=0x7efcdc00b330 comp rx=0 tx=0).stop 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 -- 172.21.15.52:0/652628021 shutdown_connections 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 --2- 172.21.15.52:0/652628021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce4115f80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 --2- 172.21.15.52:0/652628021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce41092d0 0x7efce41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 --2- 172.21.15.52:0/652628021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce4109c80 0x7efce410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 -- 172.21.15.52:0/652628021 >> 172.21.15.52:0/652628021 conn(0x7efce4076210 msgr2=0x7efce4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 -- 172.21.15.52:0/652628021 shutdown_connections 2024-04-26T02:08:20.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.389+0000 7efce9000700 1 -- 172.21.15.52:0/652628021 wait complete. 2024-04-26T02:08:20.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.390+0000 7efce9000700 1 Processor -- start 2024-04-26T02:08:20.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.390+0000 7efce9000700 1 -- start start 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 0x7efce419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce4109c80 0x7efce419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efce41192a0 con 0x7efce410e3b0 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efce4119120 con 0x7efce41092d0 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce9000700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efce4119420 con 0x7efce4109c80 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:20.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 0x7efce419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:20.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efcdbe00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce4109c80 0x7efce419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:20.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35914/0 (socket says 172.21.15.52:35914) 2024-04-26T02:08:20.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce2800700 1 -- 172.21.15.52:0/2702270433 learned_addr learned my addr 172.21.15.52:0/2702270433 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:20.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.391+0000 7efce1e00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 0x7efce419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33764/0 (socket says 172.21.15.52:33764) 2024-04-26T02:08:20.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 -- 172.21.15.52:0/2702270433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce4109c80 msgr2=0x7efce419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:20.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce4109c80 0x7efce419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 -- 172.21.15.52:0/2702270433 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 msgr2=0x7efce419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 0x7efce419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 -- 172.21.15.52:0/2702270433 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce41a46e0 con 0x7efce410e3b0 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.392+0000 7efce2800700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009790 tx=0x7efcdc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.393+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc021070 con 0x7efce410e3b0 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.393+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcdc007ac0 con 0x7efce410e3b0 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.393+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc005040 con 0x7efce410e3b0 2024-04-26T02:08:20.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.393+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce41a4970 con 0x7efce410e3b0 2024-04-26T02:08:20.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.393+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce41ac250 con 0x7efce410e3b0 2024-04-26T02:08:20.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.394+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce4069730 con 0x7efce410e3b0 2024-04-26T02:08:20.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.396+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7efcdc024070 con 0x7efce410e3b0 2024-04-26T02:08:20.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.397+0000 7efcdaa00700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 0x7efcc80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:20.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.398+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 8521+0+0 (secure 0 0 0) 0x7efcdc00f020 con 0x7efce410e3b0 2024-04-26T02:08:20.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.398+0000 7efce1e00700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 0x7efcc80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:20.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.399+0000 7efce1e00700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 0x7efcc80a6270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efcd4005fd0 tx=0x7efcd4009310 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:20.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.402+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efcdc02b020 con 0x7efce410e3b0 2024-04-26T02:08:20.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.688+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efce402df80 con 0x7efce410e3b0 2024-04-26T02:08:20.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.693+0000 7efcdaa00700 1 -- 172.21.15.52:0/2702270433 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v76) ==== 161+0+0 (secure 0 0 0) 0x7efcdc09e300 con 0x7efce410e3b0 2024-04-26T02:08:20.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.697+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 msgr2=0x7efcc80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:20.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.697+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 0x7efcc80a6270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efcd4005fd0 tx=0x7efcd4009310 comp rx=0 tx=0).stop 2024-04-26T02:08:20.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.697+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 msgr2=0x7efce41a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:20.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.697+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009790 tx=0x7efcdc00cc30 comp rx=0 tx=0).stop 2024-04-26T02:08:20.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 shutdown_connections 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7efcc80a3db0 0x7efcc80a6270 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efcd4005fd0 tx=0x7efcd4009310 comp rx=0 tx=0).stop 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efce410e3b0 0x7efce41a3fa0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efce41092d0 0x7efce419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 --2- 172.21.15.52:0/2702270433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce4109c80 0x7efce419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 >> 172.21.15.52:0/2702270433 conn(0x7efce4076210 msgr2=0x7efce4076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 shutdown_connections 2024-04-26T02:08:20.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:20.698+0000 7efce9000700 1 -- 172.21.15.52:0/2702270433 wait complete. 2024-04-26T02:08:20.700 INFO:teuthology.orchestra.run.smithi052.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T02:08:20.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a[27847]: 2024-04-26T02:08:20.689+0000 7fe8acc00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:20.963 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:08:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:20.964 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:08:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:20.964 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:08:20 smithi052 ceph-mds[70463]: -- 172.21.15.52:0/1435359927 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e252149a40 con 0x55e25211d000 2024-04-26T02:08:20.964 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:08:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: starting mds.a at 2024-04-26T02:08:21.153 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:21.154 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:21.154 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-mds[63320]: -- 172.21.15.191:0/3720254890 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56297e477a40 con 0x56297e44a800 2024-04-26T02:08:21.154 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: starting mds.c at 2024-04-26T02:08:21.154 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-mds[63008]: -- 172.21.15.191:0/850963173 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559691af9a40 con 0x559691acc800 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: starting mds.f at 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-mds[63628]: -- 172.21.15.191:0/1878495351 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f939a9a40 con 0x559f9397c800 2024-04-26T02:08:21.155 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:08:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: starting mds.i at 2024-04-26T02:08:21.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:21.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:21.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-mds[63453]: -- 172.21.15.183:0/2770442722 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617ca69fa40 con 0x5617ca672800 2024-04-26T02:08:21.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: starting mds.b at 2024-04-26T02:08:21.177 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setuser ceph since I am not root 2024-04-26T02:08:21.177 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: ignoring --setgroup ceph since I am not root 2024-04-26T02:08:21.177 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-mds[63131]: -- 172.21.15.183:0/2382799343 <== mon.2 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a871bda40 con 0x559a87190c00 2024-04-26T02:08:21.177 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:08:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: starting mds.h at 2024-04-26T02:08:21.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T02:08:21.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:21.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:21 smithi191 ceph-mon[32745]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:21.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:21.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:21 smithi183 ceph-mon[32752]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: pgmap v742: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/2702270433' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T02:08:21.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:21 smithi052 ceph-mon[27851]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T02:08:22.100 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:22.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6836/1583930844,v1:172.21.15.183:6838/1583930844] up:boot 2024-04-26T02:08:22.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.183:6837/575603638,v1:172.21.15.183:6839/575603638] up:boot 2024-04-26T02:08:22.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6834/3734117654,v1:172.21.15.191:6835/3734117654] up:boot 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6836/2002095344,v1:172.21.15.191:6838/2002095344] up:boot 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.191:6837/2959605731,v1:172.21.15.191:6839/2959605731] up:boot 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: mds.? [v2:172.21.15.52:6840/18069620,v1:172.21.15.52:6841/18069620] up:boot 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:22.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:22.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:22.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:22.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:22 smithi052 ceph-mon[27851]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T02:08:23.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6836/1583930844,v1:172.21.15.183:6838/1583930844] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.183:6837/575603638,v1:172.21.15.183:6839/575603638] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6834/3734117654,v1:172.21.15.191:6835/3734117654] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6836/2002095344,v1:172.21.15.191:6838/2002095344] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.191:6837/2959605731,v1:172.21.15.191:6839/2959605731] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: mds.? [v2:172.21.15.52:6840/18069620,v1:172.21.15.52:6841/18069620] up:boot 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:23.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:23.155 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:23.155 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:23.155 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:23.155 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:23.155 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:22 smithi191 ceph-mon[32745]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6836/1583930844,v1:172.21.15.183:6838/1583930844] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.183:6837/575603638,v1:172.21.15.183:6839/575603638] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6834/3734117654,v1:172.21.15.191:6835/3734117654] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6836/2002095344,v1:172.21.15.191:6838/2002095344] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.191:6837/2959605731,v1:172.21.15.191:6839/2959605731] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: mds.? [v2:172.21.15.52:6840/18069620,v1:172.21.15.52:6841/18069620] up:boot 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-26T02:08:23.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T02:08:23.177 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T02:08:23.177 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T02:08:23.177 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T02:08:23.177 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T02:08:23.177 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:22 smithi183 ceph-mon[32752]: from='mgr.25039 172.21.15.52:0/1709280702' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T02:08:23.942 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:23 smithi052 ceph-mon[27851]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 10 op/s 2024-04-26T02:08:24.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:23 smithi191 ceph-mon[32745]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 10 op/s 2024-04-26T02:08:24.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:23 smithi183 ceph-mon[32752]: pgmap v744: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 10 op/s 2024-04-26T02:08:24.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 -- 172.21.15.52:0/4091140287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac100ec0 msgr2=0x7f2cac1012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 --2- 172.21.15.52:0/4091140287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac100ec0 0x7f2cac1012a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000b0a0 tx=0x7f2ca000b3b0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 -- 172.21.15.52:0/4091140287 shutdown_connections 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 --2- 172.21.15.52:0/4091140287 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 0x7f2cac10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 --2- 172.21.15.52:0/4091140287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac100ec0 0x7f2cac1012a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 --2- 172.21.15.52:0/4091140287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac101870 0x7f2cac1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 -- 172.21.15.52:0/4091140287 >> 172.21.15.52:0/4091140287 conn(0x7f2cac077b40 msgr2=0x7f2cac0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:24.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 -- 172.21.15.52:0/4091140287 shutdown_connections 2024-04-26T02:08:24.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.496+0000 7f2cb3600700 1 -- 172.21.15.52:0/4091140287 wait complete. 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.498+0000 7f2cb3600700 1 Processor -- start 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.498+0000 7f2cb3600700 1 -- start start 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac100ec0 0x7f2cac0fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 0x7f2cac0fda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac111b50 con 0x7f2cac105fa0 2024-04-26T02:08:24.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac1119d0 con 0x7f2cac101870 2024-04-26T02:08:24.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb3600700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac111cd0 con 0x7f2cac100ec0 2024-04-26T02:08:24.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:24.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb1600700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 0x7f2cac0fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:24.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:33800/0 (socket says 172.21.15.52:33800) 2024-04-26T02:08:24.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 -- 172.21.15.52:0/3091410459 learned_addr learned my addr 172.21.15.52:0/3091410459 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:24.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cb0c00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac100ec0 0x7f2cac0fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 -- 172.21.15.52:0/3091410459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac100ec0 msgr2=0x7f2cac0fcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac100ec0 0x7f2cac0fcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 -- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 msgr2=0x7f2cac0fda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 0x7f2cac0fda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.499+0000 7f2cabe00700 1 -- 172.21.15.52:0/3091410459 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac1a76d0 con 0x7f2cac101870 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2cabe00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800e970 tx=0x7f2c9800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98004a30 con 0x7f2cac101870 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cac1a79c0 con 0x7f2cac101870 2024-04-26T02:08:24.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c9800cc90 con 0x7f2cac101870 2024-04-26T02:08:24.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98005610 con 0x7f2cac101870 2024-04-26T02:08:24.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.500+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cac1a7f50 con 0x7f2cac101870 2024-04-26T02:08:24.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.502+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cac1023a0 con 0x7f2cac101870 2024-04-26T02:08:24.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.504+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2c9800c7e0 con 0x7f2cac101870 2024-04-26T02:08:24.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.505+0000 7f2ca9600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 0x7f2c940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:24.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.505+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 8735+0+0 (secure 0 0 0) 0x7f2c980d0cc0 con 0x7f2cac101870 2024-04-26T02:08:24.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.505+0000 7f2cb0c00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 0x7f2c940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:24.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.506+0000 7f2cb0c00700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 0x7f2c940a6270 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009fd0 tx=0x7f2ca0006210 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:24.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.512+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c98053ff0 con 0x7f2cac101870 2024-04-26T02:08:24.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:24.801+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2cac0fe490 con 0x7f2cac101870 2024-04-26T02:08:25.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.722+0000 7f2ca9600700 1 -- 172.21.15.52:0/3091410459 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v78) ==== 104+0+0 (secure 0 0 0) 0x7f2c9809d230 con 0x7f2cac101870 2024-04-26T02:08:25.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.725+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 msgr2=0x7f2c940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:25.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.725+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 0x7f2c940a6270 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009fd0 tx=0x7f2ca0006210 comp rx=0 tx=0).stop 2024-04-26T02:08:25.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.725+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 msgr2=0x7f2cac0fd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:25.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.725+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800e970 tx=0x7f2c9800ec80 comp rx=0 tx=0).stop 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 shutdown_connections 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7f2c940a3db0 0x7f2c940a6270 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2cac105fa0 0x7f2cac0fda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cac100ec0 0x7f2cac0fcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 --2- 172.21.15.52:0/3091410459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2cac101870 0x7f2cac0fd4f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:25.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 >> 172.21.15.52:0/3091410459 conn(0x7f2cac077b40 msgr2=0x7f2cac10ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:25.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 shutdown_connections 2024-04-26T02:08:25.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:25.726+0000 7f2cb3600700 1 -- 172.21.15.52:0/3091410459 wait complete. 2024-04-26T02:08:26.084 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:25 smithi052 ceph-mon[27851]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2024-04-26T02:08:26.084 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:25 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3091410459' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.084 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:25 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:25 smithi191 ceph-mon[32745]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2024-04-26T02:08:26.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:25 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3091410459' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:25 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:25 smithi183 ceph-mon[32752]: pgmap v745: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2024-04-26T02:08:26.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:25 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3091410459' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:25 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T02:08:26.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T02:08:26.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:26.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:26.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: Cluster is now healthy 2024-04-26T02:08:26.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:26.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: fsmap 12 up:standby 2024-04-26T02:08:26.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:26 smithi052 ceph-mon[27851]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2024-04-26T02:08:27.114 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:27.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:27.153 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:27.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: Cluster is now healthy 2024-04-26T02:08:27.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:27.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: fsmap 12 up:standby 2024-04-26T02:08:27.154 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:26 smithi191 ceph-mon[32745]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: Cluster is now healthy 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: fsmap 12 up:standby 2024-04-26T02:08:27.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:26 smithi183 ceph-mon[32752]: pgmap v746: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2024-04-26T02:08:29.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 -- 172.21.15.52:0/703686823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 msgr2=0x7fcab410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 --2- 172.21.15.52:0/703686823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab410f8b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00b0a0 tx=0x7fcaac00b3b0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 -- 172.21.15.52:0/703686823 shutdown_connections 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 --2- 172.21.15.52:0/703686823 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab410f8b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 --2- 172.21.15.52:0/703686823 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 --2- 172.21.15.52:0/703686823 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.560+0000 7fcab8a00700 1 -- 172.21.15.52:0/703686823 >> 172.21.15.52:0/703686823 conn(0x7fcab4079ba0 msgr2=0x7fcab4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:29.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.561+0000 7fcab8a00700 1 -- 172.21.15.52:0/703686823 shutdown_connections 2024-04-26T02:08:29.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:29 smithi052 ceph-mon[27851]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:29.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.561+0000 7fcab8a00700 1 -- 172.21.15.52:0/703686823 wait complete. 2024-04-26T02:08:29.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.562+0000 7fcab8a00700 1 Processor -- start 2024-04-26T02:08:29.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.562+0000 7fcab8a00700 1 -- start start 2024-04-26T02:08:29.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:29.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4112c50 con 0x7fcab4108a50 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4112ad0 con 0x7fcab4103450 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab8a00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4112dd0 con 0x7fcab4102a90 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcaabe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab2800700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcaabe00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38250/0 (socket says 172.21.15.52:38250) 2024-04-26T02:08:29.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcaabe00700 1 -- 172.21.15.52:0/1802356632 learned_addr learned my addr 172.21.15.52:0/1802356632 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcab1e00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcaabe00700 1 -- 172.21.15.52:0/1802356632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 msgr2=0x7fcab419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.563+0000 7fcaabe00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaabe00700 1 -- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 msgr2=0x7fcab419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaabe00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaabe00700 1 -- 172.21.15.52:0/1802356632 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab41a03d0 con 0x7fcab4103450 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcab2800700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:29.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcab1e00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaabe00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00e9c0 tx=0x7fca9c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00cbd0 con 0x7fcab4103450 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca9c00ee60 con 0x7fcab4103450 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab41a06c0 con 0x7fcab4103450 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.564+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c0055c0 con 0x7fcab4103450 2024-04-26T02:08:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.565+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab41a7f40 con 0x7fcab4103450 2024-04-26T02:08:29.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.566+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab4069730 con 0x7fcab4103450 2024-04-26T02:08:29.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.568+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fca9c004750 con 0x7fcab4103450 2024-04-26T02:08:29.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.569+0000 7fcaaaa00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 0x7fca980a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:29.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.569+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 8735+0+0 (secure 0 0 0) 0x7fca9c0d0b20 con 0x7fcab4103450 2024-04-26T02:08:29.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.569+0000 7fcab1e00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 0x7fca980a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:29.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.570+0000 7fcab1e00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 0x7fca980a62f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009fd0 tx=0x7fcaa4009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:29.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.574+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca9c09d090 con 0x7fcab4103450 2024-04-26T02:08:29.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:29.805+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcab41a8110 con 0x7fcab4103450 2024-04-26T02:08:29.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:29 smithi191 ceph-mon[32745]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:29.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:29 smithi183 ceph-mon[32752]: pgmap v747: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:30.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.501+0000 7fcaaaa00700 1 -- 172.21.15.52:0/1802356632 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v136) ==== 181+0+0 (secure 0 0 0) 0x7fca9c09c990 con 0x7fcab4103450 2024-04-26T02:08:30.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 msgr2=0x7fca980a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:30.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 0x7fca980a62f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009fd0 tx=0x7fcaa4009260 comp rx=0 tx=0).stop 2024-04-26T02:08:30.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 msgr2=0x7fcab419b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:30.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00e9c0 tx=0x7fca9c00ecd0 comp rx=0 tx=0).stop 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 shutdown_connections 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fca980a3e30 0x7fca980a62f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcab4108a50 0x7fcab419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcab4102a90 0x7fcab419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 --2- 172.21.15.52:0/1802356632 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcab4103450 0x7fcab419b900 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.507+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 >> 172.21.15.52:0/1802356632 conn(0x7fcab4079ba0 msgr2=0x7fcab410df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:30.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.508+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 shutdown_connections 2024-04-26T02:08:30.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:30.508+0000 7fcab8a00700 1 -- 172.21.15.52:0/1802356632 wait complete. 2024-04-26T02:08:30.512 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_metadata' removed 2024-04-26T02:08:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:30 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/1802356632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:30.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:30 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:30.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:30 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/1802356632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:30.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:30 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:30.946 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:30 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/1802356632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:30.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:30 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:31.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 8b150f36-036c-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T02:08:31.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:31 smithi052 ceph-mon[27851]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:31.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:31 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:31.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:31 smithi052 ceph-mon[27851]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:31 smithi191 ceph-mon[32745]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:31 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:31.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:31 smithi191 ceph-mon[32745]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:31.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:31 smithi183 ceph-mon[32752]: pgmap v748: 97 pgs: 97 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.5 KiB/s wr, 3 op/s 2024-04-26T02:08:31.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:31 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:31.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:31 smithi183 ceph-mon[32752]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T02:08:31.926 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T02:08:33.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:33 smithi052 ceph-mon[27851]: pgmap v750: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:33.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:33 smithi191 ceph-mon[32745]: pgmap v750: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:33.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:33 smithi183 ceph-mon[32752]: pgmap v750: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:34.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 -- 172.21.15.52:0/894541938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 msgr2=0x7fe3e8108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:34.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 --2- 172.21.15.52:0/894541938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e8108e50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009f90 tx=0x7fe3d800b330 comp rx=0 tx=0).stop 2024-04-26T02:08:34.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 -- 172.21.15.52:0/894541938 shutdown_connections 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 --2- 172.21.15.52:0/894541938 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 0x7fe3e810f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 --2- 172.21.15.52:0/894541938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e8108e50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 --2- 172.21.15.52:0/894541938 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e8102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 -- 172.21.15.52:0/894541938 >> 172.21.15.52:0/894541938 conn(0x7fe3e8079b10 msgr2=0x7fe3e8079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 -- 172.21.15.52:0/894541938 shutdown_connections 2024-04-26T02:08:34.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.243+0000 7fe3ede00700 1 -- 172.21.15.52:0/894541938 wait complete. 2024-04-26T02:08:34.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.245+0000 7fe3ede00700 1 Processor -- start 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.245+0000 7fe3ede00700 1 -- start start 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 0x7fe3e8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e819d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8112bf0 con 0x7fe3e8103460 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8112a70 con 0x7fe3e8102ac0 2024-04-26T02:08:34.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3ede00700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8112d70 con 0x7fe3e8108a70 2024-04-26T02:08:34.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3e6a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:34.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3e6a00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.52:38268/0 (socket says 172.21.15.52:38268) 2024-04-26T02:08:34.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3e6a00700 1 -- 172.21.15.52:0/3652837315 learned_addr learned my addr 172.21.15.52:0/3652837315 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-04-26T02:08:34.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3e6000700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 0x7fe3e8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:34.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.246+0000 7fe3e7400700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e819d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 -- 172.21.15.52:0/3652837315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 msgr2=0x7fe3e819d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e819d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 -- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 msgr2=0x7fe3e8199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 0x7fe3e8199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 -- 172.21.15.52:0/3652837315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e819e0d0 con 0x7fe3e8102ac0 2024-04-26T02:08:34.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3e6a00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009f60 tx=0x7fe3d800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.247+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8021070 con 0x7fe3e8102ac0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.248+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e819e360 con 0x7fe3e8102ac0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.248+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3d8007ac0 con 0x7fe3e8102ac0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.248+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8005040 con 0x7fe3e8102ac0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.248+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e81a5c40 con 0x7fe3e8102ac0 2024-04-26T02:08:34.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.249+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e8069730 con 0x7fe3e8102ac0 2024-04-26T02:08:34.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.251+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe3d8024070 con 0x7fe3e8102ac0 2024-04-26T02:08:34.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.252+0000 7fe3eca00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 0x7fe3cc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T02:08:34.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.252+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 8210+0+0 (secure 0 0 0) 0x7fe3d800f020 con 0x7fe3e8102ac0 2024-04-26T02:08:34.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.252+0000 7fe3e6000700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 0x7fe3cc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T02:08:34.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.253+0000 7fe3e6000700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 0x7fe3cc0a6270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe3e807a180 tx=0x7fe3d4009260 comp rx=0 tx=0).ready entity=mgr.25039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T02:08:34.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.256+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3d802b020 con 0x7fe3e8102ac0 2024-04-26T02:08:34.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.490+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe3e802df80 con 0x7fe3e8102ac0 2024-04-26T02:08:34.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.514+0000 7fe3eca00700 1 -- 172.21.15.52:0/3652837315 <== mon.2 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v137) ==== 169+0+0 (secure 0 0 0) 0x7fe3d809e2d0 con 0x7fe3e8102ac0 2024-04-26T02:08:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 msgr2=0x7fe3cc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 0x7fe3cc0a6270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe3e807a180 tx=0x7fe3d4009260 comp rx=0 tx=0).stop 2024-04-26T02:08:34.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 msgr2=0x7fe3e81990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T02:08:34.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009f60 tx=0x7fe3d800ccb0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 shutdown_connections 2024-04-26T02:08:34.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:6800/1185318772,v1:172.21.15.52:6801/1185318772] conn(0x7fe3cc0a3db0 0x7fe3cc0a6270 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3e8108a70 0x7fe3e819d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3e8102ac0 0x7fe3e81990c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 --2- 172.21.15.52:0/3652837315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3e8103460 0x7fe3e8199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T02:08:34.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.522+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 >> 172.21.15.52:0/3652837315 conn(0x7fe3e8079b10 msgr2=0x7fe3e8100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T02:08:34.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.523+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 shutdown_connections 2024-04-26T02:08:34.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-04-26T02:08:34.523+0000 7fe3ede00700 1 -- 172.21.15.52:0/3652837315 wait complete. 2024-04-26T02:08:34.526 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data' removed 2024-04-26T02:08:34.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:34 smithi191 ceph-mon[32745]: from='client.? 172.21.15.52:0/3652837315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:34.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:34 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:34.925 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:34 smithi183 ceph-mon[32752]: from='client.? 172.21.15.52:0/3652837315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:34.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:34 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:34.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:34 smithi052 ceph-mon[27851]: from='client.? 172.21.15.52:0/3652837315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:34.932 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:34 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T02:08:35.463 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:35.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:35.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:08:35.623 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T02:08:35.623 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:35.682 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:35.708 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T02:08:35.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:35 smithi052 ceph-mon[27851]: pgmap v751: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:35.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:35 smithi052 ceph-mon[27851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:35.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:35 smithi052 ceph-mon[27851]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:35.722 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T02:08:35.723 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T02:08:35.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a 2024-04-26T02:08:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:35 smithi191 ceph-mon[32745]: pgmap v751: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:35.903 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:35 smithi191 ceph-mon[32745]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:35.904 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:35 smithi191 ceph-mon[32745]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:35.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:35 smithi183 ceph-mon[32752]: pgmap v751: 65 pgs: 65 active+clean; 332 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T02:08:35.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:35 smithi183 ceph-mon[32752]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T02:08:35.926 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:35 smithi183 ceph-mon[32752]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T02:08:36.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:35 smithi052 systemd[1]: Stopping Ceph mon.a for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:36.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a[27847]: 2024-04-26T02:08:36.319+0000 7fe8b3a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:36.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a[27847]: 2024-04-26T02:08:36.319+0000 7fe8b3a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T02:08:36.943 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:36 smithi052 podman[109104]: 2024-04-26 02:08:36.641624861 +0000 UTC m=+0.479666920 container died d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:08:37.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:37 smithi052 podman[109104]: 2024-04-26 02:08:37.086658244 +0000 UTC m=+0.924700305 container cleanup d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-26T02:08:37.213 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:37 smithi052 bash[109104]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a 2024-04-26T02:08:37.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Apr 26 02:08:37 smithi052 podman[109118]: 2024-04-26 02:08:37.346173489 +0000 UTC m=+0.686929779 container remove d282b1058cde95763a5f8e3f1fcaf8e3ee1ec7a5ed8d24af4b779a96e9fb22a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T02:08:37.829 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T02:08:37.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:37.959 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T02:08:37.959 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T02:08:37.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.b 2024-04-26T02:08:38.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 systemd[1]: Stopping Ceph mon.b for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:38.556 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-b[32748]: 2024-04-26T02:08:38.291+0000 7ff9d9151700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:38.556 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-b[32748]: 2024-04-26T02:08:38.291+0000 7ff9d9151700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T02:08:38.833 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 podman[72739]: 2024-04-26 02:08:38.557592053 +0000 UTC m=+0.387644326 container died 00c660ae7e9e93449592e416ea13f51be22d52017c906243edb4787599d3c611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:08:39.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 podman[72739]: 2024-04-26 02:08:38.933636683 +0000 UTC m=+0.763688956 container cleanup 00c660ae7e9e93449592e416ea13f51be22d52017c906243edb4787599d3c611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T02:08:39.176 INFO:journalctl@ceph.mon.b.smithi183.stdout:Apr 26 02:08:38 smithi183 bash[72739]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-b 2024-04-26T02:08:39.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T02:08:39.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:39.405 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T02:08:39.405 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T02:08:39.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.c 2024-04-26T02:08:39.748 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:39 smithi191 systemd[1]: Stopping Ceph mon.c for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:40.087 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-c[32728]: 2024-04-26T02:08:39.748+0000 7f27e9c2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:40.088 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-c[32728]: 2024-04-26T02:08:39.748+0000 7f27e9c2e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T02:08:40.088 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:39 smithi191 podman[73217]: 2024-04-26 02:08:39.800483846 +0000 UTC m=+0.175279405 container died 7dfd944d208e15c2acd8e0307256629799a7676f9ac315b81f9e553fe2ae6b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T02:08:40.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:40 smithi191 podman[73217]: 2024-04-26 02:08:40.190703027 +0000 UTC m=+0.565498573 container cleanup 7dfd944d208e15c2acd8e0307256629799a7676f9ac315b81f9e553fe2ae6b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T02:08:40.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Apr 26 02:08:40 smithi191 bash[73217]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mon-c 2024-04-26T02:08:40.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T02:08:40.653 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:40.654 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T02:08:40.654 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T02:08:40.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x 2024-04-26T02:08:41.213 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:40 smithi052 systemd[1]: Stopping Ceph mgr.x for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:41.651 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T02:08:41.222+0000 7f177d000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:41.652 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x[28090]: 2024-04-26T02:08:41.222+0000 7f177d000700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T02:08:41.652 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:41 smithi052 podman[109222]: 2024-04-26 02:08:41.319220574 +0000 UTC m=+0.253155872 container died cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:08:42.213 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:41 smithi052 podman[109222]: 2024-04-26 02:08:41.949893942 +0000 UTC m=+0.883829264 container cleanup cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-26T02:08:42.243 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:41 smithi052 bash[109222]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x 2024-04-26T02:08:42.243 INFO:journalctl@ceph.mgr.x.smithi052.stdout:Apr 26 02:08:42 smithi052 podman[109236]: 2024-04-26 02:08:42.176459324 +0000 UTC m=+0.841635108 container remove cce5110b139c7f879e658b45c527ab8138a01164810155bed8e75226c891ec61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T02:08:42.705 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T02:08:42.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:42.833 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T02:08:42.833 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T02:08:42.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.y 2024-04-26T02:08:43.157 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 02:08:42 smithi183 systemd[1]: Stopping Ceph mgr.y for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:43.426 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 02:08:43 smithi183 podman[72861]: 2024-04-26 02:08:43.158432928 +0000 UTC m=+0.136421685 container died 25eeb8bc80e78abf2431ed0c92c90d681927b80a15e60a0336af069b53a27c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T02:08:43.811 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 02:08:43 smithi183 podman[72861]: 2024-04-26 02:08:43.53727164 +0000 UTC m=+0.515260388 container cleanup 25eeb8bc80e78abf2431ed0c92c90d681927b80a15e60a0336af069b53a27c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-26T02:08:43.812 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Apr 26 02:08:43 smithi183 bash[72861]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-y 2024-04-26T02:08:43.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T02:08:44.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:44.009 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T02:08:44.009 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T02:08:44.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.z 2024-04-26T02:08:44.347 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 02:08:44 smithi191 systemd[1]: Stopping Ceph mgr.z for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:44.644 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 02:08:44 smithi191 podman[73337]: 2024-04-26 02:08:44.348868263 +0000 UTC m=+0.147481912 container died 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5) 2024-04-26T02:08:44.903 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 02:08:44 smithi191 podman[73337]: 2024-04-26 02:08:44.703197099 +0000 UTC m=+0.501810790 container cleanup 071cf2d73ef6865e5c5e973b1b6af9c03d9304eb9b6142dc21dc15f5d5793fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T02:08:44.904 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Apr 26 02:08:44 smithi191 bash[73337]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mgr-z 2024-04-26T02:08:45.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T02:08:45.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:45.205 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T02:08:45.206 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T02:08:45.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.0 2024-04-26T02:08:45.533 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:45 smithi052 systemd[1]: Stopping Ceph osd.0 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:45.962 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0[42537]: 2024-04-26T02:08:45.719+0000 7fd62a400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:45.963 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0[42537]: 2024-04-26T02:08:45.719+0000 7fd62a400700 -1 osd.0 137 *** Got signal Terminated *** 2024-04-26T02:08:45.963 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0[42537]: 2024-04-26T02:08:45.719+0000 7fd62a400700 -1 osd.0 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:08:51.607 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:51 smithi052 podman[109353]: 2024-04-26 02:08:51.356351032 +0000 UTC m=+5.770638614 container died 3020e08b797eae524ec2f326a2a0f3cebc7b10263a9a523c0dda7db2e5d9016e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T02:08:52.205 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:51 smithi052 podman[109353]: 2024-04-26 02:08:51.920861486 +0000 UTC m=+6.335149061 container cleanup 3020e08b797eae524ec2f326a2a0f3cebc7b10263a9a523c0dda7db2e5d9016e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:52.206 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:51 smithi052 bash[109353]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0 2024-04-26T02:08:52.463 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:52 smithi052 podman[109371]: 2024-04-26 02:08:52.205618556 +0000 UTC m=+0.836449034 container remove 3020e08b797eae524ec2f326a2a0f3cebc7b10263a9a523c0dda7db2e5d9016e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:53.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:52.966453608 +0000 UTC m=+0.037536290 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:08:53.385 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:53.073184706 +0000 UTC m=+0.144267431 container create 09d880f59eee5758a16b8ce82938fce8505f0f14a6ffa836461416f5411e1285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T02:08:53.385 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:53.385065809 +0000 UTC m=+0.456148531 container init 09d880f59eee5758a16b8ce82938fce8505f0f14a6ffa836461416f5411e1285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:08:53.713 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:53.397076844 +0000 UTC m=+0.468159522 container start 09d880f59eee5758a16b8ce82938fce8505f0f14a6ffa836461416f5411e1285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:08:53.714 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:53.49211256 +0000 UTC m=+0.563195288 container attach 09d880f59eee5758a16b8ce82938fce8505f0f14a6ffa836461416f5411e1285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:08:54.213 INFO:journalctl@ceph.osd.0.smithi052.stdout:Apr 26 02:08:53 smithi052 podman[109459]: 2024-04-26 02:08:53.946564987 +0000 UTC m=+1.017647690 container died 09d880f59eee5758a16b8ce82938fce8505f0f14a6ffa836461416f5411e1285 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-26T02:08:54.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T02:08:54.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:08:54.850 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T02:08:54.850 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T02:08:54.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.1 2024-04-26T02:08:54.926 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:08:54 smithi183 systemd[1]: Stopping Ceph osd.1 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:08:55.426 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:08:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1[37336]: 2024-04-26T02:08:55.140+0000 7f5e3363a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:08:55.429 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:08:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1[37336]: 2024-04-26T02:08:55.140+0000 7f5e3363a700 -1 osd.1 137 *** Got signal Terminated *** 2024-04-26T02:08:55.429 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:08:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1[37336]: 2024-04-26T02:08:55.140+0000 7f5e3363a700 -1 osd.1 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:00.783 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:00 smithi183 podman[72996]: 2024-04-26 02:09:00.521861716 +0000 UTC m=+5.477501955 container died 49ea6141af819f4bd53958a0ca860edd5c104e8bda31cda2dfcd0a9ca73ac8a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:01.176 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:00 smithi183 podman[72996]: 2024-04-26 02:09:00.88238237 +0000 UTC m=+5.838022595 container cleanup 49ea6141af819f4bd53958a0ca860edd5c104e8bda31cda2dfcd0a9ca73ac8a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T02:09:01.176 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:00 smithi183 bash[72996]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1 2024-04-26T02:09:01.547 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73012]: 2024-04-26 02:09:01.200138595 +0000 UTC m=+0.673229155 container remove 49ea6141af819f4bd53958a0ca860edd5c104e8bda31cda2dfcd0a9ca73ac8a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T02:09:01.869 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73101]: 2024-04-26 02:09:01.452195982 +0000 UTC m=+0.018031912 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:01.869 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73101]: 2024-04-26 02:09:01.550890383 +0000 UTC m=+0.116726335 container create 5e44855950f18682362ac75c738fb02b2c0783c1fbc3569b1252925da5f2039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T02:09:01.869 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73101]: 2024-04-26 02:09:01.786777441 +0000 UTC m=+0.352613359 container init 5e44855950f18682362ac75c738fb02b2c0783c1fbc3569b1252925da5f2039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T02:09:01.869 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73101]: 2024-04-26 02:09:01.789904522 +0000 UTC m=+0.355740432 container start 5e44855950f18682362ac75c738fb02b2c0783c1fbc3569b1252925da5f2039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:01.869 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:01 smithi183 podman[73101]: 2024-04-26 02:09:01.870040877 +0000 UTC m=+0.435876805 container attach 5e44855950f18682362ac75c738fb02b2c0783c1fbc3569b1252925da5f2039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-26T02:09:02.391 INFO:journalctl@ceph.osd.1.smithi183.stdout:Apr 26 02:09:02 smithi183 podman[73101]: 2024-04-26 02:09:02.122728034 +0000 UTC m=+0.688563944 container died 5e44855950f18682362ac75c738fb02b2c0783c1fbc3569b1252925da5f2039a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-26T02:09:02.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T02:09:02.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:02.825 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T02:09:02.825 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T02:09:02.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.2 2024-04-26T02:09:03.153 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:02 smithi191 systemd[1]: Stopping Ceph osd.2 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:03.412 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2[37876]: 2024-04-26T02:09:03.213+0000 7f11fff05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:03.412 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2[37876]: 2024-04-26T02:09:03.213+0000 7f11fff05700 -1 osd.2 137 *** Got signal Terminated *** 2024-04-26T02:09:03.412 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2[37876]: 2024-04-26T02:09:03.213+0000 7f11fff05700 -1 osd.2 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:08.903 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:08 smithi191 podman[73464]: 2024-04-26 02:09:08.644184929 +0000 UTC m=+5.631308685 container died 5aa3835bcce038451b2c4f105c9972c568f51b41819499c1bd5138c4baf2561c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T02:09:09.346 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73464]: 2024-04-26 02:09:09.02193148 +0000 UTC m=+6.009055234 container cleanup 5aa3835bcce038451b2c4f105c9972c568f51b41819499c1bd5138c4baf2561c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-26T02:09:09.347 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 bash[73464]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2 2024-04-26T02:09:09.653 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73493]: 2024-04-26 02:09:09.348315901 +0000 UTC m=+0.701906263 container remove 5aa3835bcce038451b2c4f105c9972c568f51b41819499c1bd5138c4baf2561c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-26T02:09:09.967 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73584]: 2024-04-26 02:09:09.595139249 +0000 UTC m=+0.018122368 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:09.968 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73584]: 2024-04-26 02:09:09.715795771 +0000 UTC m=+0.138778873 container create acc245077f6343a47df11a051132039c641ecf08aca6382732ef407d0b137e1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T02:09:09.968 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73584]: 2024-04-26 02:09:09.968865058 +0000 UTC m=+0.391848171 container init acc245077f6343a47df11a051132039c641ecf08aca6382732ef407d0b137e1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-26T02:09:10.304 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:09 smithi191 podman[73584]: 2024-04-26 02:09:09.972306691 +0000 UTC m=+0.395289802 container start acc245077f6343a47df11a051132039c641ecf08aca6382732ef407d0b137e1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-26T02:09:10.305 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:10 smithi191 podman[73584]: 2024-04-26 02:09:10.085388039 +0000 UTC m=+0.508371145 container attach acc245077f6343a47df11a051132039c641ecf08aca6382732ef407d0b137e1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-26T02:09:10.653 INFO:journalctl@ceph.osd.2.smithi191.stdout:Apr 26 02:09:10 smithi191 podman[73584]: 2024-04-26 02:09:10.305779541 +0000 UTC m=+0.728762652 container died acc245077f6343a47df11a051132039c641ecf08aca6382732ef407d0b137e1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T02:09:10.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T02:09:10.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:10.990 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T02:09:10.990 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T02:09:10.990 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.3 2024-04-26T02:09:11.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:11 smithi052 systemd[1]: Stopping Ceph osd.3 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:11.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:11 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:11.571+0000 7f9741600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:11.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:11 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:11.571+0000 7f9741600700 -1 osd.3 137 *** Got signal Terminated *** 2024-04-26T02:09:11.964 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:11 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:11.571+0000 7f9741600700 -1 osd.3 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:12.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:12 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:12.485+0000 7f9737e00700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.660413+0000 front 2024-04-26T02:08:50.660515+0000 (oldest deadline 2024-04-26T02:09:12.360863+0000) 2024-04-26T02:09:13.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:12.892+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:13.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:12 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:12.987+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:13.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:13 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:13.488+0000 7f9737e00700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.660413+0000 front 2024-04-26T02:08:50.660515+0000 (oldest deadline 2024-04-26T02:09:12.360863+0000) 2024-04-26T02:09:14.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:13.887+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:14.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:13 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:13.980+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:14.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:14.535+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:14.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:14 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:14.484+0000 7f9737e00700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.660413+0000 front 2024-04-26T02:08:50.660515+0000 (oldest deadline 2024-04-26T02:09:12.360863+0000) 2024-04-26T02:09:15.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:14.935+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:15.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:15 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:15.015+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:15.653 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:15.550+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:15.394+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:15.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:15.486+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:15.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:15 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:15.529+0000 7f9737e00700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.660413+0000 front 2024-04-26T02:08:50.660515+0000 (oldest deadline 2024-04-26T02:09:12.360863+0000) 2024-04-26T02:09:16.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:15.974+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:16.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:15 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:15.979+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:16.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:15 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:15.995+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:16.903 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:16.588+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:16.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:16.438+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:16.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:16.518+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:16.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3[49028]: 2024-04-26T02:09:16.526+0000 7f9737e00700 -1 osd.3 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.660413+0000 front 2024-04-26T02:08:50.660515+0000 (oldest deadline 2024-04-26T02:09:12.360863+0000) 2024-04-26T02:09:17.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:16.941+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:17.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:16 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:16.971+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:17.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:17 smithi052 podman[109565]: 2024-04-26 02:09:17.210584342 +0000 UTC m=+5.756132564 container died 24a9718409f064a282f04e48e866b2f3b6c2ece034d9fd9fde3d2cc6c11c2ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T02:09:17.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:17 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:17.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:17.903 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:17.413+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:17.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:17.634+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:17.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:17 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:17.469+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:17.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:17 smithi052 podman[109565]: 2024-04-26 02:09:17.697912939 +0000 UTC m=+6.243461154 container cleanup 24a9718409f064a282f04e48e866b2f3b6c2ece034d9fd9fde3d2cc6c11c2ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:17.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:17 smithi052 bash[109565]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3 2024-04-26T02:09:18.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:17 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:17.987+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:18.462 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:18 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:18.016+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:18.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:17 smithi052 podman[109585]: 2024-04-26 02:09:17.982332403 +0000 UTC m=+0.745458274 container remove 24a9718409f064a282f04e48e866b2f3b6c2ece034d9fd9fde3d2cc6c11c2ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:18.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:18 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:18.019+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:18.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:18.656+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:18.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:18.451+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:18.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:18 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:18.450+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:18.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:18 smithi052 podman[109670]: 2024-04-26 02:09:18.525866301 +0000 UTC m=+0.038809717 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:18.963 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:18 smithi052 podman[109670]: 2024-04-26 02:09:18.624592947 +0000 UTC m=+0.137536383 container create 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:19.411 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:19 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:19.022+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:19.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:19 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:19.047+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:19.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:19 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:19.055+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:19.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:19 smithi052 podman[109670]: 2024-04-26 02:09:19.045234884 +0000 UTC m=+0.558178360 container init 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T02:09:19.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:19 smithi052 podman[109670]: 2024-04-26 02:09:19.058212814 +0000 UTC m=+0.571156274 container start 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T02:09:19.463 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:19 smithi052 podman[109670]: 2024-04-26 02:09:19.135724904 +0000 UTC m=+0.648668414 container attach 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:19.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:19 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:19.412+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:19.903 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:19.498+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:19.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:19.623+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:19.913 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:19 smithi052 podman[109670]: 2024-04-26 02:09:19.531300956 +0000 UTC m=+1.044244373 container died 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T02:09:20.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:20.062+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:20.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:20.005+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:20.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:20.032+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:20.503 INFO:journalctl@ceph.osd.3.smithi052.stdout:Apr 26 02:09:20 smithi052 podman[109695]: 2024-04-26 02:09:20.472463552 +0000 UTC m=+0.918885628 container remove 2eeee77ea333b60cb72f3d6213ac74b94685d7429091f2cadae665bdd0345254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-26T02:09:20.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T02:09:20.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:20.618 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T02:09:20.618 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T02:09:20.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.4 2024-04-26T02:09:20.646 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:20.483+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:20.903 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:20.647+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:20.925 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:20 smithi183 systemd[1]: Stopping Ceph osd.4 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:20.925 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:20.439+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:21.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:20.926+0000 7fcce482b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:21.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:20.926+0000 7fcce482b700 -1 osd.4 137 *** Got signal Terminated *** 2024-04-26T02:09:21.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:20 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:20.926+0000 7fcce482b700 -1 osd.4 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:21.176 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:21 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:21.076+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:21.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:20 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:20.994+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:21.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:21 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:21.038+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:21.903 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:21.691+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:21.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:21.749+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:21.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:21.501+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:21.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:21.501+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:21.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:21 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:21.480+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:22.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:22 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:22.071+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:22.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:22 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:22.034+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:22.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:22 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:22.028+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:22.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:22 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:22.028+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:22.903 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:22.645+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:22.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:22.751+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:22.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:22.518+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:22.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:22.518+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:22.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:22 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:22.466+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:22.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:22 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:22.699+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:23.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:23 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:23.091+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:23.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:23 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:23.026+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:23.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:23 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:23.053+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:23.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:23 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:23.053+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:23.799 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:23.665+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:23.799 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:23.532+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:23.799 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:23.532+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:23.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:23 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:23.670+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:23.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:23 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:23.485+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:24.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:23.800+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:24.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:24 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:24.067+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:24.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:24 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:24.028+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:24.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:24 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:24.083+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:24.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:24 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:24.083+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:24.770 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:24.519+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:24.770 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:24.519+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:24.771 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:24.626+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:24.771 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:24.626+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:24.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:24 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:24.623+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:24.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:24 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:24.528+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:25.153 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:24.771+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:25.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:25 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4[43859]: 2024-04-26T02:09:25.038+0000 7fccdce17700 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.678483+0000 front 2024-04-26T02:08:49.678325+0000 (oldest deadline 2024-04-26T02:09:11.978196+0000) 2024-04-26T02:09:25.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:25 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:25.043+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:25.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:25 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:25.043+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:25.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:25 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:25.035+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:25.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:25 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:25.035+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:25.784 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:25 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:25.475+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:25.784 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:25 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:25.475+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:25.784 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:25 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:25.671+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:25.785 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:25 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:25.671+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:25.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:25 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:25.500+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:25.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:25 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:25.500+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:25.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:25 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:25.590+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:26.153 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:25 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:25.785+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:26.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:26.083+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:26.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:26.083+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:26.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:26.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:26.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:26.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:26.573 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:26 smithi183 podman[73222]: 2024-04-26 02:09:26.301405424 +0000 UTC m=+5.485406775 container died 7f5c8f6fe7ffdc03efb6b9896172544984e8f8e8a1896af5a0eef42234f8225d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:26.573 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:26 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:26.464+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:26.574 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:26 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:26.464+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:26.761 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:26 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:26.479+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:26.761 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:26 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:26.479+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:26.761 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:26 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:26.688+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:26.761 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:26 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:26.688+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:26.843 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:26 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:26.574+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:26.843 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:26 smithi183 podman[73222]: 2024-04-26 02:09:26.610048232 +0000 UTC m=+5.794049615 container cleanup 7f5c8f6fe7ffdc03efb6b9896172544984e8f8e8a1896af5a0eef42234f8225d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T02:09:26.843 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:26 smithi183 bash[73222]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4 2024-04-26T02:09:27.153 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:26 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:26.761+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:27.172 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:26 smithi183 podman[73239]: 2024-04-26 02:09:26.844502977 +0000 UTC m=+0.538247996 container remove 7f5c8f6fe7ffdc03efb6b9896172544984e8f8e8a1896af5a0eef42234f8225d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-26T02:09:27.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.07671961 +0000 UTC m=+0.017467987 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:27.426 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.20354219 +0000 UTC m=+0.144290568 container create 8f685229c17e5eac70cfd7be833f14862e49b64ebafb837cf750d3ea2ecabffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:27.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:27 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:27.066+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:27.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:27 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:27.066+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:27.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:26.982+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:27.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:26 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:26.982+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:27.759 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:27 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:27.544+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:27.759 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:27 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:27.426+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:27.759 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:27 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:27.426+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:27.759 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.456738243 +0000 UTC m=+0.397486622 container init 8f685229c17e5eac70cfd7be833f14862e49b64ebafb837cf750d3ea2ecabffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:27.760 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.459912285 +0000 UTC m=+0.400660664 container start 8f685229c17e5eac70cfd7be833f14862e49b64ebafb837cf750d3ea2ecabffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T02:09:27.760 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.539242832 +0000 UTC m=+0.479991240 container attach 8f685229c17e5eac70cfd7be833f14862e49b64ebafb837cf750d3ea2ecabffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T02:09:27.903 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:27.685+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:27.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:27.685+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:27.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:27.728+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:27.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:27.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:27.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:27 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:27.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:28.049 INFO:journalctl@ceph.osd.4.smithi183.stdout:Apr 26 02:09:27 smithi183 podman[73331]: 2024-04-26 02:09:27.760237362 +0000 UTC m=+0.700985735 container died 8f685229c17e5eac70cfd7be833f14862e49b64ebafb837cf750d3ea2ecabffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-26T02:09:28.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T02:09:28.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:28.377 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T02:09:28.377 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T02:09:28.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.5 2024-04-26T02:09:28.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:28.075+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:28.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:28.075+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:28.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:28.000+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:28.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:28.000+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:28.579 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:28.382+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:28.579 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:28.382+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:28.692 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:28.531+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:28.692 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:28.531+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:28.693 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 systemd[1]: Stopping Ceph osd.5 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:28.693 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:28.649+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:28.693 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:28.649+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:28.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:28 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:28.579+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:28.948 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:28.693+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:29.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:29 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:29.096+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:29.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:29 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:29.096+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:29.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:28.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:29.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:28.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:29.403 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:28.950+0000 7f38f8614700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:29.404 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:28.950+0000 7f38f8614700 -1 osd.5 137 *** Got signal Terminated *** 2024-04-26T02:09:29.404 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:28 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:28.950+0000 7f38f8614700 -1 osd.5 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:29.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:29.576+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:29.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:29.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:29.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:29 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:29.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:29.903 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:29.710+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:29.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:29.615+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:29.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:29.615+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:29.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:29.615+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.746910+0000 front 2024-04-26T02:09:07.746883+0000 (oldest deadline 2024-04-26T02:09:29.447082+0000) 2024-04-26T02:09:29.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:29.570+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:29.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:29 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:29.570+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:30.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:30 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:30.110+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:30.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:30 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:30.110+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:30.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:29 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:29.940+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:30.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:29 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:29.940+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:30.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:30 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:30.623+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:30.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:30 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:30.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:30.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:30 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:30.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:30.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:30 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:30.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:30.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:30.708+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:30.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:30.708+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:30.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:30.534+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:30.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:30.534+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:30.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:30.590+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:30.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:30.590+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:30.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:30 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:30.590+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.746910+0000 front 2024-04-26T02:09:07.746883+0000 (oldest deadline 2024-04-26T02:09:29.447082+0000) 2024-04-26T02:09:31.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:31 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:31.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:31.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:31 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:31.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:31.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:30 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:30.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:31.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:30 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:30.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:31.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:31 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:31.639+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:31.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:31 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:31.398+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:31.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:31 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:31.398+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:31.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:31 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:31.398+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:31.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:31.694+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:31.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:31.694+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:31.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:31.490+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:31.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:31.490+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:31.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:31.634+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:31.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:31.634+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:31.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:31 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:31.634+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.746910+0000 front 2024-04-26T02:09:07.746883+0000 (oldest deadline 2024-04-26T02:09:29.447082+0000) 2024-04-26T02:09:32.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:31 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:31.907+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:32.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:31 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:31.907+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:32.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:32.137+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:32.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:32.137+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:32.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:32.137+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:32.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:32.744+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:32.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:32.744+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:32.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:32.653+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:32.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:32.653+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:32.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:32.653+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.746910+0000 front 2024-04-26T02:09:07.746883+0000 (oldest deadline 2024-04-26T02:09:29.447082+0000) 2024-04-26T02:09:32.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:32.508+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:32.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:32.508+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:32.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:32 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:32.509+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:32.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:32 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:32.646+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:32.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:32 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:32.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:32.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:32 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:32.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:32.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:32 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:32.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:33.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:33.098+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:33.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:33.098+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:33.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:33.098+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:33.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:32.947+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:33.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:32.947+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:33.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:32 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:32.947+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:33.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:33 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:33.410+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:33.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:33 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:33.410+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:33.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:33 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:33.410+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:33.805 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:33.755+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:33.805 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:33.755+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:33.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:33.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:33.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:33.673+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.043907+0000 front 2024-04-26T02:08:49.043966+0000 (oldest deadline 2024-04-26T02:09:14.943867+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:33.673+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.645141+0000 front 2024-04-26T02:08:59.645196+0000 (oldest deadline 2024-04-26T02:09:23.745137+0000) 2024-04-26T02:09:33.806 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:33 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5[44861]: 2024-04-26T02:09:33.673+0000 7f38f0c00700 -1 osd.5 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.746910+0000 front 2024-04-26T02:09:07.746883+0000 (oldest deadline 2024-04-26T02:09:29.447082+0000) 2024-04-26T02:09:33.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:33 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:33.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:33.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:33 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:33.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:34.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:34.136+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:34.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:34.136+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:34.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:34.136+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:34.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:33.917+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:34.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:33.917+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:34.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:33 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:33.917+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:34.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:34.383+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:34.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:34.383+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:34.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:34.383+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:34.797 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:34 smithi191 podman[73690]: 2024-04-26 02:09:34.502871186 +0000 UTC m=+5.933566145 container died 76dab11e4986f4782822fd36eb61f580685780ae75bc615d4ea610d26b07c7d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:34.798 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:34 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:34.724+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:34.798 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:34 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:34.724+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:34.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:34 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:34.564+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:34.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:34 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:34.564+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:34.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:34 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:34.564+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:34.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:34.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:34.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:34 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:34.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:35.149 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:34 smithi191 podman[73690]: 2024-04-26 02:09:34.89073331 +0000 UTC m=+6.321428280 container cleanup 76dab11e4986f4782822fd36eb61f580685780ae75bc615d4ea610d26b07c7d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T02:09:35.149 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:34 smithi191 bash[73690]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5 2024-04-26T02:09:35.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:35.126+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:35.213 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:35.126+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:35.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:35.126+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:35.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:34.934+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:35.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:34.934+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:35.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:34.934+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:35.403 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73707]: 2024-04-26 02:09:35.150402489 +0000 UTC m=+0.646085132 container remove 76dab11e4986f4782822fd36eb61f580685780ae75bc615d4ea610d26b07c7d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-26T02:09:35.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:35 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:35.645+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:35.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:35 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:35.645+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:35.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:35 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:35.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:35.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:35 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:35.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:35.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:35 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:35.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:35.734 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:35 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:35.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:35.735 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:35 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:35.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:35.735 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:35 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:35.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:35.735 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73807]: 2024-04-26 02:09:35.387481339 +0000 UTC m=+0.018192905 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:35.735 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73807]: 2024-04-26 02:09:35.518275325 +0000 UTC m=+0.148986857 container create 96029bbe018b448b2dff32051f58d9f177ac25c8eaf3be06ed1929728ae7a5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T02:09:36.098 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73807]: 2024-04-26 02:09:35.804700781 +0000 UTC m=+0.435412322 container init 96029bbe018b448b2dff32051f58d9f177ac25c8eaf3be06ed1929728ae7a5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T02:09:36.098 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73807]: 2024-04-26 02:09:35.807840501 +0000 UTC m=+0.438552033 container start 96029bbe018b448b2dff32051f58d9f177ac25c8eaf3be06ed1929728ae7a5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-26T02:09:36.098 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:35 smithi191 podman[73807]: 2024-04-26 02:09:35.887873083 +0000 UTC m=+0.518584635 container attach 96029bbe018b448b2dff32051f58d9f177ac25c8eaf3be06ed1929728ae7a5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-26T02:09:36.099 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:35 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:35.734+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:36.099 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:35 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:35.734+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:36.390 INFO:journalctl@ceph.osd.5.smithi191.stdout:Apr 26 02:09:36 smithi191 podman[73807]: 2024-04-26 02:09:36.099462002 +0000 UTC m=+0.730173540 container died 96029bbe018b448b2dff32051f58d9f177ac25c8eaf3be06ed1929728ae7a5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-26T02:09:36.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:36.167+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:36.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:36.167+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:36.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:36.167+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:36.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:35.975+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:36.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:35.975+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:36.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:35 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:35.975+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:36.653 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:36 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:36.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:36.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:36 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:36.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:36.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:36 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:36.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:36.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:36 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:36.642+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:36.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:36 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:36.642+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:36.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:36 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:36.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:36.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:36 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:36.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:36.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:36 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:36.422+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:36.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T02:09:36.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:36.739 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T02:09:36.739 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T02:09:36.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.6 2024-04-26T02:09:37.147 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:36 smithi052 systemd[1]: Stopping Ceph osd.6 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:37.148 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:36.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:37.148 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:36.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:37.148 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:36 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:36.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:37.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:36 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:36.708+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:37.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:36 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:36.708+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:37.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.147+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:37.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.147+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:37.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.147+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:37.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.314+0000 7fe816400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:37.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.314+0000 7fe816400700 -1 osd.6 137 *** Got signal Terminated *** 2024-04-26T02:09:37.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:37 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:37.314+0000 7fe816400700 -1 osd.6 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:37.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:37 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:37.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:37.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:37 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:37.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:37.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:37 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:37.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:37.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:37 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:37.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:37.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:37 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:37.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:37.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:37 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:37.690+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:37.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:37 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:37.690+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:37.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:37 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:37.469+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:37.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:37 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:37.469+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:37.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:37 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:37.469+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:38.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:38.168+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:38.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:38.168+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:38.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:38.168+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:38.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.009+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:38.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.009+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:38.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.009+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:38.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:38 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:38.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:38.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:38 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:38.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:38.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:38 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:38.589+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:38.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:38 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:38.730+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:38.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:38 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:38.730+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:38.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:38.654+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:38.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:38.654+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:38.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:38.456+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:38.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:38.456+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:38.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:38.456+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:38.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:38 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:38.456+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:39.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:39.154+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:39.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:39.154+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:39.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:39.154+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:39.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.996+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:39.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.996+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:39.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:38 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:38.996+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:39.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:39.586+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:39.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:39.586+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:39.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:39.586+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:39.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:39.741+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:39.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:39.741+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:39.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:39 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:39.741+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:39.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:39.617+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:39.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:39.617+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:39.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:39.617+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:39.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:39.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:39.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:39.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:39.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:39.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:39.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:39 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:39.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:40.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:40.122+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:40.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:40.122+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:40.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:40.122+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:40.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:40.122+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:15.313589+0000 front 2024-04-26T02:09:15.314248+0000 (oldest deadline 2024-04-26T02:09:39.915800+0000) 2024-04-26T02:09:40.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:39.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:40.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:39.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:40.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:39 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:39.991+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:40.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:40.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:40.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:40.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:40.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:40.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:40.666 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:40.401+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:40.903 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:40.559+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:40.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:40.559+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:40.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:40.559+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:40.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:40.752+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:40.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:40.752+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:40.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:40 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:40.752+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:40.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:40.666+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:40.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:40.666+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:40.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:40 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:40.666+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:41.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:41.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:41.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:41.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:41.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:41.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:41.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:41.156+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:15.313589+0000 front 2024-04-26T02:09:15.314248+0000 (oldest deadline 2024-04-26T02:09:39.915800+0000) 2024-04-26T02:09:41.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.008+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:41.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.008+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:41.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.008+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:41.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:41.368+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:41.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:41.368+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:41.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:41.368+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:41.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:41.368+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:41.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:41.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:41.554+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:41.722+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:41.722+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:41.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:41 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:41.722+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:42.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:41.703+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:42.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:41.703+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:42.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:41 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:41.703+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:42.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:42.114+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:47.595370+0000 front 2024-04-26T02:08:47.595953+0000 (oldest deadline 2024-04-26T02:09:12.895520+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:42.114+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:59.201438+0000 front 2024-04-26T02:08:59.201517+0000 (oldest deadline 2024-04-26T02:09:21.501663+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:42.114+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.809259+0000 front 2024-04-26T02:09:07.809337+0000 (oldest deadline 2024-04-26T02:09:31.309437+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:42 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6[56029]: 2024-04-26T02:09:42.114+0000 7fe811c00700 -1 osd.6 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:15.313589+0000 front 2024-04-26T02:09:15.314248+0000 (oldest deadline 2024-04-26T02:09:39.915800+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:42.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:42.465 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:41 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:41.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:42.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:42.340+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:42.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:42.340+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:42.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:42.340+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:42.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:42.340+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:42.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:42.509+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:42.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:42.509+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:42.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:42.509+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:42.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:42.677+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:42.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:42.677+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:42.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:42 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:42.677+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:43.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:42.713+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:43.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:42.713+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:43.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:42 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:42.713+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:43.214 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:42 smithi052 podman[109786]: 2024-04-26 02:09:42.94404658 +0000 UTC m=+5.754811245 container died 4ebad74a2cfdd273ae66317fedf99c039bece88a08b86ef58549a4516bcce00c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T02:09:43.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:43.036+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:43.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:43.036+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:43.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:43.036+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:43.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:43 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:43.036+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:43.447 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:43.385+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:43.447 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:43.385+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:43.448 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:43.385+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:43.448 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:43.385+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:43.584 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:43 smithi052 podman[109786]: 2024-04-26 02:09:43.400211456 +0000 UTC m=+6.210976118 container cleanup 4ebad74a2cfdd273ae66317fedf99c039bece88a08b86ef58549a4516bcce00c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-26T02:09:43.584 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:43 smithi052 bash[109786]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6 2024-04-26T02:09:43.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:43.467+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:43.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:43.467+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:43.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:43.467+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:43.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:43.682+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:43.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:43.682+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:43.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:43 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:43.682+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:43.963 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:43 smithi052 podman[109804]: 2024-04-26 02:09:43.584400336 +0000 UTC m=+0.613688048 container remove 4ebad74a2cfdd273ae66317fedf99c039bece88a08b86ef58549a4516bcce00c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:44.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:43.724+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:44.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:43.724+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:44.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:43 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:43.724+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:44.418 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:44.028+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:44.419 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:44.028+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:44.419 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:44.028+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:44.419 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:44.028+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:44.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:44.424+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:44.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:44.424+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:44.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:44.424+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:44.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:44.424+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:44.423+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:44.423+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:44.693 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:44.423+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:44.713 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:44 smithi052 podman[109889]: 2024-04-26 02:09:44.325648217 +0000 UTC m=+0.038696566 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:44.713 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:44 smithi052 podman[109889]: 2024-04-26 02:09:44.43931203 +0000 UTC m=+0.152360408 container create 93ecc28b266f27b7755e41244416c45e8072016bc71a991fb1a19f8ea86800d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:09:45.045 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:44 smithi052 podman[109889]: 2024-04-26 02:09:44.722159885 +0000 UTC m=+0.435208280 container init 93ecc28b266f27b7755e41244416c45e8072016bc71a991fb1a19f8ea86800d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T02:09:45.045 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:44 smithi052 podman[109889]: 2024-04-26 02:09:44.736214391 +0000 UTC m=+0.449262773 container start 93ecc28b266f27b7755e41244416c45e8072016bc71a991fb1a19f8ea86800d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T02:09:45.046 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:44 smithi052 podman[109889]: 2024-04-26 02:09:44.812553608 +0000 UTC m=+0.525601972 container attach 93ecc28b266f27b7755e41244416c45e8072016bc71a991fb1a19f8ea86800d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:45.153 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:44.694+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:45.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:44.694+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:45.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:44 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:44.694+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:45.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:44.697+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:45.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:44.697+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:45.176 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:44 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:44.697+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:45.463 INFO:journalctl@ceph.osd.6.smithi052.stdout:Apr 26 02:09:45 smithi052 podman[109889]: 2024-04-26 02:09:45.214346067 +0000 UTC m=+0.927394449 container died 93ecc28b266f27b7755e41244416c45e8072016bc71a991fb1a19f8ea86800d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-26T02:09:45.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:45.044+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:45.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:45.044+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:45.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:45.044+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:45.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:45 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:45.044+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:45.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:45.375+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:45.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:45.375+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:45.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:45.375+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:45.667 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:45.375+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:45.742 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:45.425+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:45.742 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:45.425+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:45.742 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:45.425+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:45.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:45.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:45.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:45.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:45.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:45 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:45.667+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:45.939 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T02:09:46.042 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:45.742+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:46.043 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:45.742+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:46.043 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:45 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:45.742+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:46.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:46.061 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T02:09:46.061 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T02:09:46.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.7 2024-04-26T02:09:46.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.030+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:46.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.030+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:46.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.030+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:46.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.030+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:46.355 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 systemd[1]: Stopping Ceph osd.7 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:46.631 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:46.356+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:46.356+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:46.356+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:46.356+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.368+0000 7ff3ca62e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.368+0000 7ff3ca62e700 -1 osd.7 137 *** Got signal Terminated *** 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.368+0000 7ff3ca62e700 -1 osd.7 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:46.632 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.632+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:46.633 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.632+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:46.633 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.632+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:46.633 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:46 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:46.632+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.940507+0000 front 2024-04-26T02:09:22.940529+0000 (oldest deadline 2024-04-26T02:09:46.440683+0000) 2024-04-26T02:09:46.779 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:46.452+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:46.779 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:46.452+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:46.779 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:46.452+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:47.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:46.779+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:47.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:46.779+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:47.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:46 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:46.779+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:47.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.999+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:47.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:46 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.999+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:47.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:47 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.999+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:47.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:47 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.999+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:47.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:47 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:46.999+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:47.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:47.592+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:47.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:47.592+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:47.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:47.592+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:47.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:47.592+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.940507+0000 front 2024-04-26T02:09:22.940529+0000 (oldest deadline 2024-04-26T02:09:46.440683+0000) 2024-04-26T02:09:47.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:47.402+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:47.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:47.402+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:47.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:47.402+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:47.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:47 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:47.402+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:47.774 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:47.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:47.774 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:47.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:47.775 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:47.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:47.775 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:47.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:48.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:47.774+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:48.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:47.774+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:48.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:47 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:47.774+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:48.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:48.037+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:48.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:48.037+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:48.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:48.037+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:48.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:48.037+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:48.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:48 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:48.037+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:48.903 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:48.497+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:48.497+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:48.497+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:48.497+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:48.738+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:48.738+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:48.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:48.738+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:48.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:48 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:48.738+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:48.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:48.621+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:48.926 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:48.621+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:48.621+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:48.621+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.940507+0000 front 2024-04-26T02:09:22.940529+0000 (oldest deadline 2024-04-26T02:09:46.440683+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:48.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:48.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:48.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:48.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:48.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:48 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:48.429+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:49.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:49.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:49.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:49.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:49.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:49.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:49.626+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:49.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:49.626+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:49.626+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:49.626+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.940507+0000 front 2024-04-26T02:09:22.940529+0000 (oldest deadline 2024-04-26T02:09:46.440683+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:49.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:49.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:49.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:49.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:49.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:49.414+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:49.745 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:49.474+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:49.745 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:49.474+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:49.745 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:49.474+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:49.745 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:49.474+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:50.153 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:49.745+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:50.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:49.745+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:50.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:49.745+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:50.154 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:49 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:49.745+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:50.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.983+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:50.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.983+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:50.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.983+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:50.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.983+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:50.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:49 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:49.983+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:50.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:50.587+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.837681+0000 front 2024-04-26T02:08:58.837887+0000 (oldest deadline 2024-04-26T02:09:22.337841+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:50.587+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.638870+0000 front 2024-04-26T02:09:07.638858+0000 (oldest deadline 2024-04-26T02:09:33.538930+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:50.587+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:13.539585+0000 front 2024-04-26T02:09:13.539867+0000 (oldest deadline 2024-04-26T02:09:39.439444+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7[50842]: 2024-04-26T02:09:50.587+0000 7ff3c6c22700 -1 osd.7 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.940507+0000 front 2024-04-26T02:09:22.940529+0000 (oldest deadline 2024-04-26T02:09:46.440683+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:50.404+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:50.404+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:50.404+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:50.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:50.404+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:50.678 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:50 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:50.404+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:50.903 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:50.709+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:50.709+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:50.709+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:50.709+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:50.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:50.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:50.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:50.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:50.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:50 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:50.514+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:51.365 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:50.974+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:51.366 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:50.974+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:51.366 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:50.974+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:51.366 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:50.974+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:51.366 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:50 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:50.974+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:51.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:51.416+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:51.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:51.416+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:51.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:51.416+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:51.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:51.416+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:51.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:51 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:51.416+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:51.535+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:51.535+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:51.535+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:51.535+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:51.662+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:51.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:51.662+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:51.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:51.662+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:51.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:51 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:51.662+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:52.310 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:52 smithi183 podman[73450]: 2024-04-26 02:09:52.000662023 +0000 UTC m=+5.743333132 container died 3e1e71c8e9901178a6e78482054d7a33a4e24cafa1bb0ca5e84b8df02f87456a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:09:52.402 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:52.403 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:52.403 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:52.403 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:52.403 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.006+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:52.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:52 smithi183 podman[73450]: 2024-04-26 02:09:52.410650831 +0000 UTC m=+6.153321940 container cleanup 3e1e71c8e9901178a6e78482054d7a33a4e24cafa1bb0ca5e84b8df02f87456a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-26T02:09:52.676 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:52 smithi183 bash[73450]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7 2024-04-26T02:09:52.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:52.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:52.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:52.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:52.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:52.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:52.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:52.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:52.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:52 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:52.445+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:52.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:52.673+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:52.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:52.673+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:52.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:52.673+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:52.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:52.673+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:52.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:52.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:52.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:52.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:52.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:52.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:52.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:52 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:52.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:53.006 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:52 smithi183 podman[73467]: 2024-04-26 02:09:52.703364546 +0000 UTC m=+0.673789364 container remove 3e1e71c8e9901178a6e78482054d7a33a4e24cafa1bb0ca5e84b8df02f87456a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T02:09:53.095 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.980+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:53.095 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.980+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:53.095 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.980+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:53.095 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.980+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:53.095 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:52 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:52.980+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:53.323 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:52.911099424 +0000 UTC m=+0.018153768 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:09:53.323 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:53.038275202 +0000 UTC m=+0.145329514 container create ac35c4366b148ae454de3c91dee53282fb8ff6ec07ea5adbba17f8aa5e4e3cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:53.323 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:53.24894279 +0000 UTC m=+0.355997102 container init ac35c4366b148ae454de3c91dee53282fb8ff6ec07ea5adbba17f8aa5e4e3cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T02:09:53.324 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:53.251993327 +0000 UTC m=+0.359047643 container start ac35c4366b148ae454de3c91dee53282fb8ff6ec07ea5adbba17f8aa5e4e3cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T02:09:53.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:53.441+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:53.676 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:53.441+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:53.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:53.441+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:53.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:53.441+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:53.677 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:53 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:53.441+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:53.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:53.323903155 +0000 UTC m=+0.430957472 container attach ac35c4366b148ae454de3c91dee53282fb8ff6ec07ea5adbba17f8aa5e4e3cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:09:53.677 INFO:journalctl@ceph.osd.7.smithi183.stdout:Apr 26 02:09:53 smithi183 podman[73557]: 2024-04-26 02:09:53.544018376 +0000 UTC m=+0.651072698 container died ac35c4366b148ae454de3c91dee53282fb8ff6ec07ea5adbba17f8aa5e4e3cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:09:53.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:53.695+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:53.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:53.695+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:53.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:53.695+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:53.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:53.695+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:53.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:53.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:53.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:53.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:53.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:53.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:53.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:53 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:53.516+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:54.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T02:09:54.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:09:54.198 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T02:09:54.198 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T02:09:54.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.8 2024-04-26T02:09:54.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:54.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:54.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:54.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:54.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:54.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:54.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:54.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:54.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:54 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:54.013+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:54.492 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 systemd[1]: Stopping Ceph osd.8 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:09:54.492 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.492+0000 7f7524e2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:09:54.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:54.550+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:54.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:54.550+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:54.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:54.550+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:54.550+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.492+0000 7f7524e2a700 -1 osd.8 137 *** Got signal Terminated *** 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.492+0000 7f7524e2a700 -1 osd.8 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.649+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.649+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.649+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:54.905 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:54 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:54.649+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:54.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:54.453+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:54.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:54.453+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:54.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:54.453+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:54.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:54.453+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:54.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:54.453+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:55.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:55.010+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:55.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:55.010+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:55.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:55.010+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:55.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:55.010+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:55.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:55 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:55.010+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:55.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:55.675+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:55.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:55.675+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:55.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:55.675+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:55.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:55.675+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:55.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:55.675+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:32.620626+0000 front 2024-04-26T02:09:32.620598+0000 (oldest deadline 2024-04-26T02:09:55.520723+0000) 2024-04-26T02:09:55.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:55.542+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:55.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:55.542+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:55.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:55.542+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:55.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:55.542+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:55.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:55 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:55.542+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:09:55.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:55.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:55.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:55.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:55.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:55.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:55 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:55.494+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:09:56.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:56.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:56.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:56.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:56.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:56.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:56 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:56.047+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:09:56.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:56.642+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:56.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:56.642+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:56.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:56.642+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:56.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:56.642+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:56.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:56.642+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:32.620626+0000 front 2024-04-26T02:09:32.620598+0000 (oldest deadline 2024-04-26T02:09:55.520723+0000) 2024-04-26T02:09:56.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:56.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:56.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:56.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:56.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:56.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:56.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:56.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:56.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:56 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:56.592+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:09:56.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:56.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:56.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:56.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:56.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:56.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:56 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:56.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:09:57.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:57.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:57.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:57.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:57.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:57.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:57 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:57.054+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:09:57.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:57.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:57.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:57.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:57.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:57.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:57.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:57.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:57.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:57.688+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:32.620626+0000 front 2024-04-26T02:09:32.620598+0000 (oldest deadline 2024-04-26T02:09:55.520723+0000) 2024-04-26T02:09:57.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:57.585+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:57.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:57.585+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:57.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:57.585+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:57.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:57.585+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:57.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:57 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:57.585+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:09:57.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:57.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:57.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:57.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:57.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:57.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:57 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:57.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:09:58.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:58.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:58.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:58.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:58.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:58.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.029+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:09:58.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:58.703+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.714903+0000 front 2024-04-26T02:08:58.714920+0000 (oldest deadline 2024-04-26T02:09:21.014893+0000) 2024-04-26T02:09:58.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:58.703+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.216823+0000 front 2024-04-26T02:09:06.216844+0000 (oldest deadline 2024-04-26T02:09:29.716914+0000) 2024-04-26T02:09:58.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:58.703+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:14.418317+0000 front 2024-04-26T02:09:14.418393+0000 (oldest deadline 2024-04-26T02:09:39.118179+0000) 2024-04-26T02:09:58.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:58.703+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.819506+0000 front 2024-04-26T02:09:23.819464+0000 (oldest deadline 2024-04-26T02:09:48.519535+0000) 2024-04-26T02:09:58.904 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8[51426]: 2024-04-26T02:09:58.703+0000 7f752141e700 -1 osd.8 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:32.620626+0000 front 2024-04-26T02:09:32.620598+0000 (oldest deadline 2024-04-26T02:09:55.520723+0000) 2024-04-26T02:09:58.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:58.568+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:58.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:58.568+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:58.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:58.568+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:58.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:58.568+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:58.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:58 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:58.568+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:09:58.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:58.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:58.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:58.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:58.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:58.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:58.468+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:09:59.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:09:59.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:09:59.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:09:59.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:09:59.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:09:59.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:58 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:58.987+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:09:59.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:59 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:59.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:09:59.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:59 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:59.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:09:59.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:59 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:59.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:09:59.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:59 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:59.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:09:59.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:09:59 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:09:59.573+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:09:59.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:09:59.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:09:59.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:09:59.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:09:59.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:09:59.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:09:59 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:09:59.516+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:00.321 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:00 smithi191 podman[73916]: 2024-04-26 02:10:00.007859025 +0000 UTC m=+5.619997782 container died 574e917e63173929724ba2e89352936ea48fa71f5e272a6eb450c6a590b97a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T02:10:00.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:00.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:00.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:00.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:00.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:00.464 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:09:59 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:09:59.989+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:00.608 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:00 smithi191 podman[73916]: 2024-04-26 02:10:00.43510689 +0000 UTC m=+6.047245648 container cleanup 574e917e63173929724ba2e89352936ea48fa71f5e272a6eb450c6a590b97a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-26T02:10:00.608 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:00 smithi191 bash[73916]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8 2024-04-26T02:10:00.903 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:00 smithi191 podman[73935]: 2024-04-26 02:10:00.644725853 +0000 UTC m=+0.631195963 container remove 574e917e63173929724ba2e89352936ea48fa71f5e272a6eb450c6a590b97a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T02:10:00.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:00 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:00.608+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:00.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:00 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:00.608+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:00.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:00 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:00.608+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:00.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:00 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:00.608+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:00.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:00 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:00.608+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:00.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:00.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:00.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:00.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:00.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:00.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:00 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:00.504+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:01.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:01.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:01.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:01.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:01.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:01.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:00 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:00.957+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:01.330 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:00.958857444 +0000 UTC m=+0.018183197 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:01.330 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:01.078978565 +0000 UTC m=+0.138304306 container create 720bae483477fe90834c5f9d5c1fceba06a1de3c6b864dd64419f7cbaa7ce560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T02:10:01.330 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:01.27286846 +0000 UTC m=+0.332194200 container init 720bae483477fe90834c5f9d5c1fceba06a1de3c6b864dd64419f7cbaa7ce560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T02:10:01.330 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:01.27606548 +0000 UTC m=+0.335391221 container start 720bae483477fe90834c5f9d5c1fceba06a1de3c6b864dd64419f7cbaa7ce560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T02:10:01.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:01 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:01.567+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:01.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:01 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:01.567+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:01.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:01 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:01.567+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:01.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:01 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:01.567+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:01.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:01 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:01.567+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:01.655 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:01.331102418 +0000 UTC m=+0.390428160 container attach 720bae483477fe90834c5f9d5c1fceba06a1de3c6b864dd64419f7cbaa7ce560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-26T02:10:01.655 INFO:journalctl@ceph.osd.8.smithi191.stdout:Apr 26 02:10:01 smithi191 podman[74025]: 2024-04-26 02:10:01.564275131 +0000 UTC m=+0.623600889 container died 720bae483477fe90834c5f9d5c1fceba06a1de3c6b864dd64419f7cbaa7ce560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:01.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:01.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:01.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:01.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:01.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:01.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:01.479+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:02.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T02:10:02.217 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:02.218 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:02.218 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:02.218 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:02.218 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:02.218 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:01 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:01.939+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:02.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:02.257 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T02:10:02.257 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T02:10:02.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.9 2024-04-26T02:10:02.713 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 systemd[1]: Stopping Ceph osd.9 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:02.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:02 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:02.563+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:02.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:02 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:02.563+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:02.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:02 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:02.563+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:02.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:02 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:02.563+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:02.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:02 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:02.563+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:02.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:02.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:02.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:02.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:02.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:02.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:02 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:02.461+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:03.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.832+0000 7fc4cd400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:03.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.832+0000 7fc4cd400700 -1 osd.9 137 *** Got signal Terminated *** 2024-04-26T02:10:03.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.832+0000 7fc4cd400700 -1 osd.9 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:03.215 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:02 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:02.905+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:03.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:03.541+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:03.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:03.541+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:03.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:03.541+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:03.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:03.541+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:03.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:03 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:03.541+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:03.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:03.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:03.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:03.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:03.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:03.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:03 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:03.455+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:04.166 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:03 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:03.871+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:04.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:04 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:04.496+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:04.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:04 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:04.496+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:04.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:04 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:04.496+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:04.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:04 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:04.496+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:04.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:04 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:04.496+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:04.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:04.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:04.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:04.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:04.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:04.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:04 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:04.473+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:05.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:05.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:05.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:05.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:05.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:05.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:04 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:04.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:05.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:05.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:05.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:05.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:05.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:05.905 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:05.481+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:05.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:05.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:05.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:05.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:05.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:05.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:05 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:05.498+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:06.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:06.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:06.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:06.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:06.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:06.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:05 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:05.858+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:06.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:06 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:06.434+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:06.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:06.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:06.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:06.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:06.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:06.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:06.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:06 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:06.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:07.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:07.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:07.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:07.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:07.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:07.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:06 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:06.823+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:07.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:07.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:07.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:07.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:07.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:07.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:07 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:07.385+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:07.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:07.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:07 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:07.578+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:08.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.085821+0000 front 2024-04-26T02:08:50.086030+0000 (oldest deadline 2024-04-26T02:09:15.386149+0000) 2024-04-26T02:10:08.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:58.190109+0000 front 2024-04-26T02:08:58.189903+0000 (oldest deadline 2024-04-26T02:09:24.090187+0000) 2024-04-26T02:10:08.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:07.592748+0000 front 2024-04-26T02:09:07.592678+0000 (oldest deadline 2024-04-26T02:09:32.292916+0000) 2024-04-26T02:10:08.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:12.294610+0000 front 2024-04-26T02:09:12.294578+0000 (oldest deadline 2024-04-26T02:09:41.095586+0000) 2024-04-26T02:10:08.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:21.096409+0000 front 2024-04-26T02:09:21.096613+0000 (oldest deadline 2024-04-26T02:09:46.996390+0000) 2024-04-26T02:10:08.214 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:07 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9[63023]: 2024-04-26T02:10:07.827+0000 7fc4c3c00700 -1 osd.9 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:31.000699+0000 front 2024-04-26T02:09:31.000768+0000 (oldest deadline 2024-04-26T02:09:55.700895+0000) 2024-04-26T02:10:08.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:08.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:08.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:08.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:08.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:08.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:08 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:08.344+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:08.693 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:08 smithi052 podman[110008]: 2024-04-26 02:10:08.426041801 +0000 UTC m=+5.723664083 container died 6a5fb73e59c77c3831d0f9f6d46bcfd11818cc16dde5377a2960f74be81f4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T02:10:08.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:08.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:08 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:08.574+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:08.963 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:08 smithi052 podman[110008]: 2024-04-26 02:10:08.809559805 +0000 UTC m=+6.107182100 container cleanup 6a5fb73e59c77c3831d0f9f6d46bcfd11818cc16dde5377a2960f74be81f4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:08.963 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:08 smithi052 bash[110008]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9 2024-04-26T02:10:09.463 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:09 smithi052 podman[110028]: 2024-04-26 02:10:09.060872542 +0000 UTC m=+0.600017816 container remove 6a5fb73e59c77c3831d0f9f6d46bcfd11818cc16dde5377a2960f74be81f4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:09.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:09.386+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:09.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:09.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:09.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:09.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:09.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:09.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:09.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:09 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:09.543+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:10.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:09 smithi052 podman[110116]: 2024-04-26 02:10:09.840617652 +0000 UTC m=+0.036166885 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:10.213 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:09 smithi052 podman[110116]: 2024-04-26 02:10:09.953655606 +0000 UTC m=+0.149204880 container create 6abc7409f9c6f1506d06beffe20796e85698e4029b737eb4d8ec2cd38d0984ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T02:10:10.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:10.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:10.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:10.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:10.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:10.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:10 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:10.342+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:10.713 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:10 smithi052 podman[110116]: 2024-04-26 02:10:10.248967543 +0000 UTC m=+0.444516813 container init 6abc7409f9c6f1506d06beffe20796e85698e4029b737eb4d8ec2cd38d0984ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T02:10:10.713 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:10 smithi052 podman[110116]: 2024-04-26 02:10:10.25954844 +0000 UTC m=+0.455097710 container start 6abc7409f9c6f1506d06beffe20796e85698e4029b737eb4d8ec2cd38d0984ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:10.713 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:10 smithi052 podman[110116]: 2024-04-26 02:10:10.347678663 +0000 UTC m=+0.543227933 container attach 6abc7409f9c6f1506d06beffe20796e85698e4029b737eb4d8ec2cd38d0984ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:10.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:10.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:10.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:10.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:10.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:10.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:10.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:10 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:10.559+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:11.104 INFO:journalctl@ceph.osd.9.smithi052.stdout:Apr 26 02:10:10 smithi052 podman[110116]: 2024-04-26 02:10:10.718493282 +0000 UTC m=+0.914042518 container died 6abc7409f9c6f1506d06beffe20796e85698e4029b737eb4d8ec2cd38d0984ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-26T02:10:11.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T02:10:11.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:11.558 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T02:10:11.558 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T02:10:11.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.10 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:11.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:11 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:11.297+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:11.887 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:11.887 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:11.887 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:11.887 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.589+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 systemd[1]: Stopping Ceph osd.10 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.887+0000 7f9650b4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.887+0000 7f9650b4e700 -1 osd.10 137 *** Got signal Terminated *** 2024-04-26T02:10:11.888 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:11 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:11.887+0000 7f9650b4e700 -1 osd.10 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:10:12.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:12.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:12.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:12.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:12.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:12.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:12 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:12.307+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:12.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:12.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:12 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:12.596+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:13.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:13.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:13.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:13.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:13.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:13.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:13.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:13.320+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:13.828 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:13.829 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:13 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:13.548+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:14.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:14.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:14 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:14.328+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:14.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:14.928 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:14 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:14.513+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:15.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:15.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:15.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:15 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:15.300+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:15.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:15.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:15 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:15.534+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:16.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:16.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:16.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:16.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:16.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:16.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:16.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:16 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:16.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:16.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:49.962867+0000 front 2024-04-26T02:08:49.962832+0000 (oldest deadline 2024-04-26T02:09:14.062546+0000) 2024-04-26T02:10:16.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:55.163349+0000 front 2024-04-26T02:08:55.163447+0000 (oldest deadline 2024-04-26T02:09:25.164212+0000) 2024-04-26T02:10:16.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:05.164802+0000 front 2024-04-26T02:09:05.164698+0000 (oldest deadline 2024-04-26T02:09:29.864727+0000) 2024-04-26T02:10:16.928 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-04-26T02:09:16.166914+0000 front 2024-04-26T02:09:16.167016+0000 (oldest deadline 2024-04-26T02:09:38.367432+0000) 2024-04-26T02:10:16.928 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:23.668629+0000 front 2024-04-26T02:09:23.668596+0000 (oldest deadline 2024-04-26T02:09:47.768622+0000) 2024-04-26T02:10:16.928 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:30.669797+0000 front 2024-04-26T02:09:30.669840+0000 (oldest deadline 2024-04-26T02:09:54.769831+0000) 2024-04-26T02:10:16.928 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:16 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10[57891]: 2024-04-26T02:10:16.511+0000 7f964913a700 -1 osd.10 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:41.572853+0000 front 2024-04-26T02:09:41.572955+0000 (oldest deadline 2024-04-26T02:10:06.272608+0000) 2024-04-26T02:10:17.579 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:17 smithi183 podman[73671]: 2024-04-26 02:10:17.306930744 +0000 UTC m=+5.545950638 container died 93db56c9974ad428c67a2be990fa2cbefdf23bd9438775cf31e6c6926285ed8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T02:10:17.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:17.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:17.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:17.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:17.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:17.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:17.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:17.278+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:17.926 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:17 smithi183 podman[73671]: 2024-04-26 02:10:17.678955822 +0000 UTC m=+5.917975724 container cleanup 93db56c9974ad428c67a2be990fa2cbefdf23bd9438775cf31e6c6926285ed8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:17.927 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:17 smithi183 bash[73671]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10 2024-04-26T02:10:18.293 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:17 smithi183 podman[73700]: 2024-04-26 02:10:17.963482221 +0000 UTC m=+0.651251940 container remove 93db56c9974ad428c67a2be990fa2cbefdf23bd9438775cf31e6c6926285ed8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T02:10:18.574 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.197390189 +0000 UTC m=+0.017501186 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:18.574 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.314174159 +0000 UTC m=+0.134285167 container create 43353326834f70315184121facfc931739448c08ec056ae93a3693d3aae48745 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:18.574 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.575192888 +0000 UTC m=+0.395303854 container init 43353326834f70315184121facfc931739448c08ec056ae93a3693d3aae48745 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:18.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:18.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:18 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:18.236+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:18.870 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.578660105 +0000 UTC m=+0.398771080 container start 43353326834f70315184121facfc931739448c08ec056ae93a3693d3aae48745 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T02:10:18.870 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.658647615 +0000 UTC m=+0.478758582 container attach 43353326834f70315184121facfc931739448c08ec056ae93a3693d3aae48745 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:19.168 INFO:journalctl@ceph.osd.10.smithi183.stdout:Apr 26 02:10:18 smithi183 podman[73794]: 2024-04-26 02:10:18.871650766 +0000 UTC m=+0.691761750 container died 43353326834f70315184121facfc931739448c08ec056ae93a3693d3aae48745 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:19.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T02:10:19.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:19.499 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T02:10:19.499 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T02:10:19.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.11 2024-04-26T02:10:19.525 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:19.525 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:19.525 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:19.525 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:19.525 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:19.526 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:19.526 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.207+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:19.802 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 systemd[1]: Stopping Ceph osd.11 for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:20.154 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.803+0000 7f6366bce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:20.154 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.803+0000 7f6366bce700 -1 osd.11 137 *** Got signal Terminated *** 2024-04-26T02:10:20.154 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:19 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:19.803+0000 7f6366bce700 -1 osd.11 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T02:10:20.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:20.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:20.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:20.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:20.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:20.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:20.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:20.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:20 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:20.223+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-26T02:09:57.114942+0000 front 2024-04-26T02:09:57.114964+0000 (oldest deadline 2024-04-26T02:10:20.014927+0000) 2024-04-26T02:10:21.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:21.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:21.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:21.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:21.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:21.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:21.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:21.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:21 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:21.186+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-26T02:09:57.114942+0000 front 2024-04-26T02:09:57.114964+0000 (oldest deadline 2024-04-26T02:10:20.014927+0000) 2024-04-26T02:10:22.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:22.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:22.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:22.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:22.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:22.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:22.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:22.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:22 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:22.206+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-26T02:09:57.114942+0000 front 2024-04-26T02:09:57.114964+0000 (oldest deadline 2024-04-26T02:10:20.014927+0000) 2024-04-26T02:10:23.654 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:23.655 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:23 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:23.170+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-26T02:09:57.114942+0000 front 2024-04-26T02:09:57.114964+0000 (oldest deadline 2024-04-26T02:10:20.014927+0000) 2024-04-26T02:10:24.404 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-04-26T02:08:50.103512+0000 front 2024-04-26T02:08:50.103282+0000 (oldest deadline 2024-04-26T02:09:14.803330+0000) 2024-04-26T02:10:24.404 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6804 osd.1 since back 2024-04-26T02:08:54.804009+0000 front 2024-04-26T02:08:54.803868+0000 (oldest deadline 2024-04-26T02:09:20.704022+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-04-26T02:09:06.605411+0000 front 2024-04-26T02:09:06.605499+0000 (oldest deadline 2024-04-26T02:09:31.905592+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-04-26T02:09:22.408390+0000 front 2024-04-26T02:09:22.408289+0000 (oldest deadline 2024-04-26T02:09:47.108286+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-04-26T02:09:33.910549+0000 front 2024-04-26T02:09:33.910688+0000 (oldest deadline 2024-04-26T02:09:55.010720+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-04-26T02:09:40.212543+0000 front 2024-04-26T02:09:40.212559+0000 (oldest deadline 2024-04-26T02:10:04.912305+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.183:6820 osd.7 since back 2024-04-26T02:09:47.813513+0000 front 2024-04-26T02:09:47.813577+0000 (oldest deadline 2024-04-26T02:10:13.014102+0000) 2024-04-26T02:10:24.405 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:24 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11[58015]: 2024-04-26T02:10:24.130+0000 7f635f1ba700 -1 osd.11 137 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-04-26T02:09:57.114942+0000 front 2024-04-26T02:09:57.114964+0000 (oldest deadline 2024-04-26T02:10:20.014927+0000) 2024-04-26T02:10:25.596 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:25 smithi191 podman[74143]: 2024-04-26 02:10:25.282189441 +0000 UTC m=+5.589604457 container died fbf7f49b39c69a692cdea01b50f32c3e82afafc253685424ffd4168bfe5dbca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:10:25.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:25 smithi191 podman[74143]: 2024-04-26 02:10:25.694470114 +0000 UTC m=+6.001885165 container cleanup fbf7f49b39c69a692cdea01b50f32c3e82afafc253685424ffd4168bfe5dbca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T02:10:25.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:25 smithi191 bash[74143]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11 2024-04-26T02:10:26.311 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:25 smithi191 podman[74160]: 2024-04-26 02:10:25.919533271 +0000 UTC m=+0.637031898 container remove fbf7f49b39c69a692cdea01b50f32c3e82afafc253685424ffd4168bfe5dbca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:26.632 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.214839553 +0000 UTC m=+0.016883380 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-26T02:10:26.632 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.321875385 +0000 UTC m=+0.123919272 container create 61789c8fd52bd3b50481407d484860ee032fb2d113dc69c7f80dd6eec2445f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:26.632 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.541704983 +0000 UTC m=+0.343748804 container init 61789c8fd52bd3b50481407d484860ee032fb2d113dc69c7f80dd6eec2445f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T02:10:26.632 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.544722093 +0000 UTC m=+0.346765915 container start 61789c8fd52bd3b50481407d484860ee032fb2d113dc69c7f80dd6eec2445f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:26.633 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.633286069 +0000 UTC m=+0.435329900 container attach 61789c8fd52bd3b50481407d484860ee032fb2d113dc69c7f80dd6eec2445f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T02:10:26.904 INFO:journalctl@ceph.osd.11.smithi191.stdout:Apr 26 02:10:26 smithi191 podman[74254]: 2024-04-26 02:10:26.839317069 +0000 UTC m=+0.641360883 container died 61789c8fd52bd3b50481407d484860ee032fb2d113dc69c7f80dd6eec2445f34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:10:27.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T02:10:27.493 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:27.493 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T02:10:27.493 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T02:10:27.494 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.a 2024-04-26T02:10:27.963 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:27 smithi052 systemd[1]: Stopping Ceph mds.a for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:28.464 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: 2024-04-26T02:10:28.102+0000 7fc647600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:28.464 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:28 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a[70457]: 2024-04-26T02:10:28.102+0000 7fc647600700 -1 mds.a *** got signal Terminated *** 2024-04-26T02:10:32.451 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:32 smithi052 podman[110234]: 2024-04-26 02:10:32.135193854 +0000 UTC m=+4.192165040 container died b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T02:10:32.713 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:32 smithi052 podman[110234]: 2024-04-26 02:10:32.560616589 +0000 UTC m=+4.617587801 container cleanup b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:10:32.714 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:32 smithi052 bash[110234]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a 2024-04-26T02:10:33.213 INFO:journalctl@ceph.mds.a.smithi052.stdout:Apr 26 02:10:32 smithi052 podman[110247]: 2024-04-26 02:10:32.895166391 +0000 UTC m=+0.740942597 container remove b6df1cc916a6481f60494ff49bda4ce6d525cc7a446b60bb94ba68f2a2a687e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T02:10:33.401 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T02:10:33.513 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:33.514 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T02:10:33.514 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T02:10:33.514 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.d 2024-04-26T02:10:33.963 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:33 smithi052 systemd[1]: Stopping Ceph mds.d for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:34.463 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d[69232]: 2024-04-26T02:10:34.094+0000 7f2aba600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:34.463 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:34 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d[69232]: 2024-04-26T02:10:34.094+0000 7f2aba600700 -1 mds.d *** got signal Terminated *** 2024-04-26T02:10:38.432 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:38 smithi052 podman[110352]: 2024-04-26 02:10:38.119836305 +0000 UTC m=+4.150972902 container died 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:10:38.713 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:38 smithi052 podman[110352]: 2024-04-26 02:10:38.539232216 +0000 UTC m=+4.570368853 container cleanup 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:38.714 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:38 smithi052 bash[110352]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d 2024-04-26T02:10:39.213 INFO:journalctl@ceph.mds.d.smithi052.stdout:Apr 26 02:10:38 smithi052 podman[110366]: 2024-04-26 02:10:38.915617636 +0000 UTC m=+0.773660668 container remove 541dde23f7cf7eebfd22c7e161abd6ee3f90803a27575245d56eb402b9acc803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-d, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:10:39.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T02:10:39.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:39.590 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T02:10:39.591 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T02:10:39.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.g 2024-04-26T02:10:40.213 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:39 smithi052 systemd[1]: Stopping Ceph mds.g for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:40.713 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g[70038]: 2024-04-26T02:10:40.242+0000 7face7000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:40.713 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:40 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g[70038]: 2024-04-26T02:10:40.242+0000 7face7000700 -1 mds.g *** got signal Terminated *** 2024-04-26T02:10:42.614 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:42 smithi052 podman[110472]: 2024-04-26 02:10:42.295475546 +0000 UTC m=+2.231569188 container died 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-26T02:10:42.963 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:42 smithi052 podman[110472]: 2024-04-26 02:10:42.726746013 +0000 UTC m=+2.662839655 container cleanup 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-26T02:10:42.963 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:42 smithi052 bash[110472]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g 2024-04-26T02:10:43.463 INFO:journalctl@ceph.mds.g.smithi052.stdout:Apr 26 02:10:43 smithi052 podman[110497]: 2024-04-26 02:10:43.03621896 +0000 UTC m=+0.719890974 container remove 3195b405ccc9d97522c1f8906e00d09cac2bd3732d345a66374cff79e1827946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T02:10:43.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T02:10:43.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:43.696 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T02:10:43.696 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T02:10:43.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.j 2024-04-26T02:10:43.964 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:43 smithi052 systemd[1]: Stopping Ceph mds.j for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:44.713 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j[68804]: 2024-04-26T02:10:44.286+0000 7f1a4b000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:44.714 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:44 smithi052 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j[68804]: 2024-04-26T02:10:44.286+0000 7f1a4b000700 -1 mds.j *** got signal Terminated *** 2024-04-26T02:10:48.583 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:48 smithi052 podman[110603]: 2024-04-26 02:10:48.331837541 +0000 UTC m=+4.172186097 container died 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:48.914 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:48 smithi052 podman[110603]: 2024-04-26 02:10:48.730369515 +0000 UTC m=+4.570718080 container cleanup 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:10:48.914 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:48 smithi052 bash[110603]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j 2024-04-26T02:10:49.213 INFO:journalctl@ceph.mds.j.smithi052.stdout:Apr 26 02:10:48 smithi052 podman[110618]: 2024-04-26 02:10:48.914371658 +0000 UTC m=+0.569470404 container remove 67b9a82133eaa48291f303b0d99e8568d18b000561ef38326aec15f8d7969291 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T02:10:49.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T02:10:49.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:49.498 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T02:10:49.498 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T02:10:49.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.b 2024-04-26T02:10:49.836 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:49 smithi183 systemd[1]: Stopping Ceph mds.b for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:50.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: 2024-04-26T02:10:49.836+0000 7fa21bf0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:50.177 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:49 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b[63449]: 2024-04-26T02:10:49.836+0000 7fa21bf0e700 -1 mds.b *** got signal Terminated *** 2024-04-26T02:10:53.176 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:52 smithi183 podman[73899]: 2024-04-26 02:10:52.762343808 +0000 UTC m=+3.067107635 container died cef119fcec4f5825996c239f84dc0e7057227f5ef92c22cd57aaa96a542d253e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T02:10:53.594 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:53 smithi183 podman[73899]: 2024-04-26 02:10:53.377725716 +0000 UTC m=+3.682489541 container cleanup cef119fcec4f5825996c239f84dc0e7057227f5ef92c22cd57aaa96a542d253e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T02:10:53.594 INFO:journalctl@ceph.mds.b.smithi183.stdout:Apr 26 02:10:53 smithi183 bash[73899]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-b 2024-04-26T02:10:53.730 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T02:10:53.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:53.765 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T02:10:53.765 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T02:10:53.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.e 2024-04-26T02:10:54.176 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:53 smithi183 systemd[1]: Stopping Ceph mds.e for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:54.177 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e[62816]: 2024-04-26T02:10:54.080+0000 7f83953be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:54.177 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:54 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e[62816]: 2024-04-26T02:10:54.080+0000 7f83953be700 -1 mds.e *** got signal Terminated *** 2024-04-26T02:10:57.589 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:57 smithi183 podman[74033]: 2024-04-26 02:10:57.308980097 +0000 UTC m=+3.327422330 container died 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T02:10:57.926 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:57 smithi183 podman[74033]: 2024-04-26 02:10:57.68875893 +0000 UTC m=+3.707201161 container cleanup 8c567928518ceab7ea4a9666484b39b88f19bbd2469780bbcc3442634e113c1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T02:10:57.926 INFO:journalctl@ceph.mds.e.smithi183.stdout:Apr 26 02:10:57 smithi183 bash[74033]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-e 2024-04-26T02:10:58.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T02:10:58.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:10:58.102 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T02:10:58.102 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T02:10:58.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.h 2024-04-26T02:10:58.427 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:10:58 smithi183 systemd[1]: Stopping Ceph mds.h for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:10:58.926 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:10:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: 2024-04-26T02:10:58.433+0000 7fbbce191700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:10:58.927 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:10:58 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h[63127]: 2024-04-26T02:10:58.433+0000 7fbbce191700 -1 mds.h *** got signal Terminated *** 2024-04-26T02:11:01.008 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:00 smithi183 podman[74153]: 2024-04-26 02:11:00.759133444 +0000 UTC m=+2.439884056 container died afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:11:01.409 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 podman[74153]: 2024-04-26 02:11:01.225335655 +0000 UTC m=+2.906086275 container cleanup afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T02:11:01.409 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 bash[74153]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h 2024-04-26T02:11:01.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T02:11:01.660 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 podman[74167]: 2024-04-26 02:11:01.409996008 +0000 UTC m=+0.616055830 container remove afe162391a93bb6ade2bb40f0a2748937f224f8d64a87c8b9b86b8fe9293d5f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T02:11:01.660 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 systemd[1]: ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-26T02:11:01.661 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 systemd[1]: Stopped Ceph mds.h for 8b150f36-036c-11ef-bc93-c7b262605968. 2024-04-26T02:11:01.661 INFO:journalctl@ceph.mds.h.smithi183.stdout:Apr 26 02:11:01 smithi183 systemd[1]: ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.h.service: Consumed 33.588s CPU time. 2024-04-26T02:11:01.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:01.671 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T02:11:01.671 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T02:11:01.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.k 2024-04-26T02:11:01.926 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:01 smithi183 systemd[1]: Stopping Ceph mds.k for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:02.426 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k[62491]: 2024-04-26T02:11:01.987+0000 7f4b011b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:02.427 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:01 smithi183 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k[62491]: 2024-04-26T02:11:01.987+0000 7f4b011b5700 -1 mds.k *** got signal Terminated *** 2024-04-26T02:11:04.176 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:03 smithi183 podman[74278]: 2024-04-26 02:11:03.912478788 +0000 UTC m=+2.030623537 container died b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T02:11:04.445 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:04 smithi183 podman[74278]: 2024-04-26 02:11:04.254207821 +0000 UTC m=+2.372352566 container cleanup b67a73380ba3f9e2606a73fb2803f2778297bd3fd7802889b78bdc1ff37ba627 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T02:11:04.446 INFO:journalctl@ceph.mds.k.smithi183.stdout:Apr 26 02:11:04 smithi183 bash[74278]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-k 2024-04-26T02:11:04.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T02:11:04.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:04.693 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T02:11:04.693 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T02:11:04.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.c 2024-04-26T02:11:05.154 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:04 smithi191 systemd[1]: Stopping Ceph mds.c for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:05.417 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: 2024-04-26T02:11:05.218+0000 7fc83cb8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:05.417 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:05 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c[63316]: 2024-04-26T02:11:05.218+0000 7fc83cb8c700 -1 mds.c *** got signal Terminated *** 2024-04-26T02:11:09.076 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:08 smithi191 podman[74357]: 2024-04-26 02:11:08.766054863 +0000 UTC m=+3.876319285 container died f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-26T02:11:09.076 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:09 smithi191 podman[74357]: 2024-04-26 02:11:09.077159215 +0000 UTC m=+4.187423640 container cleanup f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T02:11:09.404 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:09 smithi191 bash[74357]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c 2024-04-26T02:11:09.404 INFO:journalctl@ceph.mds.c.smithi191.stdout:Apr 26 02:11:09 smithi191 podman[74383]: 2024-04-26 02:11:09.303491748 +0000 UTC m=+0.535342591 container remove f0ff8f7696e618a8c8fb1a1e19620a4639aa8eee43a72424c7dcc84fa86b5ed4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T02:11:09.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T02:11:09.542 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:09.542 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T02:11:09.543 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T02:11:09.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.f 2024-04-26T02:11:09.855 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:09 smithi191 systemd[1]: Stopping Ceph mds.f for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:10.154 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: 2024-04-26T02:11:09.855+0000 7f22b4315700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:10.154 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:09 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f[63004]: 2024-04-26T02:11:09.855+0000 7f22b4315700 -1 mds.f *** got signal Terminated *** 2024-04-26T02:11:13.064 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:12 smithi191 podman[74492]: 2024-04-26 02:11:12.774056814 +0000 UTC m=+3.018186782 container died 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T02:11:13.397 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:13 smithi191 podman[74492]: 2024-04-26 02:11:13.147582916 +0000 UTC m=+3.391712896 container cleanup 2757e8f5f1bc4b2688256aedebb6c013a1952bffb1d7582c58af44fcc5fdcaeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416) 2024-04-26T02:11:13.398 INFO:journalctl@ceph.mds.f.smithi191.stdout:Apr 26 02:11:13 smithi191 bash[74492]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-f 2024-04-26T02:11:13.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T02:11:13.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:13.568 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T02:11:13.568 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T02:11:13.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.i 2024-04-26T02:11:13.904 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:13 smithi191 systemd[1]: Stopping Ceph mds.i for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:13.904 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: 2024-04-26T02:11:13.891+0000 7f573381b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:13.905 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:13 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i[63624]: 2024-04-26T02:11:13.891+0000 7f573381b700 -1 mds.i *** got signal Terminated *** 2024-04-26T02:11:17.025 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:16 smithi191 podman[74615]: 2024-04-26 02:11:16.772698727 +0000 UTC m=+2.983720648 container died 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-26T02:11:17.404 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:17 smithi191 podman[74615]: 2024-04-26 02:11:17.142594675 +0000 UTC m=+3.353616594 container cleanup 03e3269b72bfd2f7913182dfbb029f684615b0aa2f1bc6066448f1067949aa74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T02:11:17.404 INFO:journalctl@ceph.mds.i.smithi191.stdout:Apr 26 02:11:17 smithi191 bash[74615]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-i 2024-04-26T02:11:17.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T02:11:17.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:17.573 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T02:11:17.573 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T02:11:17.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.l 2024-04-26T02:11:17.895 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:17 smithi191 systemd[1]: Stopping Ceph mds.l for 8b150f36-036c-11ef-bc93-c7b262605968... 2024-04-26T02:11:18.154 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l[62704]: 2024-04-26T02:11:17.895+0000 7f27104b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T02:11:18.154 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:17 smithi191 ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l[62704]: 2024-04-26T02:11:17.895+0000 7f27104b3700 -1 mds.l *** got signal Terminated *** 2024-04-26T02:11:21.404 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:21 smithi191 podman[74735]: 2024-04-26 02:11:21.153595969 +0000 UTC m=+3.363261310 container died dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T02:11:21.754 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:21 smithi191 podman[74735]: 2024-04-26 02:11:21.546030684 +0000 UTC m=+3.755696075 container cleanup dbad05a3ba9c644b8cb0984e7c5825733a7cfcb5902c301d31c443a9b5a16a85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T02:11:21.754 INFO:journalctl@ceph.mds.l.smithi191.stdout:Apr 26 02:11:21 smithi191 bash[74735]: ceph-8b150f36-036c-11ef-bc93-c7b262605968-mds-l 2024-04-26T02:11:21.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b150f36-036c-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T02:11:21.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T02:11:21.909 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T02:11:21.909 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:22.241 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:27.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:27.691 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:30.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T02:11:30.406 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:11:32.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:32.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:32.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T02:11:32.969 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T02:11:32.971 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi052/crash 2024-04-26T02:11:32.972 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:33.077 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:33.078 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:33.082 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi183/crash 2024-04-26T02:11:33.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:33.126 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:33.126 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:33.128 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi191/crash 2024-04-26T02:11:33.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash -- . 2024-04-26T02:11:33.168 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/8b150f36-036c-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T02:11:33.168 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-04-26T02:11:33.169 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T02:11:33.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-26T02:11:33.300 INFO:tasks.cephadm:Compressing logs... 2024-04-26T02:11:33.301 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:33.305 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:33.308 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:33.336 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T02:11:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:33.338 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T02:11:33.338 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:33.351 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T02:11:33.351 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:33.351 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:33.353 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T02:11:33.353 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T02:11:33.353 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T02:11:33.353 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T02:11:33.357 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T02:11:33.357 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:33.376 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log 2024-04-26T02:11:33.377 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-04-26T02:11:33.377 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T02:11:33.378 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T02:11:33.380 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T02:11:33.381 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T02:11:33.382 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T02:11:33.383 INFO:teuthology.orchestra.run.smithi052.stderr: 87.4% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:33.384 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T02:11:33.385 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T02:11:33.386 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T02:11:33.387 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T02:11:33.388 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T02:11:33.388 INFO:teuthology.orchestra.run.smithi052.stderr: 94.4% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T02:11:33.388 INFO:teuthology.orchestra.run.smithi052.stderr: 94.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T02:11:33.389 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T02:11:33.390 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T02:11:33.391 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T02:11:33.391 INFO:teuthology.orchestra.run.smithi052.stderr: 90.3% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:33.392 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T02:11:33.395 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 84.4% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:33.398 INFO:teuthology.orchestra.run.smithi052.stderr: 94.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T02:11:33.404 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 84.2% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:33.434 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 84.1% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T02:11:33.435 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:33.445 INFO:teuthology.orchestra.run.smithi052.stderr: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:33.492 INFO:teuthology.orchestra.run.smithi191.stderr: 94.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T02:11:33.534 INFO:teuthology.orchestra.run.smithi183.stderr: 94.4% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T02:11:33.557 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:33.592 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T02:11:33.638 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:33.653 INFO:teuthology.orchestra.run.smithi183.stderr: 92.6% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T02:11:33.679 INFO:teuthology.orchestra.run.smithi183.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T02:11:33.685 INFO:teuthology.orchestra.run.smithi052.stderr: 90.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:33.752 INFO:teuthology.orchestra.run.smithi183.stderr: 94.4% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T02:11:33.893 INFO:teuthology.orchestra.run.smithi191.stderr: 91.0% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:33.902 INFO:teuthology.orchestra.run.smithi052.stderr: 90.1% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T02:11:34.126 INFO:teuthology.orchestra.run.smithi183.stderr: 87.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T02:11:34.127 INFO:teuthology.orchestra.run.smithi183.stderr: 90.5% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T02:11:34.512 INFO:teuthology.orchestra.run.smithi183.stderr: 91.0% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T02:11:34.628 INFO:teuthology.orchestra.run.smithi191.stderr: 91.7% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T02:11:36.069 INFO:teuthology.orchestra.run.smithi052.stderr: 91.7% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T02:11:36.759 INFO:teuthology.orchestra.run.smithi183.stderr: 91.6% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T02:11:44.298 INFO:teuthology.orchestra.run.smithi191.stderr: 94.8% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T02:11:44.437 INFO:teuthology.orchestra.run.smithi183.stderr: 94.1% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T02:11:45.126 INFO:teuthology.orchestra.run.smithi052.stderr: 93.8% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T02:11:46.977 INFO:teuthology.orchestra.run.smithi052.stderr: 93.6% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T02:11:47.898 INFO:teuthology.orchestra.run.smithi052.stderr: 93.8% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T02:11:48.129 INFO:teuthology.orchestra.run.smithi191.stderr: 94.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T02:11:50.196 INFO:teuthology.orchestra.run.smithi191.stderr: 94.0% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T02:11:52.432 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T02:11:52.537 INFO:teuthology.orchestra.run.smithi052.stderr: 93.7% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T02:11:54.258 INFO:teuthology.orchestra.run.smithi183.stderr: 93.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T02:11:55.097 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T02:11:55.666 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T02:11:56.529 INFO:teuthology.orchestra.run.smithi183.stderr: 93.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T02:11:56.832 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T02:12:01.963 INFO:teuthology.orchestra.run.smithi183.stderr: 94.6% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T02:14:11.063 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T02:14:11.065 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-04-26T02:14:11.065 INFO:teuthology.orchestra.run.smithi191.stderr:real 2m37.742s 2024-04-26T02:14:11.065 INFO:teuthology.orchestra.run.smithi191.stderr:user 3m24.734s 2024-04-26T02:14:11.066 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m6.913s 2024-04-26T02:14:20.803 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T02:14:20.807 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-04-26T02:14:20.808 INFO:teuthology.orchestra.run.smithi052.stderr:real 2m47.466s 2024-04-26T02:14:20.808 INFO:teuthology.orchestra.run.smithi052.stderr:user 3m28.143s 2024-04-26T02:14:20.808 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m15.708s 2024-04-26T02:14:46.720 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/8b150f36-036c-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T02:14:46.723 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-04-26T02:14:46.723 INFO:teuthology.orchestra.run.smithi183.stderr:real 3m13.399s 2024-04-26T02:14:46.724 INFO:teuthology.orchestra.run.smithi183.stderr:user 3m37.558s 2024-04-26T02:14:46.724 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m7.457s 2024-04-26T02:14:46.724 INFO:tasks.cephadm:Archiving logs... 2024-04-26T02:14:46.725 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi052/log 2024-04-26T02:14:46.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:15:17.874 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi183/log 2024-04-26T02:15:17.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:15:53.686 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi191/log 2024-04-26T02:15:53.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T02:16:25.655 INFO:tasks.cephadm:Removing cluster... 2024-04-26T02:16:25.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:16:25.998 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:16:27.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:16:27.414 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:16:27.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b150f36-036c-11ef-bc93-c7b262605968 --force 2024-04-26T02:16:28.119 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 8b150f36-036c-11ef-bc93-c7b262605968 2024-04-26T02:16:28.641 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T02:16:28.641 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:16:28.701 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:16:28.718 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T02:16:28.733 INFO:tasks.cephadm:Teardown complete 2024-04-26T02:16:28.734 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T02:16:28.745 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T02:16:28.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:16:28.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:16:28.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T02:16:28.926 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T02:16:28.937 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T02:16:28.938 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:16:28.946 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:16:28.949 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T02:16:28.961 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:16:28.962 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +1775us[+1767us] +/- 56ms 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 39 -2053us[-2061us] +/- 77ms 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -487us[ -487us] +/- 86ms 2024-04-26T02:16:28.976 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:16:28.978 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:16:28.978 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-04-26T02:16:28.978 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 38 +1774us[+1764us] +/- 56ms 2024-04-26T02:16:28.979 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -2053us[-2053us] +/- 77ms 2024-04-26T02:16:28.979 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -493us[ -493us] +/- 86ms 2024-04-26T02:16:28.979 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:16:28.997 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-04-26T02:16:29.008 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T02:16:29.008 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-04-26T02:16:29.009 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 +1802us[+1776us] +/- 56ms 2024-04-26T02:16:29.009 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -2037us[-2063us] +/- 77ms 2024-04-26T02:16:29.009 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -462us[ -462us] +/- 86ms 2024-04-26T02:16:29.009 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T02:16:29.010 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T02:16:29.021 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T02:16:29.022 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T02:16:29.036 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:16:29.241 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:16:29.347 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T02:16:29.468 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:16:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:29.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:16:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:29.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T02:16:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:29.645 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T02:16:29.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T02:16:30.050 INFO:teuthology.task.internal:Duration was 3204.740413 seconds 2024-04-26T02:16:30.051 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T02:16:30.359 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T02:16:30.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:16:30.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:16:30.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T02:16:30.430 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:16:30.457 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:16:30.524 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T02:16:30.852 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T02:16:30.853 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-04-26T02:16:30.853 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:16:30.920 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-04-26T02:16:30.921 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:16:30.951 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-04-26T02:16:30.951 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T02:16:30.980 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T02:16:30.980 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:16:30.987 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:16:30.994 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T02:16:31.079 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T02:16:31.079 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:16:31.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:16:31.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T02:16:32.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T02:16:32.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T02:16:32.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:16:32.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:16:32.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T02:16:32.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T02:16:32.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.291 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-04-26T02:16:32.298 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-04-26T02:16:32.354 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-04-26T02:16:32.399 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:32.450 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:32.468 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T02:16:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T02:16:32.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T02:16:32.497 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T02:16:32.499 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi052 2024-04-26T02:16:32.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:16:32.622 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi183 2024-04-26T02:16:32.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:16:32.663 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243/remote/smithi191 2024-04-26T02:16:32.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T02:16:32.704 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T02:16:32.704 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:16:32.712 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:16:32.715 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T02:16:32.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T02:16:32.779 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T02:16:32.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T02:16:32.791 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T02:16:32.792 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:16:32.798 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:16:32.800 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T02:16:32.815 INFO:teuthology.orchestra.run.smithi183.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:16 /home/ubuntu/cephtest 2024-04-26T02:16:32.816 INFO:teuthology.orchestra.run.smithi191.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:16 /home/ubuntu/cephtest 2024-04-26T02:16:32.851 INFO:teuthology.orchestra.run.smithi052.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 02:16 /home/ubuntu/cephtest 2024-04-26T02:16:32.855 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T02:16:32.867 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T02:16:32.892 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T02:16:32.916 DEBUG:teuthology.nuke:shortname: smithi052 2024-04-26T02:16:32.917 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:16:32.942 DEBUG:teuthology.nuke:shortname: smithi183 2024-04-26T02:16:32.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:16:32.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.640272', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlsE4bIuJOYjuYlcXDpAU37FmMSFR02PenOXHuN6ARC6AXVZlCAcocRMjgVxzY0r2yPVaMOlof5GJe05/u/eMg='} 2024-04-26T02:16:32.966 DEBUG:teuthology.nuke:shortname: smithi191 2024-04-26T02:16:32.966 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T02:16:32.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.637573', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDiP3f3MXF4PdVtlzdlkuUVuhuBZGLA13XYRkho5YDU/a88iMCf2zGKaCdBwhNeacLDgYewsdU/77wxVmEpZx+o='} 2024-04-26T02:16:32.994 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-26_01:02:28-fs-wip-lusov-quiescer-distro-default-smithi/7674243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:13:58.639046', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOCKKC3srvdG16hmd3HpmC2lMy2JjxgpFJH8vzz/TbhLOHYb5EU+0HvWhKPFjzv3t6CRwXt4gvn+ExZ2BGOLd4='} 2024-04-26T02:16:33.006 INFO:teuthology.orchestra.console.smithi052:Power off 2024-04-26T02:16:33.006 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:16:33.031 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-04-26T02:16:33.031 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:33.049 INFO:teuthology.orchestra.console.smithi183:Power off 2024-04-26T02:16:33.050 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:16:33.064 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-04-26T02:16:33.074 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-04-26T02:16:33.074 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:33.092 INFO:teuthology.orchestra.console.smithi191:Power off 2024-04-26T02:16:33.093 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T02:16:33.103 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-04-26T02:16:33.118 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-04-26T02:16:33.118 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:33.148 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-04-26T02:16:37.065 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:37.105 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:37.149 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:37.176 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-04-26T02:16:37.216 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-04-26T02:16:37.262 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-04-26T02:16:41.178 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:41.218 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:41.264 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:41.291 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-04-26T02:16:41.328 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-04-26T02:16:41.374 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-04-26T02:16:41.391 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-04-26T02:16:41.475 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-04-26T02:16:45.332 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T02:16:45.443 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-04-26T02:16:45.543 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-04-26T02:16:45.650 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 3204.7404131889343 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi052 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=afa1933b0fdbb6c99c947d1eda34d661d23cd327 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=326fac8eca2948e9b95dfbf16bce577a status: fail success: false 2024-04-26T02:16:45.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T02:16:45.731 INFO:teuthology.run:FAIL